# Use AWS IoT Greengrass Testing Framework Greengrass Testing Framework (GTF) is a collection of building blocks that supports end-to-end automation from the customer perspective. GTF uses [Cucumber](https://cucumber.io) as the feature driver. AWS IoT Greengrass uses the same building blocks to qualify software changes on various devices. For more information, see [Greengrass Testing Framework on Github](https://github.com/aws-greengrass/aws-greengrass-testing/tree/dev_v1). GTF is implemented using Cucumber, a tool used to run automated tests, to encourage a Behavior-Driven Development (BDD) of the components. In Cucumber, the features of this system are outlined in a special type of file called `feature`. Each feature is described in a human-readable format called scenarios which are specifications that can be converted into automated tests. Each scenario is outlined as a series of steps that define the interactions and outcomes of this system under test using a domain-specific language called Gherkin. A [Gherkin step](https://cucumber.io/docs/gherkin/reference/#steps) is linked to the programming code using a method called step definition which hard wires the specification to the test flow. Step definitions in GTF are implemented with Java. **Topics** + [How it works](#gg-testing-framework-how-gtf-works) + [Changelog](#gtf-changelog) + [Greengrass Testing Framework configuration options](configuration-options-gtf.md) + [Tutorial: Run end-to-end tests using Greengrass Testing Framework and Greengrass Development Kit](run-e2e-tests-tutorial.md) + [Tutorial: Use a confidence test from the confidence test suite](confidence-tests-tutorial.md) ## How it works AWS IoT Greengrass distributes the GTF as a standalone JAR that consists of several Java modules. To use GTF for end-to-end testing of components, you must implement the tests within a Java project. Adding the testing standable JAR as a dependency in your Java project enables you to use the existing functionality of the GTF and extend it by writing your own custom test cases. To run the custom test cases, you can build your Java project and run the target JAR with the configuration options described in [Greengrass Testing Framework configuration options](configuration-options-gtf.md). ### GTF standalone JAR Greengrass uses Cloudfront as a [Maven](https://maven.apache.org/) repository to host different versions of the GTF standalone JAR. For a full list of GTF versions, see [GTF releases](https://github.com/aws-greengrass/aws-greengrass-testing/releases). GTF standalone JAR includes the following modules. It is not limited to only these modules. You can pick and choose each of these dependencies separately in your project or include all of them at once with the [testing standalone JAR file](https://github.com/aws-greengrass/aws-greengrass-testing/tree/dev_v1/aws-greengrass-testing-standalone). + `aws-greengrass-testing-resources`: This module provides abstraction for managing the lifecycle of an AWS resource during the course of a test. You can use this to define your custom AWS resources using `ResourceSpec` abstraction so GTF can take care of creation and removal of those resources for you. + `aws-greengrass-testing-platform`: This module provides platform-level abstraction for the device under test during the test lifecycle. It contains APIs used to interact with the OS independent of the platform and can be used to simulate the commands running in the device shell. + `aws-greengrass-testing-components`: This module consists of sample components that are used for testing the Greengrass core features such as deployments, IPC, and other features. + `aws-greengrass-testing-features`: This module consists of reusable common steps and their definitions which are used for testing within in the Greengrass environment. **Topics** + [How it works](#gg-testing-framework-how-gtf-works) + [Changelog](#gtf-changelog) + [Greengrass Testing Framework configuration options](configuration-options-gtf.md) + [Tutorial: Run end-to-end tests using Greengrass Testing Framework and Greengrass Development Kit](run-e2e-tests-tutorial.md) + [Tutorial: Use a confidence test from the confidence test suite](confidence-tests-tutorial.md) ## Changelog The following table describes the changes in each version of the GTF. For more information, see the [GTF Releases page](https://github.com/aws-greengrass/aws-greengrass-testing/releases) on GitHub. | **Version** | **Changes** | | --- | --- | | 1.2.0 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/greengrass/v2/developerguide/gg-testing-framework.html) [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/greengrass/v2/developerguide/gg-testing-framework.html) | | 1.1.0 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/greengrass/v2/developerguide/gg-testing-framework.html) | | 1.0.0 | Initial version. | # Greengrass Testing Framework configuration options ## GTF configuration options Greengrass Testing Framework (GTF) enables you to configure certain parameters during the launch of the end-to-end testing process to orchestrate the test flow. You can specify these configuration options as CLI arguments for the GTF standalone JAR. GTF version 1.1.0 and later provides the following configuration options. + `additional-plugins` – (Optional) Additional Cucumber plugins + `aws-region` – Targets specific regional endpoints for AWS services. Defaults to what the AWS SDK discovers. + `credentials-path` – Optional AWS profile credentials path. Defaults to credentials discovered on host environment. + `credentials-path-rotation` – Optional rotation duration for AWS credentials. Defaults to 15 minutes or `PT15M`. + `csr-path` – The path for the CSR using which the device certificate will be generated. + `device-mode` – The target device under test. Defaults to local device. + `env-stage` – Targets the deployment environment of Greengrass. Defaults to production. + `existing-device-cert-arn` – The arn of an existing certificate that you want to use as a device certificate for Greengrass. + `feature-path` – File or directory containing additional feature files. Default is no additional feature files are used. + `gg-cli-version` – Overrides the version of the Greengrass CLI. Defaults to the value found in `ggc.version`. + `gg-component-bucket` – The name of an existing Amazon S3 bucket that houses Greengrass components. + `gg-component-overrides` – A list of Greengrass component overrides. + `gg-persist` – A list of test elements to persist after a test run. Default behavior is to persist nothing. Accepted values are: `aws.resources`, `installed.software`, and `generated.files`. + `gg-runtime` – A list of values to influence how the test interacts with testing resources. These values supersede the `gg.persist` parameter. If the default is empty, it assumes all testing resources are manged by test case, including the installed Greengrass runtime. Accepted values are: `aws.resources`, `installed.software`, and `generated.files`. + `ggc-archive` – The path to the archived Greengrass nucleus component. + `ggc-install-root` – Directory to install the Greengrass nucleus component. Defaults to test.temp.path and test run folder. + `ggc-log-level` – Set the Greengrass nucleus log level for the test run. Default is "INFO". + `ggc-tes-rolename` – The IAM role that AWS IoT Greengrass Core will assume to access AWS services. If a role with given name does not exist then one will be created and default access policy. + `ggc-trusted-plugins` – The comma separate list of the paths (on host) of the trusted plugins that need to added to Greengrass. To provide the path on the DUT itself, prefix the path with 'dut:' + `ggc-user-name` – The user:group posixUser value for the Greengrass nucleus. Defaults to the current username that is logged in. + `ggc-version` – Overrides the version of the running Greengrass nucleus component. Defaults to the value found in ggc.archive. + `log-level` – Log level of the test run. Defaults to "INFO". + `parallel-config` – Set of batch index and number of batches as a JSON String. Default value of batch index is 0 and number of batches is 1. + `proxy-url` – Configure all tests to route traffic through this URL. + `tags` – Only run feature tags. Can be intersected with '&' + `test-id-prefix` – A common prefix applied to all test specific resources including AWS resource names and tags. Default is a "gg" prefix. + `test-log-path` – Directory that will contain the results of the entire test run. Defaults to "testResults". + `test-results-json` – Flag to determine if a resulting Cucumber JSON report is generated written to disk. Defaults to true. + `test-results-log` – Flag to determine if the console output is generated written to disk. Defaults to false. + `test-results-xml` – Flag to determine if a resulting JUnit XML report is generated written to disk. Defaults to true. + `test-temp-path` – Directory to generate local test artifacts. Defaults to a random temp directory prefixed with gg-testing. + `timeout-multiplier` – Multiplier provided to all test timeouts. Default is 1.0. # Tutorial: Run end-to-end tests using Greengrass Testing Framework and Greengrass Development Kit AWS IoT Greengrass Testing Framework (GTF) and Greengrass Development Kit (GDK) offer developers ways to run end-to-end tests. You can complete this tutorial to initialize a GDK project with a component, initialize a GDK project with an end-to-end test module, and build a custom test case. After you build your custom test case, you can then run the test. In this tutorial, you do the following: 1. Initialize a GDK project with a component. 1. Initialize a GDK project with an end-to-end test module. 1. Build a custom test case. 1. Add a tag to the new test case. 1. Build the test JAR. 1. Run the test. **Topics** + [Prerequisites](#run-e2e-tests-tutorial-prerequisites) + [Step 1: Initialize a GDK project with a component](#init-gdk-with-component) + [Step 2: Initialize a GDK project with an end-to-end test module](#init-gdk-with-e2e-test) + [Step 3: Build a custom test case](#run-e2e-tests-tutorial-instructions) + [Step 4: Add a tag to the new test case](#add-tag-to-test-case) + [Step 5: Build the test JAR](#build-test-jar) + [Step 6: Run the test](#run-test-gtf) + [Example: Build a custom test case](#build-test-case-example) ## Prerequisites To complete this tutorial, you need the following: + GDK version 1.3.0 or later + Java + Maven + Git ## Step 1: Initialize a GDK project with a component + Initialize an empty folder with a GDK project. Download the `HelloWorld` component implemented in Python by running the following command. ``` gdk component init -t HelloWorld -l python -n HelloWorld ``` This command creates a new directory named `HelloWorld` in the current directory. ## Step 2: Initialize a GDK project with an end-to-end test module + GDK enables you to download the testing module template consisting of a feature and step implementation. Run the following command to open the `HelloWorld` directory and initialize the existing GDK project using a testing module. ``` cd HelloWorld gdk test-e2e init ``` This command creates a new directory named `gg-e2e-tests` within the `HelloWorld` directory. This test directory is a [Maven](https://maven.apache.org/) project which has a dependency on the Greengrass testing standalone JAR. ## Step 3: Build a custom test case Writing a custom test case broadly consists of two steps: create a feature file with a test scenario and implement step definitions. For an example of building a custom test case, see [Example: Build a custom test case](#build-test-case-example). Use the following steps to build your custom test case: 1. Create a feature file with a test scenario A feature typically describes a specific functionality of the software that is being tested. In Cucumber, each feature is specified as an individual feature file with a title, a detailed description, and one or more examples of specific cases called scenarios. Each scenario consists of a title, a detailed description, and a series of steps that define the interactions and expected outcomes. Scenarios are written in a structured format using "given," "when," and "then" keywords. 1. Implement step definitions A step definition links the [Gherkin step](https://cucumber.io/docs/gherkin/reference/#steps) in plain language to the programmatic code. When Cucumber identifies a Gherkin step in a scenario, it will look for a matching step definition to run. ## Step 4: Add a tag to the new test case + You can assign tags to the features and scenarios to organize the test process. You can use tags to categorize the subsets of scenarios and also select hooks conditionally to run. Features and scenarios can have multiple tags separated by a space. In this example, we are using the `HelloWorld` component. In the feature file, add a new tag named `@HelloWorld` beside the `@Sample` tag. ``` @Sample @HelloWorld Scenario: As a developer, I can create a component and deploy it on my device .... ``` ## Step 5: Build the test JAR 1. Build the component. You must build the component before building the test module. ``` gdk component build ``` 1. Build the test module using the following command. This command will build the testing JAR in the `greengrass-build` folder. ``` gdk test-e2e build ``` ## Step 6: Run the test When you run a custom test case, the GTF automates the lifecycle of the test along with managing resources that were created during the test. It first provisions a device under test (DUT) as an AWS IoT thing and installs the Greengrass core software on it. It will then create a new component named `HelloWorld` using the recipe specified in that path. The `HelloWorld` component is then deployed onto the core device through a Greengrass thing deployment. It will then be verified if the deployment is successful. The deployment status will changed to `COMPLETED` within 3 minutes if the deployment is successful. 1. Go to the `gdk-config.json` file in the project directory to target the tests with the `HelloWorld` tag. Update the the `test-e2e` key using the following command. ``` "test-e2e":{ "gtf_options" : { "tags":"HelloWorld" } } ``` 1. Before running the tests, you must provide AWS credentials to the host device. GTF uses these credentials to manage the AWS resources during the testing process. Make sure the role you provide has permissions to automate the necessary operations that are included in the test. Run the following commands to provide the AWS credentials. 1. ------ #### [ Linux or Unix ] ``` export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY ``` ------ #### [ Windows Command Prompt (CMD) ] ``` set AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE set AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY ``` ------ #### [ PowerShell ] ``` $env:AWS_ACCESS_KEY_ID="AKIAIOSFODNN7EXAMPLE" $env:AWS_SECRET_ACCESS_KEY="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" ``` ------ 1. Run the test using the following command. ``` gdk test-e2e run ``` This command downloads the latest version of the Greengrass nucleus in the `greengrass-build` folder and runs tests using it. This command also targets only the scenarios with the `HelloWorld` tag and generates a report for those scenarios. You will see the AWS resources that were created during this test are discarded at the end of the test. ## Example: Build a custom test case **Example** The downloaded testing module in the GDK project consists of a sample feature and a step implementation file. In the following example, we create a feature file for testing the thing deployment feature of the Greengrass software. We partially test the functionality of this feature with a scenario that performs deployment of a component through the Greengrass AWS Cloud. This is a series of steps that help us to understand the interactions and expected outcomes of this use case. 1. **Create a feature file** Navigate to the `gg-e2e-tests/src/main/resources/greengrass/features` folder in the current directory. You can find the sample `component.feature` that looks like the following example. In this feature file, you can test the thing deployment feature of the Greengrass software. You can partially test the functionality of this feature with a scenario that performs a deployment of a component through the Greengrass cloud. The scenario is a series of steps that help with understanding the interactions and expected outcomes of this use case. ``` Feature: Testing features of Greengrassv2 component Background: Given my device is registered as a Thing And my device is running Greengrass @Sample Scenario: As a developer, I can create a component and deploy it on my device When I create a Greengrass deployment with components HelloWorld | /path/to/recipe/file And I deploy the Greengrass deployment configuration Then the Greengrass deployment is COMPLETED on the device after 180 seconds And I call my custom step ``` GTF contains the step definitions of all of the following steps, except for the step named: `And I call my custom step`. 1. **Implement step definitions** GTF standalone JAR contains the step definitions of all of the steps except for one step: `And I call my custom step`. You can implement this step in the testing module. Navigate to the source code of the testing file. You can link your custom step using a step definition by using the following command. ``` @And("I call my custom step") public void customStep() { System.out.println("My custom step was called "); } ``` # Tutorial: Use a confidence test from the confidence test suite AWS IoT Greengrass Testing Framework (GTF) and Greengrass Development Kit (GDK) offer developers ways to run end-to-end tests. You can complete this tutorial to initialize a GDK project with a component, initialize a GDK project with an end-to-end test module, and use a confidence test from the confidence test suite. After you build your custom test case, you can then run the test. A confidence test is a generic test provided by Greengrass that validates fundamental component behaviors. These tests can be modified or extended to fit more specific component needs. For this tutorial we will be using a HelloWorld component. If you are using another component, replace the HelloWorld component with your component. In this tutorial, you do the following: 1. Initialize a GDK project with a component. 1. Initialize a GDK project with an end-to-end test module. 1. Use a test from the confidence test suite. 1. Add a tag to the new test case. 1. Build the test JAR. 1. Run the test. **Topics** + [Prerequisites](#confidence-tests-tutorial-prerequisites) + [Step 1: Initialize a GDK project with a component](#init-gdk-with-component) + [Step 2: Initialize a GDK project with an end-to-end test module](#init-gdk-with-e2e-test) + [Step 3: Use a test from the confidence test suite](#confidence-tests-tutorial-instructions) + [Step 4: Add a tag to the new test case](#add-tag-to-test-case) + [Step 5: Build the test JAR](#build-test-jar) + [Step 6: Run the test](#run-test-gtf) + [Example: Use a confidence test](#build-confidence-test-case-example) ## Prerequisites To complete this tutorial, you need the following: + GDK version 1.6.0 or later + Java + Maven + Git ## Step 1: Initialize a GDK project with a component + Initialize an empty folder with a GDK project. Download the `HelloWorld` component implemented in Python by running the following command. ``` gdk component init -t HelloWorld -l python -n HelloWorld ``` This command creates a new directory named `HelloWorld` in the current directory. ## Step 2: Initialize a GDK project with an end-to-end test module + GDK enables you to download the testing module template consisting of a feature and step implementation. Run the following command to open the `HelloWorld` directory and initialize the existing GDK project using a testing module. ``` cd HelloWorld gdk test-e2e init ``` This command creates a new directory named `gg-e2e-tests` within the `HelloWorld` directory. This test directory is a [Maven](https://maven.apache.org/) project which has a dependency on the Greengrass testing standalone JAR. ## Step 3: Use a test from the confidence test suite Writing a confidence test case consists of using the provided feature file and, if needed, modifying the scenarios. For an example of using a confidence test, see [Example: Build a custom test case](run-e2e-tests-tutorial.md#build-test-case-example). Use the following steps to use a confidence test: + Use the provided feature file. Navigate to `gg-e2e-tests/src/main/resources/greengrass/features` folder in the current directory. Open the sample `confidenceTest.feature` file to use the confidence test. ## Step 4: Add a tag to the new test case + You can assign tags to the features and scenarios to organize the test process. You can use tags to categorize the subsets of scenarios and also select hooks conditionally to run. Features and scenarios can have multiple tags separated by a space. In this example, we are using the `HelloWorld` component. Each scenario is tagged with `@ConfidenceTest`. Change or add tags if you want to run only a subset of the test suite. Each test scenario is described at the top of each confidence test. The scenario is a series of steps that help with understanding the interactions and expected outcomes of each test case. You can extend these tests by adding your own steps or by modifying the existing ones. ``` @ConfidenceTest Scenario: As a Developer, I can deploy GDK_COMPONENT_NAME to my device and see it is working as expected .... ``` ## Step 5: Build the test JAR 1. Build the component. You must build the component before building the test module. ``` gdk component build ``` 1. Build the test module using the following command. This command will build the testing JAR in the `greengrass-build` folder. ``` gdk test-e2e build ``` ## Step 6: Run the test When you run a confidence test, the GTF automates the lifecycle of the test along with managing resources that were created during the test. It first provisions a device under test (DUT) as an AWS IoT thing and installs the Greengrass core software on it. It will then create a new component named `HelloWorld` using the recipe specified in that path. The `HelloWorld` component is then deployed onto the core device through a Greengrass thing deployment. It will then be verified if the deployment is successful. The deployment status will changed to `COMPLETED` within 3 minutes if the deployment is successful. 1. Go to the `gdk-config.json` file in the project directory to target the tests with the `ConfidenceTest` tag or whichever tag yo8u specified in Step 4. Update the the `test-e2e` key using the following command. ``` "test-e2e":{ "gtf_options" : { "tags":"ConfidenceTest" } } ``` 1. Before running the tests, you must provide AWS credentials to the host device. GTF uses these credentials to manage the AWS resources during the testing process. Make sure the role you provide has permissions to automate the necessary operations that are included in the test. Run the following commands to provide the AWS credentials. 1. ------ #### [ Linux or Unix ] ``` export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY ``` ------ #### [ Windows Command Prompt (CMD) ] ``` set AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE set AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY ``` ------ #### [ PowerShell ] ``` $env:AWS_ACCESS_KEY_ID="AKIAIOSFODNN7EXAMPLE" $env:AWS_SECRET_ACCESS_KEY="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" ``` ------ 1. Run the test using the following command. ``` gdk test-e2e run ``` This command downloads the latest version of the Greengrass nucleus in the `greengrass-build` folder and runs tests using it. This command also targets only the scenarios with the `ConfidenceTest` tag and generates a report for those scenarios. You will see the AWS resources that were created during this test are discarded at the end of the test. ## Example: Use a confidence test **Example** The downloaded testing module in the GDK project consists of a provided feature file. In the following example, we use a feature file for testing the thing deployment feature of the Greengrass software. We partially test the functionality of this feature with a scenario that performs deployment of a component through the Greengrass AWS Cloud. This is a series of steps that help us to understand the interactions and expected outcomes of this use case. + **Use the provided feature file.** Navigate to the `gg-e2e-tests/src/main/resources/greengrass/features` folder in the current directory. You can find the sample `confidenceTest.feature` that looks like the following example. ``` Feature: Confidence Test Suite Background: Given my device is registered as a Thing And my device is running Greengrass @ConfidenceTest Scenario: As a Developer, I can deploy GDK_COMPONENT_NAME to my device and see it is working as expected When I create a Greengrass deployment with components | GDK_COMPONENT_NAME | GDK_COMPONENT_RECIPE_FILE | | aws.greengrass.Cli | LATEST | And I deploy the Greengrass deployment configuration Then the Greengrass deployment is COMPLETED on the device after 180 seconds # Update component state accordingly. Possible states: {RUNNING, FINISHED, BROKEN, STOPPING} And I verify the GDK_COMPONENT_NAME component is RUNNING using the greengrass-cli ``` Each test scenario is described at the top of each confidence test. The scenario is a series of steps that help with understanding the interactions and expected outcomes of each test case. You can extend these tests by adding your own steps or by modifying the existing ones. Each of the scenarios include comments that help you to make these adjustments.