# Device Advisor [Device Advisor](https://aws.amazon.com/iot-core/features/) is a cloud-based, fully managed test capability for validating IoT devices during device software development. Device Advisor provides pre-built tests that you can use to validate IoT devices for reliable and secure connectivity with AWS IoT Core, before deploying devices to production. Device Advisor’s pre-built tests help you validate your device software against best practices for usage of [TLS](https://docs.aws.amazon.com//iot/latest/developerguide/protocols.html), [MQTT](https://docs.aws.amazon.com//iot/latest/developerguide/protocols.html), [Device Shadow](https://docs.aws.amazon.com//iot/latest/developerguide/iot-device-shadows.html), and [IoT Jobs](https://docs.aws.amazon.com//iot/latest/developerguide/iot-jobs.html). You can also download signed qualification reports to submit to the AWS Partner Network to get your device qualified for the [AWS Partner Device Catalog](https://devices.amazonaws.com/) without the need to send your device in and wait for it to be tested. **Note** Device Advisor is supported in us-east-1, us-west-2, ap-northeast-1, and eu-west-1 regions. Device Advisor supports devices and clients that use the MQTT and the MQTT over WebSocket Secure (WSS) protocols to publish and subscribe to messages. All protocols support IPv4 and IPv6. Device Advisor supports RSA server certificates. Any device that has been built to connect to AWS IoT Core can take advantage of Device Advisor. You can access Device Advisor from the [AWS IoT console](https://us-east-1.console.aws.amazon.com/iot/home?region=us-east-1#/deviceadvisor/intro), or by using the AWS CLI or SDK. When you're ready to test your device, register it with AWS IoT Core and configure the device software with the Device Advisor endpoint. Then choose the prebuilt tests, configure them, run the tests on your device, and get the test results along with detailed logs or a qualification report. Device Advisor is a test endpoint in the AWS cloud. You can test your devices by configuring them to connect to the test endpoint provided by the Device Advisor. After a device is configured to connect to the test endpoint, you can visit the Device Advisor’s console or use the AWS SDK to choose the tests you want to run on your devices. Device Advisor then manages the full lifecycle of a test, including the provisioning of resources, scheduling of the test process, managing the state machine, recording the device behavior, logging the results, and providing the final results in form of a test report. **TLS protocols** Transport Layer Security (TLS) protocol is used to encrypt confidential data over insecure networks like the internet. The TLS protocol is the successor of the Secure Sockets Layer (SSL) protocol. Device Advisor supports the following TLS protocols: + TLS 1.3 (recommended) + TLS 1.2 **Protocols, port mappings, and authentication** The device communication protocol is used by a device or a client to connect to the message broker by using a device endpoint. The following table lists the protocols that the Device Advisor endpoints support and the authentication methods and ports used. **Protocols, authentication, and port mappings** | Protocol | Operations supported | Authentication | Port | ALPN protocol name | | --- | --- | --- | --- | --- | | MQTT over WebSocket | Publish, Subscribe | Signature Version 4 | 443 | N/A | | MQTT | Publish, Subscribe | X.509 client certificate | 8883 | `x-amzn-mqtt-ca` | | MQTT | Publish, Subscribe | X.509 client certificate | 443 | N/A | **Topics** + [Setting up](device-advisor-setting-up.md) + [Getting started with Device Advisor in the console](da-console-guide.md) + [Device Advisor workflow](device-advisor-workflow.md) + [Device Advisor detailed console workflow](device-advisor-console-tutorial.md) + [Long duration tests console workflow](device-advisor-long-duration-console-tutorial.md) + [Device Advisor VPC endpoints (AWS PrivateLink)](device-advisor-vpc-endpoint.md) + [Device Advisor test cases](device-advisor-tests.md) # Setting up Before you use Device Advisor for the first time, complete the following tasks: ## Create an IoT thing First, create an IoT thing and attach a certificate to that thing. For a tutorial on how to create things, see [Create a thing object](https://docs.aws.amazon.com/iot/latest/developerguide/create-iot-resources.html#create-aws-thing). ## Create an IAM role to use as your device role **Note** You can quickly create the device role with the Device Advisor console. To learn how to set up your device role with the Device Advisor console, see [ Getting started with the Device Advisor in the console](https://docs.aws.amazon.com/iot/latest/developerguide/da-console-guide.html). 1. Go to the [AWS Identity and Access Management console](https://console.aws.amazon.com/iam/home?region=us-west-2#/home) and log in to the AWS account you use for Device Advisor testing. 1. In the left navigation pane, chose **Policies**. 1. Choose **Create policy**. 1. Under **Create policy**, do the following: 1. For **Service**, choose **IoT**. 1. Under **Actions**, do one of the following: + (Recommended) Select actions based on the policy attached to the IoT thing or certificate you created in the previous section. + Search for the following actions in the **Filter action** box and select them: + `Connect` + `Publish` + `Subscribe` + `Receive` + `RetainPublish` 1. Under **Resources**, restrict the client, topic, and topic resources. Restricting these resources is a security best practice. To restrict resources, do the following: 1. Choose **Specify client resource ARN for the Connect action**. 1. Choose **Add ARN**, then do either of the following: **Note** The *clientId* is the MQTT client ID that your device uses to interact with Device Advisor. + Specify the **Region**, **accountID**, and **clientID** in the visual ARN editor. + Manually enter the Amazon Resource Names (ARNs) of the IoT topics you want to run your test cases with. 1. Choose **Add**. 1. Choose **Specify topic resource ARN for the Receive and one more action**. 1. Choose **Add ARN**, then do either of the following: **Note** The *topic name* is the MQTT topic that your device publishes messages to. + Specify the **Region**, **accountID**, and **Topic name** in the visual ARN editor. + Manually enter the ARNs of the IoT topics you want to run your test cases with. 1. Choose **Add**. 1. Choose **Specify topicFilter resource ARN for the Subscribe action**. 1. Choose **Add ARN**, then do either of the following: **Note** The *topic name* is the MQTT topic that your device subscribes to. + Specify the **Region**, **accountID**, and **Topic name** in the visual ARN editor. + Manually enter the ARNs of the IoT topics you want to run your test cases with. 1. Choose **Add**. 1. Choose **Next: Tags**. 1. Choose **Next: Review**. 1. Under **Review policy**, enter a **Name** for your policy. 1. Choose **Create policy**. 1. On the left navigation pane, Choose **Roles**. 1. Choose **Create Role**. 1. Under **Select trusted entity**, choose **Custom trust policy**. 1. Enter the following trust policy into the **Custom trust policy** box. To protect against the confused deputy problem, add the global condition context keys `[aws:SourceArn](https://docs.aws.amazon.com//IAM/latest/UserGuide/reference_policies_condition-keys.html#condition-keys-sourcearn)` and `[aws:SourceAccount](https://docs.aws.amazon.com//IAM/latest/UserGuide/reference_policies_condition-keys.html#condition-keys-sourceaccount)` to the policy. **Important** Your `aws:SourceArn` must comply with the `format: arn:aws:iotdeviceadvisor:region:account-id:*.` Make sure that `region` matches your AWS IoT Region and `account-id` matches your customer account ID. For more information, see [Cross-service confused deputy prevention](https://docs.aws.amazon.com/iot/latest/developerguide/security-best-practices.html#cross-service-confused-deputy-prevention-DA). **** ``` { "Version":"2012-10-17", "Statement": [ { "Sid": "AllowAwsIoTCoreDeviceAdvisor", "Effect": "Allow", "Principal": { "Service": "iotdeviceadvisor.amazonaws.com" }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "aws:SourceAccount": "123456789012" }, "ArnLike": { "aws:SourceArn": "arn:aws:iotdeviceadvisor:*:123456789012:suitedefinition/*" } } } ] } ``` 1. Choose **Next**. 1. Choose the policy you created in Step 4. 1. (Optional) Under **Set permissions boundary**, choose **Use a permissions boundary to control the maximum role permissions**, and then select the policy you created. 1. Choose **Next**. 1. Enter a **Role name** and a **Role description**. 1. Choose **Create role**. ## Create a custom-managed policy for an IAM user to use Device Advisor 1. Navigate to the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/). If prompted, enter your AWS credentials to sign in. 1. In the left navigation pane, choose **Policies**. 1. Choose **Create Policy**, then choose the **JSON** tab. 1. Add the necessary permissions to use Device Advisor. The policy document can be found in the topic [Security best practices](https://docs.aws.amazon.com/iot/latest/developerguide/security-best-practices.html#device-advisor-perms). 1. Choose **Review Policy**. 1. Enter a **Name** and **Description**. 1. Choose **Create Policy**. ## Create an IAM user to use Device Advisor **Note** We recommend that you create an IAM user to use when you run Device Advisor tests. Running Device Advisor tests from an admin user can pose security risks and isn't recommended. 1. Navigate to the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/) If prompted, enter your AWS credentials to sign in. 1. In the left navigation pane, Choose **Users**. 1. Choose **Add User**. 1. Enter a **User name**. 1. Users need programmatic access if they want to interact with AWS outside of the AWS Management Console. The way to grant programmatic access depends on the type of user that's accessing AWS. To grant users programmatic access, choose one of the following options. **** [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/iot/latest/developerguide/device-advisor-setting-up.html) 1. Choose **Next: Permissions**. 1. To provide access, add permissions to your users, groups, or roles: + Users and groups in AWS IAM Identity Center: Create a permission set. Follow the instructions in [Create a permission set](https://docs.aws.amazon.com//singlesignon/latest/userguide/howtocreatepermissionset.html) in the *AWS IAM Identity Center User Guide*. + Users managed in IAM through an identity provider: Create a role for identity federation. Follow the instructions in [Create a role for a third-party identity provider (federation)](https://docs.aws.amazon.com//IAM/latest/UserGuide/id_roles_create_for-idp.html) in the *IAM User Guide*. + IAM users: + Create a role that your user can assume. Follow the instructions in [Create a role for an IAM user](https://docs.aws.amazon.com//IAM/latest/UserGuide/id_roles_create_for-user.html) in the *IAM User Guide*. + (Not recommended) Attach a policy directly to a user or add a user to a user group. Follow the instructions in [Adding permissions to a user (console)](https://docs.aws.amazon.com//IAM/latest/UserGuide/id_users_change-permissions.html#users_change_permissions-add-console) in the *IAM User Guide*. 1. Enter the name of the custom-managed policy that you created in the search box. Then, select the check box for **Policy name**. 1. Choose **Next: Tags**. 1. Choose **Next: Review**. 1. Choose **Create user**. 1. Choose **Close**. Device Advisor requires access to your AWS resources (things, certificates, and endpoints) on your behalf. Your IAM user must have the necessary permissions. Device Advisor will also publish logs to Amazon CloudWatch if you attach the necessary permissions policy to your IAM user. ## Configure your device Device Advisor uses the server name indication (SNI) TLS extension to apply TLS configurations. Devices must use this extension when they connect and pass a server name that is identical to the Device Advisor test endpoint. Device Advisor allows the TLS connection when a test is in the `Running` state. It denies the TLS connection before and after each test run. For this reason, we recommend that you use the device connect retry mechanism for a fully automated testing experience with Device Advisor. You can run test suites that include more than one test case, such as TLS connect, MQTT connect, and MQTT publish. If you run multiple test cases, we recommend that your device try to connect to our test endpoint every five seconds. You can then automate running multiple test cases in sequence. **Note** To ready your device software for testing, we recommend that you use an SDK that can connect to AWS IoT Core. You should then update the SDK with the Device Advisor test endpoint provided for your AWS account. Device Advisor supports two types of endpoints: Account-level and Device-level endpoints. Choose the endpoint that best fits your use case. To simultaneously run multiple test suites for different devices, use a Device-level endpoint. Run the following command to get the Device-level endpoint: For MQTT customers using X.509 client certificates: ``` aws iotdeviceadvisor get-endpoint --thing-arn your-thing-arn ``` or ``` aws iotdeviceadvisor get-endpoint --certificate-arn your-certificate-arn ``` For MQTT over WebSocket customers using Signature Version 4: ``` aws iotdeviceadvisor get-endpoint --device-role-arn your-device-role-arn --authentication-method SignatureVersion4 ``` To run one test suite at a time, choose an Account-level endpoint. Run the following command to get the Account-level endpoint: ``` aws iotdeviceadvisor get-endpoint ``` # Getting started with Device Advisor in the console This tutorial helps you get started with AWS IoT Core Device Advisor on the console. Device Advisor offers features such as required tests and signed qualification reports. You can use these tests and reports to qualify and list devices in the [AWS Partner Device Catalog](https://devices.amazonaws.com/) as detailed in the [AWS IoT Core qualification program](https://aws.amazon.com/partners/dqp/). For more information about using Device Advisor, see [Device Advisor workflow](device-advisor-workflow.md) and [Device Advisor detailed console workflow](device-advisor-console-tutorial.md). To complete this tutorial, follow the steps outlined in [Setting up](device-advisor-setting-up.md). **Note** Device Advisor is supported in the following AWS Regions: US East (N. Virginia) US West (Oregon) Asia Pacific (Tokyo) Europe (Ireland) **Getting started** 1. In the [AWS IoT console's](https://console.aws.amazon.com//iot) navigation pane under **Test**, choose **Device Advisor**. Then, choose the **Start walkthrough** button on the console. ![\[Device Advisor is a fully managed test capability for IoT devices to validate secure interaction with AWS IoT Core, identify software issues, and get test results.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/da-console-gs.png) 1. The **Getting started with Device Advisor** page provides an overview of the steps required to create a test suite and run tests against your device. You can also find the Device Advisor test endpoint for your account here. You must configure the firmware or software on the device used for testing to connect to this test endpoint. To complete this tutorial, first [ create a thing and certificate](https://docs.aws.amazon.com/iot/latest/developerguide/device-advisor-setting-up.html#da-create-thing-certificate). After you review the information under **How it works**, choose **Next**. ![\[Steps to test IoT devices' connectivity: select protocol, create test suite, configure device settings.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/da-console-gs1.png) 1. In **Step 1: Select a protocol**, select a protocol from the options listed. Then, choose **Next**. ![\[Device Advisor interface showing options to select a communication protocol (MQTT 3.1.1, MQTT 3.1.1 over WebSocket, MQTT 5, MQTT 5 over WebSocket) for testing an IoT device.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/da-console-protocol.png) 1. In **Step 2**, you create and configure a custom test suite. A custom test suite must have at least one test group, and each test group must have at least one test case. We've added the **MQTT Connect** test case for you to get started. Choose **Test suite properties**. ![\[The "Create test suite" screen in Device Advisors, where users can create and configure test groups and cases for testing IoT devices with MQTT protocol.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/da-console-test-suite-create.png) Supply the test suite properties when you create your test suite. You can configure the following suite-level properties: + **Test suite name**: Enter a name for your test suite. + **Timeout** (optional): The timeout (in seconds) for each test case in the current test suite. If you don't specify a timeout value, the default value is used. + **Tags** (optional): Add tags to the test suite. When you’ve finished, choose **Update properties**. ![\[Form to update properties of a test suite, including name, timeout, and ability to add tags. Contains a "Update properties" button.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/da-console-test-suite-properties.png) 1. (Optional) To update the test suite group configuration, choose the **Edit** button next to the test group name. + **Name**: Enter a custom name for the test suite group. + **Timeout** (optional): The timeout (in seconds) for each test case in the current test suite. If you don't specify a timeout value, the default value is used. When finished, choose **Done** to continue. ![\[A test group named "Test group 1" is displayed with options to configure timeout and add more test groups.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/da-console-test-suite-config.png) 1. (Optional) To update the test case configuration for a test case, choose the **Edit** button next to the test case name. + **Name**: Enter a custom name for the test suite group. + **Timeout** (optional): The timeout (in seconds) for the selected test case. If you don't specify a timeout value, the default value is used. When finished, choose **Done** to continue. ![\[The "Create test suite" interface that shows options to configure a test suite, test groups, and individual test cases for testing IoT devices.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/da-console-test-case-config.png) 1. (Optional) To add more test groups to the test suite, choose **Add test group**, then follow the instructions in Step 5. 1. (Optional) To add more test cases, drag the test cases in the **Test cases** section into any of your test groups. ![\[The "Create test suite" interface where users can configure test groups and test cases for MQTT protocol testing of IoT devices.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/da-console-drag.png) 1. You can change the order of your test groups and test cases. To make changes, drag the listed test cases up or down the list. Device Advisor runs tests in the order you listed them in. After you've configured your test suite, choose **Next**. 1. In **Step 3**, select an AWS IoT thing or certificate to test using Device Advisor. If you don't have any existing things or certificates, see [Setting up](https://docs.aws.amazon.com/iot/latest/developerguide/device-advisor-setting-up.html). ![\[The configuration options that include selecting a protocol, creating a test suite, configuring device settings, and reviewing test runs and results.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/da-device-settings.png) 1. You can configure a device role that Device Advisor uses to perform AWS IoT MQTT actions on behalf of your test device. For **MQTT Connect** test case only, the **Connect** action is selected automatically. This is because the device role requires this permission to run the test suite. For other test cases, the corresponding actions are selected. Provide the resource values for each of the selected actions. For example, for the **Connect** action, provide the client ID your device uses to connect to the Device Advisor endpoint. You can provide multiple values with comma seperated values, and prefix values with a wildcard (\$1) character. For example, to provide permission to publish on any topic beginning with `MyTopic`, enter **MyTopic\$1** as the resource value. ![\[The Device Advisor interface where you can select a device role and define permissions for connecting, publishing, subscribing, and managing MQTT topics and client IDs.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/da-console-device-role.png) To use a previously created device role from [Setting up](https://docs.aws.amazon.com/iot/latest/developerguide/device-advisor-setting-up.html), choose **Select an existing role**. Then choose your device role under **Select role**. ![\[A web form interface for selecting a device role, with options to create a new role or select an existing role named "DeviceAdvisorServiceRole".\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/da-console-select-device-role.png) Configure your device role with one of the two provided options, and then choose **Next**. 1. In the **Test endpoint** section, select the endpoint that best fits your use case. To run multiple test suites simultaneously with the same AWS account, select **Device-level endpoint**. To run one test suite at a time, select **Account-level endpoint**. ![\[Options to select Account-level or Device-level endpoint for testing, with an endpoint URL provided and Next button.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/da-console-endpoint.png) 1. **Step 4** shows an overview of the selected test device, test endpoint, test suite, and test device role configured. To make changes to a section, choose the **Edit** button for the section you want to edit. Once you've confirmed your test configuration, choose **Run** to create the test suite and run your tests. **Note** For best results, you can connect your selected test device to the Device Advisor test endpoint before you start the test suite run. We recommend that you have a mechanism built for your device to try connecting to our test endpoint every five seconds for up to one to two minutes. ![\[A device configuration console that shows device role details, test endpoint, and options to cancel, go back, or run.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/da-console-device-review.png) ![\[A device configuration console that shows device role details, test endpoint, and options to cancel, go back, or run.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/da-console-device-review-contd.png) 1. In the navigation pane under **Test**, choose **Device Advisor**, and then choose **Test runs and results**. Select a test suite run to view its run details and logs. ![\[The test suite interface that indicates an MQTT 3.1.1 test is in progress for the device "MyThing".\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/da-console-runs-results.png) 1. To access the Amazon CloudWatch logs for the suite run: + Choose **Test suite log** to view the CloudWatch logs for the test suite run. + Choose **Test case log** for any test case to view test case-specific CloudWatch logs. 1. Based on your test results, [troubleshoot](https://docs.aws.amazon.com/iot/latest/developerguide/iot_troubleshooting.html#device-advisor-troubleshooting) your device until all tests pass. # Device Advisor workflow This tutorial explains how to create a custom test suite and run tests against the device you want to test in the console. After the tests are complete, you can view the test results and detailed logs. ## Prerequisites Before you begin this tutorial this tutorial, complete the steps outlined in [Setting up](device-advisor-setting-up.md). ## Create a test suite definition First, [install an AWS SDK](https://docs.aws.amazon.com/iot/latest/developerguide/iot-connect-service.html#iot-service-sdks). ### `rootGroup` syntax A root group is a JSON string that specifies which test cases to include in your test suite. It also specifies any necessary configurations for those test cases. Use the root group to structure and order your test suite based on your needs. The hierarchy of a test suite is: ``` test suite → test group(s) → test case(s) ``` A test suite must have at least one test group, and each test group must have at least one test case. Device Advisor runs tests in the order in which you define the test groups and test cases. Each root group follows this basic structure: ``` { "configuration": { // for all tests in the test suite "": "" } "tests": [{ "name": "" "configuration": { // for all sub-groups in this test group "": "" }, "tests": [{ "name": "" "configuration": { // for all test cases in this test group "": "" }, "test": { "id": "" "version": "" } }] }] } ``` In the root group, you define the test suite with a `name`, `configuration`, and the `tests` that the group contains. The `tests` group contains the definitions of individual tests. You define each test with a `name`, `configuration`, and a `test` block that defines the test cases for that test. Finally, each test case is defined with an `id` and `version`. For information on how to use the `"id"` and `"version"` fields for each test case (`test` block), see [Device Advisor test cases](device-advisor-tests.md). That section also contains information on the available `configuration` settings. The following block is an example of a root group configuration. This configurations specifies the *MQTT Connect Happy Case* and *MQTT Connect Exponential Backoff Retries* test cases, along with descriptions of the configuration fields. ``` { "configuration": {}, // Suite-level configuration "tests": [ // Group definitions should be provided here { "name": "My_MQTT_Connect_Group", // Group definition name "configuration": {} // Group definition-level configuration, "tests": [ // Test case definitions should be provided here { "name": "My_MQTT_Connect_Happy_Case", // Test case definition name "configuration": { "EXECUTION_TIMEOUT": 300 // Test case definition-level configuration, in seconds }, "test": { "id": "MQTT_Connect", // test case id "version": "0.0.0" // test case version } }, { "name": "My_MQTT_Connect_Jitter_Backoff_Retries", // Test case definition name "configuration": { "EXECUTION_TIMEOUT": 600 // Test case definition-level configuration, in seconds }, "test": { "id": "MQTT_Connect_Jitter_Backoff_Retries", // test case id "version": "0.0.0" // test case version } }] }] } ``` You must supply the root group configuration when you create your test suite definition. Save the `suiteDefinitionId` that is returned in the response object. You can use this ID to retrieve your test suite definition information and run your test suite. Here is a Java SDK example: ``` response = iotDeviceAdvisorClient.createSuiteDefinition( CreateSuiteDefinitionRequest.builder() .suiteDefinitionConfiguration(SuiteDefinitionConfiguration.builder() .suiteDefinitionName("your-suite-definition-name") .devices( DeviceUnderTest.builder() .thingArn("your-test-device-thing-arn") .certificateArn("your-test-device-certificate-arn") .deviceRoleArn("your-device-role-arn") //if using SigV4 for MQTT over WebSocket .build() ) .rootGroup("your-root-group-configuration") .devicePermissionRoleArn("your-device-permission-role-arn") .protocol("MqttV3_1_1 || MqttV5 || MqttV3_1_1_OverWebSocket || MqttV5_OverWebSocket") .build() ) .build() ) ``` ## Get a test suite definition After you create your test suite definition, you receive the `suiteDefinitionId` in the response object of the `CreateSuiteDefinition` API operation. When the operation returns the `suiteDefinitionId`, you may see new `id` fields within each group and test case definition within the root group. You can use these IDs to run a subset of your test suite definition. Java SDK example: ``` response = iotDeviceAdvisorClient.GetSuiteDefinition( GetSuiteDefinitionRequest.builder() .suiteDefinitionId("your-suite-definition-id") .build() ) ``` ## Get a test endpoint Use the `GetEndpoint` API operation to get the test endpoint used by your device. Select the endpoint that best fits your test. To simultaneously run multiple test suites, use the Device-level endpoint by providing a `thing ARN`, `certificate ARN`, or `device role ARN`. To run a single test suite, provide no arguments to the GetEndpoint operation to choose the Account-level endpoint. SDK example: ``` response = iotDeviceAdvisorClient.getEndpoint(GetEndpointRequest.builder() .certificateArn("your-test-device-certificate-arn") .thingArn("your-test-device-thing-arn") .deviceRoleArn("your-device-role-arn") //if using SigV4 for MQTT over WebSocket .build()) ``` ## Start a test suite run After you create a test suite definition and configureyour test device to connect to your Device Advisor test endpoint, run your test suite with the `StartSuiteRun` API. For MQTT customers, use either `certificateArn` or `thingArn` to run the test suite. If both are configured, the certificate is used if it belongs to the thing. For MQTT over WebSocket customer, use `deviceRoleArn` to run the test suite. If the specified role is different from the role specified in the test suite definition, the specified role overrides the defined role. For `.parallelRun()`, use `true` if you use a Device-level endpoint to run multiple test suites in parallel using one AWS account. SDK example: ``` response = iotDeviceAdvisorClient.startSuiteRun(StartSuiteRunRequest.builder() .suiteDefinitionId("your-suite-definition-id") .suiteRunConfiguration(SuiteRunConfiguration.builder() .primaryDevice(DeviceUnderTest.builder() .certificateArn("your-test-device-certificate-arn") .thingArn("your-test-device-thing-arn") .deviceRoleArn("your-device-role-arn") //if using SigV4 for MQTT over WebSocket .build()) .parallelRun(true | false) .build()) .build()) ``` Save the `suiteRunId` from the response. You will use this to retrieve the results of this test suite run. ## Get a test suite run After you start a test suite run, you can check its progress and its results with the `GetSuiteRun` API. SDK example: ``` // Using the SDK, call the GetSuiteRun API. response = iotDeviceAdvisorClient.GetSuiteRun( GetSuiteRunRequest.builder() .suiteDefinitionId("your-suite-definition-id") .suiteRunId("your-suite-run-id") .build()) ``` ## Stop a test suite run To stop a test suite run that is still in progress, you can call the `StopSuiteRun` API operation. After you call the `StopSuiteRun` operation, the service starts the cleanup process. While the service runs the cleanup process, the test suite run status updates to `Stopping`. The cleanup process can take several minutes. Once the process is complete, the test suite run status updates to `Stopped`. After a test run has completely stopped, you n start another test suite run. You can periodically check the suite run status using the `GetSuiteRun` API operation, as shown in the previous section. SDK example: ``` // Using the SDK, call the StopSuiteRun API. response = iotDeviceAdvisorClient.StopSuiteRun( StopSuiteRun.builder() .suiteDefinitionId("your-suite-definition-id") .suiteRunId("your-suite-run-id") .build()) ``` ## Get a qualification report for a successful qualification test suite run If you run a qualification test suite that completes successfully, you can retrieve a qualification report with the `GetSuiteRunReport` API operation. You use this qualification report to qualify your device with the AWS IoT Core qualification program. To determine whether your test suite is a qualification test suite, check whether the `intendedForQualification` parameter is set to `true`. After you call the `GetSuiteRunReport` API operation, you can download the report from the returned URL for up to 90 seconds. If more than 90 seconds elapse from the previous time you called the `GetSuiteRunReport` operation, call the operation again to retrieve a new, valid URL. SDK example: ``` // Using the SDK, call the getSuiteRunReport API. response = iotDeviceAdvisorClient.getSuiteRunReport( GetSuiteRunReportRequest.builder() .suiteDefinitionId("your-suite-definition-id") .suiteRunId("your-suite-run-id") .build() ) ``` # Device Advisor detailed console workflow In this tutorial, you'll create a custom test suite and run tests against the device you want to test in the console. After the tests are complete, you can view the test results and detailed logs. **Topics** + [Prerequisites](#da-detailed-prereqs) + [Create a test suite definition](#device-advisor-console-create-suite) + [Start a test suite run](#device-advisor-console-run-test-suite) + [Stop a test suite run (optional)](#device-advisor-stop-test-run) + [View test suite run details and logs](#device-advisor-console-view-logs) + [Download an AWS IoT qualification report](#device-advisor-console-qualification-report) ## Prerequisites To complete this tutorial, you need to [ create a thing and certificate](https://docs.aws.amazon.com/iot/latest/developerguide/device-advisor-setting-up.html#da-create-thing-certificate). ## Create a test suite definition Create a test suite suite so that you can run it for your devices and perform verification. 1. In the [AWS IoT console](https://console.aws.amazon.com//iot), in the navigation pane, expand **Test**, **Device Advisor** and then choose **Test suites**. ![\[The Device Advisor interface with options to create test suites for qualifying devices, running long duration tests, and custom test suites.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/da-testsuite.png) Choose **Create Test Suite**. 1. Select either `Use the AWS Qualification test suite` or `Create a new test suite`. For protocol, choose either **MQTT 3.1.1 ** or **MQTT 5**. ![\["Create test suite" with options to choose the test suite type (AWS IoT Core qualification, Long duration, or Custom) and protocol (MQTT 3.1.1 or MQTT 5).\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/da-create-test-suite.png) Select `Use the AWS Qualification test suite` to qualify and list your device to the AWS Partner Device Catalog. By choosing this option, test cases required for qualification of your device to the AWS IoT Core qualification program are pre-selected. Test groups and test cases can't be added or removed. You will still need to configure the test suite properties. Select `Create a new test suite` to create and configure a custom test suite. We recommend starting with this option for initial testing and troubleshooting. A custom test suite must have at least one test group, and each test group must have at least one test case. For the purpose of this tutorial, we'll select this option and choose **Next**. ![\[Configure test suite page that shows steps to create a test suite with test groups and cases for testing IoT devices.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/da-configure-test-suite.png) 1. Choose **Test suite properties**. You must create the test suite properties when you create your test suite. ![\[The "Configure test suite" interface that shows options to create test groups and add test cases for testing IoT device functionality.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/da-test-suite-properties.png) Under **Test suite properties**, fill out the following: + **Test suite name**: You can create the suite with a custom name. + **Timeout** (optional): The timeout in seconds for each test case in the current test suite. If you don't specify a timeout value, the default value is used. + **Tags** (optional): Add tags to the test suite. ![\[Window titled "Test suite properties" showing fields to specify a test suite name, timeout, and custom tags for a Device Advisor demo suite.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/da-test-suite-properties-1.png) When you've finished, choose **Update properties**. 1. To modify the group level configuration, under `Test group 1`, choose **Edit**. Then, enter a **Name** to give the group a custom name. Optionally, you can also enter a **Timeout** value in seconds under the selected test group. If you don't specify a timeout value, the default value is used. ![\[The "Configure test suite" interface for creating test groups and cases to validate IoT device functionality.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/da-edit-test-group.png) Choose **Done**. 1. Drag one of the available test cases from **Test cases** into the test group. ![\[The configuration interface for creating a test suite in Device Advisor, with options to add test groups and test cases for testing IoT devices.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/da-configure-test-suite-step5.png) 1. To modify the test case level configuration for the test case that you added to your test group, choose **Edit**. Then, enter a **Name** to give the group a custom name. Optionally, you can also enter a **Timeout** value in seconds under the selected test group. If you don't specify a timeout value, the default value is used. ![\[Test suite configuration interface with options to configure test groups, test cases, timeout settings, and starting points for the test suite execution.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/da-edit-test-case.png) Choose **Done**. **Note** To add more test groups to the test suite, choose **Add test group**. Follow the preceding steps to create and configure more test groups, or to add more test cases to one or more test groups. Test groups and test cases can be reordered by choosing and dragging a test case to the desired position. Device Advisor runs tests in the order in which you define the test groups and test cases. 1. Choose **Next**. 1. In **Step 3**, configure a device role which Device Advisor will use to perform AWS IoT MQTT actions on behalf of your test device. If you selected **MQTT Connect** test case only in **Step 2**, the **Connect** action will be checked automatically since that permission is required on device role to run this test suite. If you selected other test cases, the corresponding required actions will be checked. Ensure that the resource values values for each of the actions is provided. For example, for the **Connect** action, provide the client id that your device will be connecting to the Device Advisor endpoint with. You can provide multiple values by using commas to separate the values, and you can provide prefix values using a wildcard (\$1) character as well. For example, to provide permission to publish on any topic beginning with `MyTopic`, you can provide “`MyTopic*`” as the resource value. ![\[The "Select a device role" step in Device Advisor for creating a test suite, with options to create a new role or select an existing role, and fields to specify role name, permissions, and resource details.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/da-connect-role.png) If you already created a device role previously and would like to use that role, select **Select an existing role** and choose your device role under **Select role**. ![\[The page for selecting a device role for Device Advisor testing, with options to create a new role or select an existing one.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/da-existing-role.png) Configure your device role using one of the two provided options and choose **Next**. 1. In **Step 4**, make sure the configuration provided in each of the steps is accurate. To edit configuration provided for a particular step, choose **Edit** for the corresponding step. After you verify the configuration, choose **Create test suite**. The test suite should be created successfully and you'll be redirected to the **Test suites** page where you can view all the test suite that have been created. If the test suite creation failed, make sure the test suite, test groups, test cases, and device role have been configured according to the previous instructions. ## Start a test suite run 1. In the [AWS IoT console](https://console.aws.amazon.com//iot), in the navigation pane, expand **Test**, **Device Advisor**, and then choose **Test suites**. 1. Choose the test suite for which you'd like to view the test suite details. ![\[The console that shows a single test suite named "Device advisor demo suite" created on May 11, 2021.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/da-test-suites.png) The test suite detail page displays all of the information related to the test suite. 1. Choose **Actions**, then **Run test suite**. ![\[The demo suite page with a "Run test suite" button and an empty activity log showing no previous test suite runs.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/da-run-test-suites.png) 1. Under **Run configuration**, you'll need to select an AWS IoT thing or certificate to test using Device Advisor. If you don't have any existing things or certificates, first [create AWS IoT Core resources](device-advisor-setting-up.md). In **Test endpoint** section, select the endpoint that best fits your case. If you plan to run multiple test suites simultaneously using the same AWS account in the future, select **Device-level endpoint**. Otherwise, if you plan to only run one test suite at a time, select **Account-level endpoint**. Configure your test device with the selected Device Advisor's test endpoint. After you select a thing or certificate and choose a Device Advisor endpoint, choose **Run test**. ![\[The configuration for running a test suite on AWS IoT Core, allowing you to select test devices (things or certificates), choose a test endpoint (account-level or device-level), and optionally add tags.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/da-choose-thing-certificate.png) 1. Choose **Go to results** on the top banner for viewing the test run details. ![\[Details of a custom test suite titled 'Device Advisor demo suite' in progress with the status 'Pending'.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/da-test-run-results.png) ## Stop a test suite run (optional) 1. In the [AWS IoT console](https://console.aws.amazon.com//iot), in the navigation pane, expand **Test**, **Device Advisor**, and then choose **Test runs and results**. 1. Choose the test suite in progress that you want to stop. ![\[The results of test runs on the Device Advisor console.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/da-test-suite-to-stop.PNG) 1. Choose **Actions**, then **Stop test suite**. ![\[The results of test runs on the Device Advisor console.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/da-stop-test-suite.PNG) 1. The cleanup process will take several minutes to complete. While the cleanup process runs, the test run status will be `STOPPING`. Wait for the cleanup process to complete and for the test suite status to change to the `STOPPED` status before starting a new suite run. ![\[The stopped results of test runs on the Device Advisor console.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/da-stopped-test-suite.PNG) ## View test suite run details and logs 1. In the [AWS IoT console](https://console.aws.amazon.com//iot), in the navigation pane, expand **Test**, **Device Advisor** and then choose **Test runs and results**. This page displays: + Number of IoT things + Number of IoT certificates + Number of test suites currently running + All the test suite runs that have been created 1. Choose the test suite for which you'd like to view the run details and logs. ![\[A Test runs and results section that displays details of a test suite named "Device Advisor demo suite" that is currently in progress.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/da-test-suite-run.png) The run summary page displays the status of the current test suite run. This page automatically refreshes every 10 seconds. We recommend that you have a mechanism built for your device to try connecting to our test endpoint every five seconds for one to two minutes. Then you can run multiple test cases in sequence in an automated manner. ![\[The test case log that shows a successful MQTT Connect test with no system message displayed.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/da-run-summary.png) 1. To access the CloudWatch logs for the test suite run, choose **Test suite log**. To access CloudWatch logs for any test case, choose **Test case log**. 1. Based on your test results, [troubleshoot](https://docs.aws.amazon.com/iot/latest/developerguide/iot_troubleshooting.html#device-advisor-troubleshooting) your device until all tests pass. ## Download an AWS IoT qualification report If you chose the **Use the AWS IoT Qualification test suite** option while creating a test suite and were able to run a qualification test suite, you can download a qualification report by choosing **Download qualification report** in the test run summary page. ![\[Qualification Program test results that show passed tests for MQTT, TLS, and other components.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/da-qualification-report.png) # Long duration tests console workflow This tutorial helps you get started with the Long duration tests on Device Advisor using the console. To complete the tutorial, follow the steps at [Setting up](device-advisor-setting-up.md). 1. In the [AWS IoT console](https://console.aws.amazon.com/iot) navigation pane, expand **Test**, then **Device Advisor**, then **Test suites**. On the page, select **Create long duration test suite**. ![\[The Create long duration test suite section of the Device Advisor console.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/create-ld-ts.png) 1. On the **Create test suite** page, select **Long duration test suite** and choose **Next**. For protocol, choose either **MQTT 3.1.1** or **MQTT 5**. ![\[The Create test suite step of the Device Advisor console.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/choose-ld-ts.png) 1. Do the following on the **Configure test suite** page: 1. Update the **Test suite name** field. 1. Update the **Test group name** field. 1. Choose the **Device operations** the device can perform. This will select the tests to run. 1. Select the **Settings** option. ![\[The Create test suite step of the Device Advisor console.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/configure-ld-ts.png) 1. (Optional) Input the maximum amount of time Device Advisor must wait for the basic tests to complete. Select **Save**. ![\[The "Timeout-optional" box for "Basic tests" of the Device Advisor console.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/timeout-ld-ts.png) 1. Do the following in the **Advanced tests** and **Additional settings** sections. 1. Select or deselect the **Advanced tests** you want to run as part of this test. 1. **Edit** the configurations for the tests when applicable. 1. Configure the **Additional execution time** under the **Additional settings** section. 1. Choose **Next** to do the next step. ![\[The Device Advisor interface that allows you to configure and run tests on IoT devices.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/additional-ld-ts.png) 1. In this step, **Create a new role** or **Select an existing role**. See [Create an IAM role to use as your device role](device-advisor-setting-up.md#da-iam-role) for details. ![\[The device role step where you can create a new role or select an existing role for the device being tested. The role grants permissions for Device Advisor to perform MQTT actions like Connect, Publish, and Subscribe on behalf of the test device.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/devicerole-ld-ts.png) 1. Review all the configurations created until this step and select **Create test suite**. ![\[The "Review" page where you can review all the details of the Device Advisor configuration.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/finalconfigure1-ld-ts.png) ![\[The configuration page where you can view all the details for Device Advisor.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/finalconfigure2-ld-ts.png) 1. The created test suite is under the **Test suites** section. Select the suite to view details. ![\[A new test suite named "Long Duration Demo" has been successfully created in the Device Advisor.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/finalts-ld-ts.png) 1. To run the created test suite, select **Actions** then **Run test suite**. ![\[The Actions dropdown menu of the new test suite named "Long Duration Demo" in the Device Advisor interface.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/runts-ld-ts.png) 1. Choose the configuration options in the **Run configuration** page. 1. Select the **Things** or **Certificate** to run the test on. 1. Select either the **Account-level endpoint** or **Device-level endpoint**. 1. Choose **Run test** to run the test. ![\[The Run configuration page in the Device Advisor interface. The page shows Select test devices, Things, Test endpoint, and Tags.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/runconfiguration-ld-ts.png) 1. To view the results of the test suite run, select **Test runs and results** in the left navigation pane. Choose the test suite that ran to view the details of the results. ![\[The "Long Duration Demo" test case on the Test runs and results page.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/results-ld-ts.png) 1. The previous step brings up the test summary page. All the details of the test run are displayed in this page. When the console prompts to start the device connection, connect your device to the provided endpoint. The progress of the tests is seen on this page. ![\[The summary page of the "Long Duration Demo" test that you created.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/summary-ld-ts.png) 1. The Long duration test provides an additional **Test log summary** on the side panel which displays all the important events occurring between the device and the broker in near real time. To view more in-depth detailed logs, click on **Test case log**. ![\[The Test log summary section on the page of the "Long Duration Demo" test.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/log-ld-ts.png) # Device Advisor VPC endpoints (AWS PrivateLink) You can establish a private connection between your VPC and the AWS IoT Core Device Advisor test endpoint (data plane) by creating an *interface VPC endpoint*. You can use this endpoint to validate AWS IoT devices for reliable and secure connectivity with AWS IoT Core before deploying devices to production. Device Advisor's pre-built tests helps you validate your device software against best practices for usage of [TLS](https://docs.aws.amazon.com/iot/latest/developerguide/protocols.html), [MQTT](https://docs.aws.amazon.com/iot/latest/developerguide/protocols.html), [Device Shadow](https://docs.aws.amazon.com/iot/latest/developerguide/iot-device-shadows.html), and [AWS IoT Jobs](https://docs.aws.amazon.com/iot/latest/developerguide/iot-jobs.html). [AWS PrivateLink](http://aws.amazon.com/privatelink) powers the interface endpoints used with your IoT devices. This service helps you access the AWS IoT Core Device Advisor test endpoint privately without an internet gateway, NAT device, VPN connection, or Direct Connect connection. Instances in your VPC that send TCP and MQTT packets don't need public IP addresses to communicate with AWS IoT Core Device Advisor test endpoints. Traffic between your VPC and AWS IoT Core Device Advisor doesn't leave AWS Cloud. Any TLS and MQTT communication between IoT devices and Device Advisor test cases stay within the resources in your AWS account. Each interface endpoint is represented by one or more [elastic network interfaces](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html) in your subnets. To learn more about using interface VPC endpoints, see [Interface VPC endpoints (AWS PrivateLink)](https://docs.aws.amazon.com/vpc/latest/userguide/vpce-interface.html) in the *Amazon VPC User Guide*. ## Considerations for AWS IoT Core Device Advisor VPC endpoints Review the [ interface endpoint properties and limitations](https://docs.aws.amazon.com/vpc/latest/userguide/vpce-interface.html#vpce-interface-limitations) in the *Amazon VPC User Guide* before setting up interface VPC endpoints. Consider the following before you continue: + AWS IoT Core Device Advisor currently supports making calls to Device Advisor test endpoint (data plane) from your VPC. A message broker uses data plane communications to send and receive data. It does this with the help of TLS and MQTT packets. VPC endpoints for AWS IoT Core Device Advisor connect your AWS IoT device to Device Advisor test endpoints. [Control plane API actions](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/iotdeviceadvisor/index.html) aren't used by this VPC endpoint. To create or run a test suite or other control plane APIs, use the console, an AWS SDK, or AWS Command Line Interface over the public internet. + The following AWS Regions support VPC endpoints for AWS IoT Core Device Advisor: + US East (N. Virginia) + US West (Oregon) + Asia Pacific (Tokyo) + Europe (Ireland) + Device Advisor supports MQTT with X.509 client certificates and RSA server certificates. + [VPC endpoint policies](https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpoints-access.html) aren't supported at this time. + Check VPC endpoint [prerequisites ](https://docs.aws.amazon.com/vpc/latest/privatelink/create-interface-endpoint.html#prerequisites-interface-endpoints) for instructions on how to [create resources ](https://docs.aws.amazon.com/vpc/latest/privatelink/create-interface-endpoint.html#create-interface-endpoint-aws) that connect VPC endpoints. You must create a VPC and private subnets to use AWS IoT Core Device Advisor VPC endpoints. + There are quotas on your AWS PrivateLink resources. For more information, see [AWS PrivateLink quotas](https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-limits-endpoints.html). + VPC endpoints support only IPv4 traffic. ## Create an interface VPC endpoint for AWS IoT Core Device Advisor To get started with VPC endpoints, [ create an interface VPC endpoint](https://docs.aws.amazon.com/vpc/latest/privatelink/create-interface-endpoint.html). Next, select AWS IoT Core Device Advisor as the AWS service. If you are using the AWS CLI, call [describe-vpc-endpoint-services](https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-vpc-endpoint-services.html) to confirm that AWS IoT Core Device Advisor is present in an Availability Zone in your AWS Region. Confirm that the security group attached to the endpoint allows [TCP protocol communication](https://docs.aws.amazon.com/iot/latest/developerguide/protocols.html) for MQTT and TLS traffic. For example, in the US East (N. Virginia) Region, use the following command: ``` aws ec2 describe-vpc-endpoint-services --service-name com.amazonaws.us-east-1.deviceadvisor.iot ``` You can create a VPC endpoint for AWS IoT Core using the following service name: + com.amazonaws.*region*.deviceadvisor.iot By default, private DNS is turned on for the endpoint. This ensures that use of the default test endpoint stays within your private subnets. To get your account or device level endpoint, use the console, AWS CLI or an AWS SDK. For example, if you run [ get-endpoint](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/iotdeviceadvisor/get-endpoint.html) within a public subnet or on the public internet, you can get your endpoint and use it to connect to Device Advisor. For more information, see [Accessing a service through an interface endpoint](https://docs.aws.amazon.com/vpc/latest/userguide/vpce-interface.html#access-service-though-endpoint) in the *Amazon VPC User Guide*. To connect MQTT clients to the VPC endpoint interfaces, the AWS PrivateLink service creates DNS records in a private hosted zone attached to your VPC. These DNS records direct the AWS IoT device’s requests to the VPC endpoint. ## Controlling access to AWS IoT Core Device Advisor over VPC endpoints You can restrict device access to AWS IoT Core Device Advisor and allow access only through VPC endpoints by using VPC [condition context keys](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_condition-keys.html). AWS IoT Core supports the following VPC related context keys: + [SourceVpc](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_condition-keys.html#condition-keys-sourcevpc) + [SourceVpce](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_condition-keys.html#condition-keys-sourcevpce) + [VPCSourcelp](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_condition-keys.html#condition-keys-vpcsourceip) **Note** AWS IoT Core Device Advisor doesn't support [ VPC endpoint policies](https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpoints-access.html#vpc-endpoint-policies) at this time. The following policy grants permission to connect to AWS IoT Core Device Advisor using a client ID that matches the thing name. It also publishes to any topic prefixed by the thing name. The policy is conditional on the device connecting to a VPC endpoint with a particular VPC endpoint ID. This policy denies connection attempts to your public AWS IoT Core Device Advisor test endpoint. **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "iot:Connect" ], "Resource": [ "arn:aws:iot:us-east-1:123456789012:client/${iot:Connection.Thing.ThingName}" ], "Condition": { "StringEquals": { "aws:SourceVpce": "vpce-1a2b3c4d" } } }, { "Effect": "Allow", "Action": [ "iot:Publish" ], "Resource": [ "arn:aws:iot:us-east-1:123456789012:topic/${iot:Connection.Thing.ThingName}/*" ] } ] } ``` # Device Advisor test cases Device Advisor provides prebuilt tests in six categories. + [TLS](device-advisor-tests-tls.md) + [MQTT](device-advisor-tests-mqtt.md) + [Shadow](device-advisor-tests-shadow.md) + [Job execution](device-advisor-tests-job-execution.md) + [Permissions and policies](device-advisor-tests-permissions-policies.md) + [Long duration tests](device-advisor-tests-long-duration.md) ## Device Advisor test cases to qualify for the AWS Device Qualification Program. Your device must pass the following tests to qualify according to the [AWS Device Qualification Program](https://aws.amazon.com/partners/programs/dqp/). **Note** This is a revised list of the qualification tests. + [TLS Connect](device-advisor-tests-tls.md#TLS_Connect) ("TLS Connect")​ + [TLS Incorrect Subject Name Server Cert](device-advisor-tests-tls.md#TLS_Incorrect_Subject_Name) ("Incorrect Subject Common Name (CN) / Subject Alternative Name (SAN)") + [TLS Unsecure Server Cert](device-advisor-tests-tls.md#TLS_Unsecure_Server_Cert) ("Not Signed By Recognized CA")​ + [TLS Device Support for AWS IoT Cipher Suites](device-advisor-tests-tls.md#TLS_DeviceSupport_For_IOT) ("TLS Device Support for AWS IoT recommended Cipher Suites") + [TLS Receive Maximum Size Fragments](device-advisor-tests-tls.md#TLS_MaximumSize)("TLS Receive Maximum Size Fragments") + [TLS Expired Server Cert](device-advisor-tests-tls.md#TLS_Expired_Server_Cert)("Expired server certificate") + [TLS Large Size Server Cert](device-advisor-tests-tls.md#TLS_LargeServerCert)("TLS large Size Server Certificate") + [MQTT Connect](device-advisor-tests-mqtt.md#MQTT_Connect) ("Device send CONNECT to AWS IoT Core (Happy case)")​ + [MQTT Subscribe](device-advisor-tests-mqtt.md#MQTT_Subscribe) ("Can Subscribe (Happy Case)")​ + [MQTT Publish](device-advisor-tests-mqtt.md#MQTT_Publish) ("QoS0 (Happy Case)")​ + [MQTT Connect Jitter Retries](device-advisor-tests-mqtt.md#MQTT_ConnectJitterBackoff)("Device connect retries with jitter backoff - No CONNACK response")​ # TLS Use these tests to determine if the transport layer security protocol (TLS) between your devices and AWS IoT is secure. **Note** Device Advisor now supports TLS 1.3. ## Happy Path **TLS Connect** Validates if the device under test can complete the TLS handshake to AWS IoT. This test doesn't validate the MQTT implementation of the client device. **Example API test case definition:** `EXECUTION_TIMEOUT` has a default value of 5 minutes. For best results, we recommend a timeout value of 2 minutes. ``` "tests":[ { "name":"my_tls_connect_test", "configuration": { // optional: "EXECUTION_TIMEOUT":"300", //in seconds }, "test":{ "id":"TLS_Connect", "version":"0.0.0" } } ] ``` **Example Test case outputs:** + **Pass** — The device under test completed TLS handshake with AWS IoT. + **Pass with warnings** — The device under test completed TLS handshake with AWS IoT, but there were TLS warning messages from the device or AWS IoT. + **Fail** — The device under test failed to complete TLS handshake with AWS IoT due to handshake error. **TLS Receive Maximum Size Fragments** This test case validates that your device can receive and process TLS maximum size fragments. Your test device must subscribe to a pre-configured topic with QoS 1 to receive a large payload. You can customize the payload with the configuration `${payload}`. **Example API test case definition:** `EXECUTION_TIMEOUT` has a default value of 5 minutes. For best results, we recommend a timeout value of 2 minutes. ``` "tests":[ { "name":"TLS Receive Maximum Size Fragments", "configuration": { // optional: "EXECUTION_TIMEOUT":"300", //in seconds "PAYLOAD_FORMAT":"{"message":"${payload}"}", // A string with a placeholder ${payload}, or leave it empty to receive a plain string. "TRIGGER_TOPIC": "test_1" // A topic to which a device will subscribe, and to which a test case will publish a large payload. }, "test":{ "id":"TLS_Receive_Maximum_Size_Fragments", "version":"0.0.0" } } ] ``` ## Cipher Suites **TLS Device Support for AWS IoT recommended Cipher Suites** Validates that the cipher suites in the TLS Client Hello message from the device under test contains the recommended [AWS IoT cipher suites](transport-security.md). It provides more insights into cipher suites supported by the device. **Example API test case definition:** `EXECUTION_TIMEOUT` has a default value of 5 minutes. We recommend a timeout value of 2 minutes. ``` "tests":[ { "name":"my_tls_support_aws_iot_cipher_suites_test", "configuration": { // optional: "EXECUTION_TIMEOUT":"300", // in seconds }, "test":{ "id":"TLS_Support_AWS_IoT_Cipher_Suites", "version":"0.0.0" } } ] ``` **Example Test case outputs:** + **Pass** — The device under test cipher suites contain at least one of the recommended AWS IoT cipher suite and don't contain any unsupported cipher suites. + **Pass with warnings** — The device cipher suites contain at least one AWS IoT cipher suite but: 1. It doesn't contain any of the recommended cipher suites 1. It contains cipher suites that aren't supported by AWS IoT. We suggest that you verify that any unsupported cipher suites are safe. + **Fail** — The device under test cipher suites doesn't contain any of the AWS IoT supported cipher suites. ## Larger Size Server Certificate **TLS large Size Server Certificate** Validates at your device can complete the TLS handshake with AWS IoT when it receives and processes a larger size server certificate. The size of the server certificate (in bytes) used by this test is larger than what is currently used in the **TLS Connect** test case and IoT Core by 20 During this test case, AWS IoT tests your device’s buffer space for TLS If the buffer space is large enough, the TLS handshake ompletes without errors. This test esn't validate the MQTT implementation of the device. The test case ds after the TLS handshake process completes. **Example API test case definition:** `EXECUTION_TIMEOUT` has a default value of 5 minutes. For best results, we recommend a timeout value of 2 minutes. If this test case fails but the **TLS Connect** test case passes, we recommend you increase your device’s buffer space limit for TLS Increasing the buffer space limit sures that your device can process a larger size server certificate in case the size increases. ``` "tests":[ { "name":"my_tls_large_size_server_cert_test", "configuration": { // optional: "EXECUTION_TIMEOUT":"300", // in seconds }, "test":{ "id":"TLS_Large_Size_Server_Cert", "version":"0.0.0" } } ] ``` **Example Test case outputs:** + **Pass** — The device under test completed the TLS handshake with AWS IoT. + **Pass with warnings** — The device under test completed the TLS handshake with AWS IoT, but there are TLS warning messages either from the device or AWS IoT. + **Fail** — The device under test failed to complete the TLS handshake with AWS IoT because of an error during the handshake process. ## TLS Unsecure Server Cert **Not Signed By Recognized CA** Validates that the device under test closes the connection if it's presented with a server certificate without a valid signature from the ATS CA. A device should only connect to an endpoint that presents a valid certificate. **Example API test case definition:** `EXECUTION_TIMEOUT` has a default value of 5 minutes. We recommend a timeout value of 2 minutes. ``` "tests":[ { "name":"my_tls_unsecure_server_cert_test", "configuration": { // optional: "EXECUTION_TIMEOUT":"300", //in seconds }, "test":{ "id":"TLS_Unsecure_Server_Cert", "version":"0.0.0" } } ] ``` **Example Test case outputs:** + **Pass** — The device under test closed the connection. + **Fail** — The device under test completed TLS handshake with AWS IoT. **TLS Incorrect Subject Name Server Cert / Incorrect Subject Common Name (CN) / Subject Alternative Name (SAN)** Validates that the device under test closes the connection if it's presented with a server certificate for a domain name that is different than the one requested. **Example API test case definition:** `EXECUTION_TIMEOUT` has a default value of 5 minutes. We recommend a timeout value of 2 minutes. ``` "tests":[ { "name":"my_tls_incorrect_subject_name_cert_test", "configuration": { // optional: "EXECUTION_TIMEOUT":"300", // in seconds }, "test":{ "id":"TLS_Incorrect_Subject_Name_Server_Cert", "version":"0.0.0" } } ] ``` **Example Test case outputs:** + **Pass** — The device under test closed the connection. + **Fail** — The device under test completed the TLS handshake with AWS IoT. ## TLS Expired Server Certificate **Expired server certificate** Validates that the device under test closes the connection if it's presented with an expired server certificate. **Example API test case definition:** `EXECUTION_TIMEOUT` has a default value of 5 minutes. We recommend a timeout value of 2 minutes. ``` "tests":[ { "name":"my_tls_expired_cert_test", "configuration": { // optional: "EXECUTION_TIMEOUT":"300", //in seconds }, "test":{ "id":"TLS_Expired_Server_Cert", "version":"0.0.0" } } ] ``` **Example Test case outputs:** + **Pass** — The device under test refuses to complete the TLS handshake with AWS IoT. The device sends a TLS alert message before it closes the connection. + **Pass with warnings** — The device under test refuses to complete the TLS handshake with AWS IoT. However, it doesn’t send a TLS alert message before it closes the connection. + **Fail** — The device under test completes the TLS handshake with AWS IoT. # MQTT ## CONNECT, DISCONNECT, and RECONNECT **"Device send CONNECT to AWS IoT Core (Happy case)"** Validates that the device under test sends a CONNECT request. *API test case definition:* `EXECUTION_TIMEOUT` has a default value of 5 minutes. We recommend a timeout value of 2 minutes. ``` "tests":[ { "name":"my_mqtt_connect_test", "configuration": { // optional: "EXECUTION_TIMEOUT":"300", // in seconds }, "test":{ "id":"MQTT_Connect", "version":"0.0.0" } } ] ``` **"Device can return PUBACK to an arbitrary topic for QoS1"** This test case will check if the device (client) can return a PUBACK message if it received a publish message from the broker after subscribing to a topic with QoS1. The payload content and the payload size are configurable for this test case. If the payload size is configured, Device Advisor will overwrite the value for the payload content, and send a predefined payload to the device with the desired size. The payload size is a value between 0 to 128 and cannot exceed 128 KB. AWS IoT Core rejects publish and connect requests larger than 128 KB, as seen in the [AWS IoT Core message broker and protocol limits and quotas](https://docs.aws.amazon.com/general/latest/gr/iot-core.html#message-broker-limits) page. *API test case definition:* `EXECUTION_TIMEOUT` has a default value of 5 minutes. We recommend a timeout value of 2 minutes. `PAYLOAD_SIZE` can be configured to a value between 0 and 128 kilobytes. Defining a payload size overrides the payload content as Device Advisor will be sending a pre-defined payload with the given size back to the device. ``` "tests":[ { "name":"my_mqtt_client_puback_qos1", "configuration": { // optional:"TRIGGER_TOPIC": "myTopic", "EXECUTION_TIMEOUT":"300", // in seconds "PAYLOAD_FOR_PUBLISH_VALIDATION":"custom payload", "PAYLOAD_SIZE":"100" // in kilobytes }, "test": { "id": "MQTT_Client_Puback_QoS1", "version": "0.0.0" } } ] ``` **"Device connect retries with jitter backoff - No CONNACK response"** Validates that the device under test uses the proper jitter backoff when reconnecting with the broker for at least five times. The broker logs the timestamp of the device under test's CONNECT request, performs packet validation, pauses without sending a CONNACK to the device under test, and waits for the device under test to resend the request. The sixth connection attempt is allowed to pass through and CONNACK is allowed to flow back to the device under test. The preceding process is performed again. In total, this test case requires the device to connect at least 12 times in total. The collected timestamps are used to validate that jitter backoff is used by the device under test. If the device under test has a strictly exponential backoff delay, this test case will pass with warnings. We recommend implementation of the [Exponential Backoff And Jitter](https://aws.amazon.com/blogs//architecture/exponential-backoff-and-jitter/) mechanism on the device under test to pass this test case. *API test case definition:* `EXECUTION_TIMEOUT` has a default value of 5 minutes. We recommend a timeout value of 4 minutes. ``` "tests":[ { "name":"my_mqtt_jitter_backoff_retries_test", "configuration": { // optional: "EXECUTION_TIMEOUT":"300", // in seconds }, "test":{ "id":"MQTT_Connect_Jitter_Backoff_Retries", "version":"0.0.0" } } ] ``` **"Device connect retries with exponential backoff - No CONNACK response"** Validates that the device under test uses the proper exponential backoff when reconnecting with the broker for at least five times. The broker logs the timestamp of the device under test's CONNECT request, performs packet validation, pauses without sending a CONNACK to the client device, and waits for the device under test to resend the request. The collected timestamps are used to validate that an exponential backoff is used by the device under test. We recommend implementation of the [Exponential Backoff And Jitter](https://aws.amazon.com/blogs//architecture/exponential-backoff-and-jitter/) mechanism on the device under test to pass this test case. *API test case definition:* `EXECUTION_TIMEOUT` has a default value of 5 minutes. We recommend a timeout value of 4 minutes. ``` "tests":[ { "name":"my_mqtt_exponential_backoff_retries_test", "configuration": { // optional: "EXECUTION_TIMEOUT":"600", // in seconds }, "test":{ "id":"MQTT_Connect_Exponential_Backoff_Retries", "version":"0.0.0" } } ] ``` **"Device re-connect with jitter backoff - After server disconnect"** Validates if a device under test uses necessary jitter and backoff while reconnecting after it's been disconnected from the server. Device Advisor disconnects the device from the server for at least five times and observes the device's behavior for MQTT reconnection. Device Advisor logs the timestamp of the CONNECT request for the device under test, performs packet validation, pauses without sending a CONNACK to the client device, and waits for the device under test to resend the request. The collected timestamps are used to validate that the device under test uses jitter and backoff while reconnecting. If the device under test has a strictly exponential backoff or doesn't implement a proper jitter backoff mechanism, this test case will pass with warnings. If the device under test has implemented either a linear backoff or a constant backoff mechanism, the test will fail. To pass this test case, we recommend implementing the [Exponential Backoff And Jitter](https://aws.amazon.com/blogs//architecture/exponential-backoff-and-jitter/) mechanism on the device under test. *API test case definition:* `EXECUTION_TIMEOUT` has a default value of 5 minutes. We recommend a timeout value of 4 minutes. The number of reconnection attempts to validate for backoff can be changed by specifying the `RECONNECTION_ATTEMPTS`. The number must be between 5 and 10. The default value is 5. ``` "tests":[ { "name":"my_mqtt_reconnect_backoff_retries_on_server_disconnect", "configuration":{ // optional: "EXECUTION_TIMEOUT":"300", // in seconds "RECONNECTION_ATTEMPTS": 5 }, "test":{ "id":"MQTT_Reconnect_Backoff_Retries_On_Server_Disconnect", "version":"0.0.0" } } ] ``` **"Device re-connect with jitter backoff - On unstable connection"** Validates if a device under test uses necessary jitter and backoff while reconnecting on an unstable connection. Device Advisor disconnects the device from the server after five successful connections, and observes the device's behavior for MQTT reconnection. Device Advisor logs the timestamp of the CONNECT request for the device under test, performs packet validation, sends back CONNACK, disconnects, log the timestamp of the disconnection, and waits for the device under test to resend the request. The collected timestamps are used to validate that the device under test uses jitter and backoff while reconnecting after successful but unstable connections. If the device under test has a strictly exponential backoff or doesn't implement a proper jitter backoff mechanism, this test case will pass with warnings. If the device under test has implemented either a linear backoff or a constant backoff mechanism, the test will fail. To pass this test case, we recommend implementing the [Exponential Backoff And Jitter](https://aws.amazon.com/blogs//architecture/exponential-backoff-and-jitter/) mechanism on the device under test. *API test case definition:* `EXECUTION_TIMEOUT` has a default value of 5 minutes. We recommend a timeout value of 4 minutes. The number of reconnection attempts to validate for backoff can be changed by specifying the `RECONNECTION_ATTEMPTS`. The number must be between 5 and 10. The default value is 5. ``` "tests":[ { "name":"my_mqtt_reconnect_backoff_retries_on_unstable_connection", "configuration":{ // optional: "EXECUTION_TIMEOUT":"300", // in seconds "RECONNECTION_ATTEMPTS": 5 }, "test":{ "id":"MQTT_Reconnect_Backoff_Retries_On_Unstable_Connection", "version":"0.0.0" } } ] ``` ## Publish **"QoS0 (Happy Case)"** Validates that the device under test publishes a message with QoS0 or QoS1. You can also validate the topic of the message and payload by specifying the topic value and payload in the test settings. `EXECUTION_TIMEOUT` has a default value of 5 minutes. We recommend a timeout value of 2 minutes. ``` "tests":[ { "name":"my_mqtt_publish_test", "configuration":{ // optional: "EXECUTION_TIMEOUT":"300", // in seconds "TOPIC_FOR_PUBLISH_VALIDATION": "my_TOPIC_FOR_PUBLISH_VALIDATION", "PAYLOAD_FOR_PUBLISH_VALIDATION": "my_PAYLOAD_FOR_PUBLISH_VALIDATION", }, "test":{ "id":"MQTT_Publish", "version":"0.0.0" } } ] ``` **"QoS1 publish retry - No PUBACK"** Validates that the device under test republishes a message sent with QoS1, if the broker doesn't send PUBACK. You can also validate the topic of the message by specifying this topic in the test settings. The client device must not disconnect before republishing the message. This test also validates that the republished message has the same packet identifier as the original. During the test execution, if the device loses connection and reconnects, the test case will reset without failing and the device has to perform the test case steps again. *API test case definition:* `EXECUTION_TIMEOUT` has a default value of 5 minutes. It is recommended for at least 4 minutes. ``` "tests":[ { "name":"my_mqtt_publish_retry_test", "configuration":{ // optional: "EXECUTION_TIMEOUT":"300", // in seconds "TOPIC_FOR_PUBLISH_VALIDATION": "my_TOPIC_FOR_PUBLISH_VALIDATION", "PAYLOAD_FOR_PUBLISH_VALIDATION": "my_PAYLOAD_FOR_PUBLISH_VALIDATION", }, "test":{ "id":"MQTT_Publish_Retry_No_Puback", "version":"0.0.0" } } ] ``` **"Publish Retained messages"** Validates that the device under test publishes a message with `retainFlag` set to true. You can validate the topic and payload of the message by setting the topic value and payload in the test settings. If the `retainFlag` sent within the PUBLISH packet is not set to true, the test case will fail. *API test case definition:* `EXECUTION_TIMEOUT` has a default value of 5 minutes. We recommend a timeout value of 2 minutes. To run this test case, add the `iot:RetainPublish` action in your [ device role](https://docs.aws.amazon.com/iot/latest/developerguide/device-advisor-setting-up.html#da-iam-role). ``` "tests":[ { "name":"my_mqtt_publish_retained_messages_test", "configuration":{ // optional: "EXECUTION_TIMEOUT":"300", // in seconds "TOPIC_FOR_PUBLISH_RETAINED_VALIDATION": "my_TOPIC_FOR_PUBLISH_RETAINED_VALIDATION", "PAYLOAD_FOR_PUBLISH_RETAINED_VALIDATION": "my_PAYLOAD_FOR_PUBLISH_RETAINED_VALIDATION", }, "test":{ "id":"MQTT_Publish_Retained_Messages", "version":"0.0.0" } } ] ``` **"Publish with User Property"** Validates that the device under test publishes a message with the correct user property. You can validate the user property by setting the name-value pair in the test settings. If the user property is not provided or doesn't match, the test case fails. *API test case definition:* This is a MQTT5 only test case. `EXECUTION_TIMEOUT` has a default value of 5 minutes. We recommend a timeout value of 2 minutes. ``` "tests":[ { "name":"my_mqtt_user_property_test", "test":{ "USER_PROPERTIES": [ {"name": "name1", "value":"value1"}, {"name": "name2", "value":"value2"} ], "EXECUTION_TIMEOUT":"300", // in seconds }, "test":{ "id":"MQTT_Publish_User_Property", "version":"0.0.0" } } ] ``` ## Subscribe **"Can Subscribe (Happy Case)"** Validates that the device under test subscribes to MQTT topics. You can also validate the topic that the device under test subscribes to by specifying this topic in the test settings. *API test case definition:* `EXECUTION_TIMEOUT` has a default value of 5 minutes. We recommend a timeout value of 2 minutes. ``` "tests":[ { "name":"my_mqtt_subscribe_test", "configuration":{ // optional: "EXECUTION_TIMEOUT":"300", // in seconds "TOPIC_LIST_FOR_SUBSCRIPTION_VALIDATION":["my_TOPIC_FOR_PUBLISH_VALIDATION_a","my_TOPIC_FOR_PUBLISH_VALIDATION_b"] }, "test":{ "id":"MQTT_Subscribe", "version":"0.0.0" } } ] ``` **"Subscribe Retry - No SUBACK"** Validates that the device under test retries a failed subscription to MQTT topics. The server then waits and doesn't send a SUBACK. If the client device doesn't retry the subscription, the test fails. The client device must retry the failed subscription with the same packet Id. You can also validate the topic that the device under test subscribes to by specifying this topic in the test settings. During the test execution, if the device loses connection and reconnects, the test case will reset without failing and the device has to perform the test case steps again. *API test case definition:* `EXECUTION_TIMEOUT` has a default value of 5 minutes. We recommend a timeout value of 4 minutes. ``` "tests":[ { "name":"my_mqtt_subscribe_retry_test", "configuration":{ "EXECUTION_TIMEOUT":"300", // in seconds // optional: "TOPIC_LIST_FOR_SUBSCRIPTION_VALIDATION":["myTOPIC_FOR_PUBLISH_VALIDATION_a","my_TOPIC_FOR_PUBLISH_VALIDATION_b"] }, "test":{ "id":"MQTT_Subscribe_Retry_No_Suback", "version":"0.0.0" } } ] ``` ## Keep-Alive **"Mqtt No Ack PingResp"** This test case validates if the device under test disconnects when it doesn't receive a ping response. As part of this test case, Device Advisor blocks responses sent from AWS IoT Core for publish, subscribe, and ping requests. It also validates if the device under test disconnects the MQTT connection. *API test case definition:* `EXECUTION_TIMEOUT` has a default value of 5 minutes. We recommend a timeout greater than 1.5 times the `keepAliveTime` value. The maximum `keepAliveTime` must be no greater than 230 seconds for this test. ``` "tests":[ { "name":"Mqtt No Ack PingResp", "configuration": //optional: "EXECUTION_TIMEOUT":"306", // in seconds }, "test":{ "id":"MQTT_No_Ack_PingResp", "version":"0.0.0" } } ] ``` ## Persistent Session **"Persistent Session (Happy Case)"** This test case validates the device behavior when disconnected from a persistent session. The test case checks if the device can reconnect, resume the subscriptions to its trigger topics without explicitly re-subscribing, receive the stored messages in the topics, and work as expected during a persistent session. When this test case passes, it indicates that the client device is able to maintain a persistent session with the AWS IoT Core broker in an expected manner. For more information on AWS IoT Persistent Sessions, see [ Using MQTT persistent sessions ](https://docs.aws.amazon.com/iot/latest/developerguide/mqtt.html#mqtt-persistent-sessions). In this test case, the client device is expected to CONNECT with the AWS IoT Core with a clean session flag set to false, and then subscribe to a trigger topic. After a successful subscription, the device will be disconnected by AWS IoT Core Device Advisor. While the device is in a disconnected state, a QoS 1 message payload will be stored in that topic. Device Advisor will then allow the client device to re-connect with the test endpoint. At this point, since there is a persistent session, the client device is expected to resume its topic subscriptions without sending any additional SUBSCRIBE packets and receive the QoS 1 message from the broker. After re-connecting, if the client device re-subscribes to its trigger topic again by sending an additional SUBSCRIBE packet and/or if the client fails to receive the stored message from the trigger topic, the test case will fail. *API test case definition:* `EXECUTION_TIMEOUT` has a default value of 5 minutes. We recommend a timeout value of at least 4 minutes. In the first connection, client device needs to explicitly subscribe to a `TRIGGER_TOPIC` which was not subscribed before. To pass the test case, client device must successfully subscribe to `TRIGGER_TOPIC` with a QoS 1. After re-connecting, the client device is expected to understand that there is an active persistent session; so it should accept the stored message sent by the trigger topic and return PUBACK for that specific message. ``` "tests":[ { "name":"my_mqtt_persistent_session_happy_case", "configuration":{ //required: "TRIGGER_TOPIC": "myTrigger/topic", // optional: // if Payload not provided, a string will be stored in the trigger topic to be sent back to the client device "PAYLOAD": "The message which should be received from AWS IoT Broker after re-connecting to a persistent session from the specified trigger topic.", "EXECUTION_TIMEOUT":"300" // in seconds }, "test":{ "id":"MQTT_Persistent_Session_Happy_Case", "version":"0.0.0" } } ] ``` **"Persistent Session - Session Expiry"** This test case helps to validate device behavior when a disconnected device reconnects to an expired persistent session. After the session expires, we expect the device to resubscribe to the topics previously subscribed to by explicitly sending a new SUBSCRIBE packet. During the first connection, we expect the test device to CONNECT with the AWS IoT broker, as its `CleanSession` flag is set to false to initiate a persistent session. The device should then subscribe to a trigger topic. Then the device is disconnected by AWS IoT Core Device Advisor, after a successful subscription and initiation of a persistent session. After the disconnection, AWS IoT Core Device Advisor allows the test device to re-connect back with the test endpoint. At this point, when the test device sends another CONNECT packet, AWS IoT Core Device Advisor sends back a CONNACK packet that indicates that the persistent session is expired. The test device needs to interpret this packet properly, and it is expected to re-subscribe to the same trigger topic again as the persistent session is terminated. If the test device does not re-subscribe to its topic trigger again, the test case fails. For the test to pass, the device needs to understand that the persistent session is over, and send back a new SUBSCRIBE packet for the same trigger topic in the second connection. If this test case passes for a test device, it indicates that the device is able to handle re-connection on expiry of persistent session in an expected way. *API test case definition:* `EXECUTION_TIMEOUT` has a default value of 5 minutes. We recommend a timeout value of at least 4 minutes. The test device needs to explicitly subscribe to a `TRIGGER_TOPIC`, to which it was not subscribed before. To pass the test case, the test device must send a CONNECT packet with `CleanSession` flag set to false, and successfully subscribe to a trigger topic with a QoS 1. After a successful connection, AWS IoT Core Device Advisor disconnects the device. After the disconnection, AWS IoT Core Device Advisor allows the device to re-connect back, and the device is expected to re-subscribe to the same `TRIGGER_TOPIC` since AWS IoT Core Device Advisor would have terminated the persistent session. ``` "tests":[ { "name":"my_expired_persistent_session_test", "configuration":{ //required: "TRIGGER_TOPIC": "myTrigger/topic", // optional: "EXECUTION_TIMEOUT":"300" // in seconds }, "test":{ "id":"MQTT_Expired_Persistent_Session", "version":"0.0.0" } } ] ``` # Shadow Use these tests to verify your devices under test use AWS IoT Device Shadow service correctly. See [AWS IoT Device Shadow service](iot-device-shadows.md) for more information. If these test cases are configured in your test suite, then providing a thing is required when starting the suite run. **MQTT over WebSocket** is not supported at this time. ## Publish ***"Device publishes state after it connects (Happy case)"*** Validates if a device can publish its state after it connects to AWS IoT Core *API test case definition:* `EXECUTION_TIMEOUT` has a default value of 5 minutes. We recommend a timeout value of 2 minutes. ``` "tests":[ { "name":"my_shadow_publish_reported_state", "configuration": { // optional: "EXECUTION_TIMEOUT":"300", // in seconds "SHADOW_NAME": "SHADOW_NAME", "REPORTED_STATE": { "STATE_ATTRIBUTE": "STATE_VALUE" } }, "test":{ "id":"Shadow_Publish_Reported_State", "version":"0.0.0" } } ] ``` The `REPORTED_STATE` can be provided for additional validation on your device's exact shadow state, after it connects. By default, this test case validates your device publishing state. If `SHADOW_NAME` is not provided, the test case looks for messages published to topic prefixes of the Unnamed (classic) shadow type by default. Provide a shadow name if your device uses the named shadow type. See [Using shadows in devices](https://docs.aws.amazon.com/iot/latest/developerguide/device-shadow-comms-device.html) for more information. ## Update ***"Device updates reported state to desired state (Happy case)"*** Validates if your device reads all update messages received and synchronizes the device's state to match the desired state properties. Your device should publish its latest reported state after synchronizing. If your device already has an existing shadow before running the test, make sure the desired state configured for the test case and the existing reported state do not already match. You can identify Shadow update messages sent by Device Advisor by looking at the **ClientToken** field in the Shadow document as it will be `DeviceAdvisorShadowTestCaseSetup`. *API test case definition:* `EXECUTION_TIMEOUT` has a default value of 5 minutes. We recommend a timeout value of 2 minutes. ``` "tests":[ { "name":"my_shadow_update_reported_state", "configuration": { "DESIRED_STATE": { "STATE_ATTRIBUTE": "STATE_VALUE" }, // optional: "EXECUTION_TIMEOUT":"300", // in seconds "SHADOW_NAME": "SHADOW_NAME" }, "test":{ "id":"Shadow_Update_Reported_State", "version":"0.0.0" } } ] ``` The `DESIRED_STATE` should have at least one attribute and associated value. If `SHADOW_NAME` is not provided, then the test case looks for messages published to topic prefixes of the Unnamed (classic) shadow type by default. Provide a shadow name if your device uses the named shadow type. See [Using shadows in devices](https://docs.aws.amazon.com/iot/latest/developerguide/device-shadow-comms-device.html) for more information. # Job Execution **"Device can complete a job execution"** This test case helps you validate if your device is able to receive updates using AWS IoT Jobs, and publish the status of successful updates. For more information on AWS IoT Jobs, see [ Jobs](https://docs.aws.amazon.com//iot/latest/developerguide/iot-jobs.html). To successfully run this test case, there are two reserved AWS topics that you need to grant your [Device Role](https://docs.aws.amazon.com/iot/latest/developerguide/device-advisor-setting-up.html#da-iam-role) . To subscribe to job activity related messages, use the **notify** and **notify-next** topics. Your device role must grant PUBLISH action for the following topics: + \$1aws/things/**thingName**/jobs/**jobId**/get + \$1aws/things/**thingName**/jobs/**jobId**/update It is recommended to grant SUBSCRIBE and RECEIVE actions for the following topics: + \$1aws/things/**thingName**/jobs/get/accepted + \$1aws/things/**thingName**/jobs/**jobId**/get/rejected + \$1aws/things/**thingName**/jobs/**jobId**/update/accepted + \$1aws/things/**thingName**/jobs/**jobId**/update/rejected It is recommended to grant SUBSCRIBE action for the following topic: + \$1aws/things/**thingName**/jobs/notify-next For more information about these reserved topics, see reserved topics for [AWS IoT Jobs](https://docs.aws.amazon.com/iot/latest/developerguide/reserved-topics.html#reserved-topics-job). **MQTT over WebSocket** is not supported at this time. *API test case definition:* `EXECUTION_TIMEOUT` has a default value of 5 minutes. We recommend a timeout value of 3 minutes. Depending on the AWS IoT Job document or source provided, adjust the timeout value (for example, if a job will take a long time to run, define a longer timeout value for the test case). To run the test, either a valid AWS IoT Job document or an already existing job ID is required. An AWS IoT Job document can be provided as a JSON document or an S3 link. If a job document is provided, providing a job ID is optional. If a job ID is provided, Device Advisor will use that ID while creating the AWS IoT Job on your behalf. If the job document is not provided, you can provide an existing ID that is in the same region as you are running the test case. In this case, Device Advisor will use that AWS IoT Job while running the test case. ``` "tests": [ { "name":"my_job_execution", "configuration": { // optional: // Test case will create a job task by using either JOB_DOCUMENT or JOB_DOCUMENT_SOURCE. // If you manage the job task on your own, leave it empty and provide the JOB_JOBID (self-managed job task). // JOB_DOCUMENT is a JSON formatted string "JOB_DOCUMENT": "{ \"operation\":\"reboot\", \"files\" : { \"fileName\" : \"install.py\", \"url\" : \"${aws:iot:s3-presigned-url:https://s3.amazonaws.com/bucket-name/key}\" } }", // JOB_DOCUMENT_SOURCE is an S3 link to the job document. It will be used only if JOB_DOCUMENT is not provided. "JOB_DOCUMENT_SOURCE": "https://s3.amazonaws.com/bucket-name/key", // JOB_JOBID is mandatory, only if neither document nor document source is provided. (Test case needs to know the self-managed job task id). "JOB_JOBID": "String", // JOB_PRESIGN_ROLE_ARN is used for the presign Url, which will replace the placeholder in the JOB_DOCUMENT field "JOB_PRESIGN_ROLE_ARN": "String", // Presigned Url expiration time. It must be between 60 and 3600 seconds, with the default value being 3600. "JOB_PRESIGN_EXPIRES_IN_SEC": "Long" "EXECUTION_TIMEOUT": "300", // in seconds }, "test": { "id": "Job_Execution", "version": "0.0.0" } } ] ``` For more information on creating and using job documents see [job document](https://docs.aws.amazon.com//iot/latest/developerguide/iot-jobs.html). # Permissions and policies You can use the following tests to determine if the policies attached to your devices’ certificates follow standard best practices. **MQTT over WebSocket** is not supported at this time. **"Device certificate attached policies don’t contain wildcards"** Validates if the permission policies associated with a device follow best practices and do not grant the device more permissions than needed. *API test case definition:* `EXECUTION_TIMEOUT` has a default value of 1 minute. We recommend setting a timeout of at least 30 seconds. ``` "tests":[ { "name":"my_security_device_policies", "configuration": { // optional: "EXECUTION_TIMEOUT":"60" // in seconds }, "test": { "id": "Security_Device_Policies", "version": "0.0.0" } } ] ``` # Long duration tests Long duration tests is a new test suite that monitors a device's behavior when it operates over longer periods of time. Compared to running individual tests that focus on specific behaviors of a device, the long duration test examines the device's behavior in a variety of real-world scenarios over the device's lifespan. Device Advisor orchestrates the tests in the most efficient possible order. The test generates results and logs, including a summary log with useful metrics about the device's performance while under test. ## MQTT long duration test case In the MQTT long duration test case, the device's behavior is initially observed in happy case scenarios such as MQTT Connect, Subscribe, Publish, and Reconnect. Then, the device is observed in multiple, complex failure scenarios such as MQTT Reconnect Backoff, Long Server Disconnect, and Intermittent Connectivity. ## MQTT long duration test case execution flow There are three phases in the execution of a MQTT long duration test case: ![\[The "MQTT Long Duration test execution" that shows Basic test execution, Advanced tests execution, and Additional execution time.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/mqtt-execution-flow.png) ### Basic tests execution In this phase, the test case runs simple tests in parallel. The test validates if the device has the operations selected in the configuration. The set of basic tests can include the following, based on the operations selected: #### CONNECT This scenario validates if the device is able to make a successful connection with the broker. ![\[The basic connection flow that includes a device sending a CONNECT message and Broker responds with a CONNACK message with a successful return code.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/basic-connect.png) #### PUBLISH This scenario validates if the device successfully publishes against the broker. ##### QoS 0 This test case validates if the device successfully sends a `PUBLISH` message to the broker during a publish with QoS 0. The test does not wait on the `PUBACK` message to be received by the device. ![\[The PUBLISH QoS 0 flow that includes a device sending a PUBLISH message with QoS 0 level.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/Qos0.png) ##### QoS 1 In this test case, the device is expected to send two `PUBLISH` messages to the broker with QoS 1. After the first `PUBLISH` message, the broker waits for up to 15 seconds before it responds. The device must retry the original `PUBLISH` message with the same packet identifier within the 15 second window. If it does, the broker responds with a `PUBACK` message and the test validates. If the device doesn't retry the `PUBLISH`, the original `PUBACK` is sent to the device and the test is marked as **Pass with warnings**, along with a system message. During the test execution, if the device loses connection and reconnects, the test scenario will reset without failing and the device has to perform the test scenario steps again. ![\[The PUBLISH QoS 1 flow that includes a device sending a PUBLISH message with QoS 1 level and multiple interactions with the broker.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/Qos1.png) #### SUBSCRIBE This scenario validates if the device successfully subscribes against the broker. ##### QoS 0 This test case validates if the device successfully sends a `SUBSCRIBE` message to the broker during a subscribe with QoS 0. The test doesn't wait for the device to receive a SUBACK message. ![\[The SUBSCRIBE QoS 0 flow that includes a device sending a SUBSCRIBE message with QoS 0 level and a broker responding with a SUBACK message and Success Maximum QoS 0 code.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/subscribe-Qos0.png) ##### QoS 1 In this test case, the device is expected to send two `SUBSCRIBE` messages to the broker with QoS 1. After the first `SUBSCRIBE` message, the broker waits for up to 15 seconds before it responds. The device must retry the original `SUBSCRIBE` message with the same packet identifier within the 15 second window. If it does, the broker responds with a `SUBACK` message and the test validates. If the device doesn't retry the `SUBSCRIBE`, the original `SUBACK` is sent to the device and the test is marked as **Pass with warnings**, along with a system message. During the test execution, if the device loses connection and reconnects, the test scenario will reset without failing and the device has to perform the test scenario steps again. ![\[The SUBSCRIBE QoS 1 flow that includes a device sending a SUBSCRIBE message with QoS 1 level and multiple interactions with the broker.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/subscribe-Qos1.png) #### RECONNECT This scenario validates if the device successfully reconnects with the broker after the device is disconnected from a successful connection. Device Advisor won't disconnect the device if it connected more than once previously during the test suite. Instead, it will mark the test as **Pass**. ![\[The RECONNECT flow between DUT and the broker.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/reconnect.png) ### Advanced tests execution In this phase, the test case runs more complex tests in serial to validate if the device follows best practices. These advanced tests are available for selection and can be opted out if not required. Each advanced test has its own timeout value based on what the scenario demands. #### RETURN PUBACK ON QoS 1 SUBSCRIPTION **Note** Only select this scenario if your device is capable of performing QoS 1 subscriptions. This scenario validates if, after the device subscribes to a topic and receives a `PUBLISH` message from the broker, it returns a `PUBACK` message. ![\[The RETURN PUBACK ON QoS 1 SUBSCTIPTION flow between DUT and the broker.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/return-puback.png) #### RECEIVE LARGE PAYLOAD **Note** Select this scenario only if your device is capable of performing QoS 1 subscriptions. This scenario validates if the device responds with a `PUBACK` message after receiving a `PUBLISH` message from the broker for a QoS 1 topic with a large payload. The format of the expected payload can be configured using the `LONG_PAYLOAD_FORMAT` option. ![\[The RECEIVE LARGE PAYLOAD flow between DUT and the broker.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/large-payload.png) #### PERSISTENT SESSION **Note** Select this scenario only if your device is capable of performing QoS 1 subscriptions and can maintain a persistent session. This scenario validates the device behavior in maintaining persistent sessions. The test validates when the following conditions are met: + The device connects to the broker with an active QoS 1 subscription and persistent sessions enabled. + The device successfully disconnects from the broker during the session. + The device reconnects to the broker and resumes subscriptions to its trigger topics without explicitly resubscribing to those topics. + The device successfully receives messages stored by the broker for its subscribed topics and runs as expected. For more information on AWS IoT Persistent Sessions, see [Using MQTT persistent sessions](https://docs.aws.amazon.com//iot/latest/developerguide/mqtt.html#mqtt-persistent-sessions). ![\[The PERSISTENT SESSION flow between DUT and the broker.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/persistent-session.png) #### KEEP ALIVE This scenario validates if the device successfully disconnects after it doesn't receive a ping response from the broker. The connection must have a valid keep-alive timer configured. As part of this test, the broker blocks all responses sent for `PUBLISH`, `SUBSCRIBE`, and `PINGREQ` messages. It also validates if the device under test disconnects the MQTT connection. ![\[The KEEP ALIVE flow between DUT and the broker.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/keep-alive.png) #### INTERMITTENT CONNECTIVITY This scenario validates if the device can connect back to the broker after the broker disconnects the device at random intervals for a random period of time. ![\[The INTERMITTENT CONNECTIVITY flow between DUT and the broker.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/intermittent.png) #### RECONNECT BACKOFF This scenario validates if the device has a backoff mechanism implemented when the broker disconnects from it multiple times. Device Advisor reports the backoff type as exponential, jitter, linear or constant. The number of backoff attempts is configurable using the `BACKOFF_CONNECTION_ATTEMPTS` option. The default value is 5. The value is configurable between 5 and 10. To pass this test, we recommend implementing the [ Exponential Backoff And Jitter](http://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/) mechanism on the device under test. ![\[The RECONNECT BACKOFF flow between DUT and the broker.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/reconnect-backoff.png) #### LONG SERVER DISCONNECT This scenario validates if the device can successfully reconnect after the broker disconnects the device for a long period of time (up to 120 minutes). The time for server disconnection can be configured using the `LONG_SERVER_DISCONNECT_TIME` option. The default value is 120 minutes. This value is configurable from 30 to 120 minutes. ![\[The LONG SERVER DISCONNECT flow between DUT and the broker.\]](http://docs.aws.amazon.com/iot/latest/developerguide/images/longserver-disconnect.png) ### Additional execution time The additional execution time is the time the test waits after completing all the above tests and before ending the test case. Customers use this additional time period to monitor and log all communications between the device and the broker. The additional execution time can be configured using the `ADDITIONAL_EXECUTION_TIME` option. By default, this option is set to 0 minutes and can be 0 to 120 minutes. ## MQTT long duration test configuration options All configuration options provided for the MQTT long duration test are optional. The following options are available: **OPERATIONS** The list of operations that the device performs, such as `CONNECT`, `PUBLISH` and `SUBSCRIBE`. The test case runs scenarios based on the specified operations. Operations that aren't specified are assumed valid. ``` { "OPERATIONS": ["PUBLISH", "SUBSCRIBE"] //by default the test assumes device can CONNECT } ``` **SCENARIOS** Based on the operations selected, the test case runs scenarios to validate the device's behavior. There are two types of scenarios: + **Basic Scenarios** are simple tests that validate if the device can perform the operations selected above as part of the configuration. These are pre-selected based on the operations specified in the configuration. No more input is required in the configuration. + **Advanced Scenarios** are more complex scenarios that are performed against the device to validate if the device follows best practices when met with real world conditions. These are optional and can be passed as an array of scenarios to the configuration input of the test suite. ``` { "SCENARIOS": [ // list of advanced scenarios "PUBACK_QOS_1", "RECEIVE_LARGE_PAYLOAD", "PERSISTENT_SESSION", "KEEP_ALIVE", "INTERMITTENT_CONNECTIVITY", "RECONNECT_BACK_OFF", "LONG_SERVER_DISCONNECT" ] } ``` **BASIC\$1TESTS\$1EXECUTION\$1TIME\$1OUT:** The maximum time the test case will wait for all the basic tests to complete. The default value is 60 minutes. This value is configurable from 30 to 120 minutes. **LONG\$1SERVER\$1DISCONNECT\$1TIME:** The time taken for the test case to disconnect and reconnect the device during the Long Server Disconnect test. The default value is 60 minutes. This value is configurable from 30 to 120 minutes. **ADDITIONAL\$1EXECUTION\$1TIME:** Configuring this option provides a time window after all the tests are completed, to monitor events between the device and broker. The default value is 0 minutes. This value is configurable from 0 to 120 minutes. **BACKOFF\$1CONNECTION\$1ATTEMPTS:** This option configures the number of times the device is disconnected by the test case. This is used by the Reconnect Backoff test. The default value is 5 attempts. This value is configurable from 5 to 10. **LONG\$1PAYLOAD\$1FORMAT:** The format of the message payload that the device expects when the test case publishes to a QoS 1 topic subscribed by the device. **API test case definition:** ``` { "tests":[ { "name":"my_mqtt_long_duration_test", "configuration": { // optional "OPERATIONS": ["PUBLISH", "SUBSCRIBE"], "SCENARIOS": [ "LONG_SERVER_DISCONNECT", "RECONNECT_BACK_OFF", "KEEP_ALIVE", "RECEIVE_LARGE_PAYLOAD", "INTERMITTENT_CONNECTIVITY", "PERSISTENT_SESSION", ], "BASIC_TESTS_EXECUTION_TIMEOUT": 60, // in minutes (60 minutes by default) "LONG_SERVER_DISCONNECT_TIME": 60, // in minutes (120 minutes by default) "ADDITIONAL_EXECUTION_TIME": 60, // in minutes (0 minutes by default) "BACKOFF_CONNECTION_ATTEMPTS": "5", "LONG_PAYLOAD_FORMAT":"{"message":"${payload}"}" }, "test":{ "id":"MQTT_Long_Duration", "version":"0.0.0" } } ] } ``` ## MQTT long duration test case summary log The MQTT long duration test case runs for longer duration than regular test cases. A separate summary log is provided, which lists important events such as device connections, publish, and subscribe during the run. Details include what was tested, what was not tested and what failed. At the end of the log, the test includes a summary of all the events that happened during the test case run. This includes: + *Keep Alive timer configured on the device.* + *Persistent session flag configured on the device.* + *The number of device connections during the test run.* + *The device reconnection backoff type, if validated for the reconnect backoff test.* + *The topics the device published to, during the test case run.* + *The topics the device subscribed to, during the test case run.*