Device Advisor workflow

This tutorial explains how to create a custom test suite and run tests against the device you want to test in the console. After the tests are complete, you can view the test results and detailed logs.

Prerequisites

Before you begin this tutorial this tutorial, complete the steps outlined in Setting up.

Create a test suite definition

First, install an AWS SDK.

rootGroup syntax

A root group is a JSON string that specifies which test cases to include in your test suite. It also specifies any necessary configurations for those test cases. Use the root group to structure and order your test suite based on your needs. The hierarchy of a test suite is:

test suite → test group(s) → test case(s)

A test suite must have at least one test group, and each test group must have at least one test case. Device Advisor runs tests in the order in which you define the test groups and test cases.

Each root group follows this basic structure:

{
    "configuration": {  // for all tests in the test suite
        "": ""
    }
    "tests": [{
        "name": ""
        "configuration": {  // for all sub-groups in this test group 
            "": ""
        },
        "tests": [{
            "name": ""
            "configuration": {  // for all test cases in this test group 
                "": ""
            },
            "test": {
                "id": ""  
                "version": ""
            }
        }]
    }]
}

In the root group, you define the test suite with a name, configuration, and the tests that the group contains. The tests group contains the definitions of individual tests. You define each test with a name, configuration, and a test block that defines the test cases for that test. Finally, each test case is defined with an id and version.

For information on how to use the "id" and "version" fields for each test case (test block), see Device Advisor test cases. That section also contains information on the available configuration settings.

The following block is an example of a root group configuration. This configurations specifies the MQTT Connect Happy Case and MQTT Connect Exponential Backoff Retries test cases, along with descriptions of the configuration fields.

{
    "configuration": {},  // Suite-level configuration
    "tests": [            // Group definitions should be provided here
      {
        "name": "My_MQTT_Connect_Group",  // Group definition name
        "configuration": {}               // Group definition-level configuration,
        "tests": [                        // Test case definitions should be provided here
        {
            "name": "My_MQTT_Connect_Happy_Case",  // Test case definition name
            "configuration": {
                "EXECUTION_TIMEOUT": 300        // Test case definition-level configuration, in seconds
            }, 
            "test": {
                "id": "MQTT_Connect",              // test case id
                "version": "0.0.0"                 // test case version
            }
        },
        {
            "name": "My_MQTT_Connect_Jitter_Backoff_Retries",  // Test case definition name
            "configuration": {
                "EXECUTION_TIMEOUT": 600                 // Test case definition-level configuration,  in seconds
            },
            "test": {
                "id": "MQTT_Connect_Jitter_Backoff_Retries",  // test case id
                "version": "0.0.0"                                 // test case version
            }
        }]
    }]
}

You must supply the root group configuration when you create your test suite definition. Save the suiteDefinitionId that is returned in the response object. You can use this ID to retrieve your test suite definition information and run your test suite.

Here is a Java SDK example:

response = iotDeviceAdvisorClient.createSuiteDefinition(
        CreateSuiteDefinitionRequest.builder()
            .suiteDefinitionConfiguration(SuiteDefinitionConfiguration.builder()
                .suiteDefinitionName("your-suite-definition-name")
                .devices(
                    DeviceUnderTest.builder()
                        .thingArn("your-test-device-thing-arn")
                        .certificateArn("your-test-device-certificate-arn")
                        .deviceRoleArn("your-device-role-arn") //if using SigV4 for MQTT over WebSocket
                        .build()
                )
                .rootGroup("your-root-group-configuration")
                .devicePermissionRoleArn("your-device-permission-role-arn")
                .protocol("MqttV3_1_1 || MqttV5 || MqttV3_1_1_OverWebSocket || MqttV5_OverWebSocket")
                .build()
            )
            .build()
)

Get a test suite definition

After you create your test suite definition, you receive the suiteDefinitionId in the response object of the CreateSuiteDefinition API operation.

When the operation returns the suiteDefinitionId, you may see new id fields within each group and test case definition within the root group. You can use these IDs to run a subset of your test suite definition.

Java SDK example:

response = iotDeviceAdvisorClient.GetSuiteDefinition(
    GetSuiteDefinitionRequest.builder()
        .suiteDefinitionId("your-suite-definition-id")
        .build()
)

Get a test endpoint

Use the GetEndpoint API operation to get the test endpoint used by your device. Select the endpoint that best fits your test. To simultaneously run multiple test suites, use the Device-level endpoint by providing a thing ARN, certificate ARN, or device role ARN. To run a single test suite, provide no arguments to the GetEndpoint operation to choose the Account-level endpoint.

SDK example:

response = iotDeviceAdvisorClient.getEndpoint(GetEndpointRequest.builder()
.certificateArn("your-test-device-certificate-arn")
.thingArn("your-test-device-thing-arn")
.deviceRoleArn("your-device-role-arn") //if using SigV4 for MQTT over WebSocket                
.build())

Start a test suite run

After you create a test suite definition and configureyour test device to connect to your Device Advisor test endpoint, run your test suite with the StartSuiteRun API.

For MQTT customers, use either certificateArn or thingArn to run the test suite. If both are configured, the certificate is used if it belongs to the thing.

For MQTT over WebSocket customer, use deviceRoleArn to run the test suite. If the specified role is different from the role specified in the test suite definition, the specified role overrides the defined role.

For .parallelRun(), use true if you use a Device-level endpoint to run multiple test suites in parallel using one AWS account.

SDK example:

response = iotDeviceAdvisorClient.startSuiteRun(StartSuiteRunRequest.builder()
.suiteDefinitionId("your-suite-definition-id")
.suiteRunConfiguration(SuiteRunConfiguration.builder()
    .primaryDevice(DeviceUnderTest.builder()
        .certificateArn("your-test-device-certificate-arn")
        .thingArn("your-test-device-thing-arn")
        .deviceRoleArn("your-device-role-arn") //if using SigV4 for MQTT over WebSocket               
        .build())
    .parallelRun(true | false)    
    .build())
.build())

Save the suiteRunId from the response. You will use this to retrieve the results of this test suite run.

Get a test suite run

After you start a test suite run, you can check its progress and its results with the GetSuiteRun API.

SDK example:

// Using the SDK, call the GetSuiteRun API.

response = iotDeviceAdvisorClient.GetSuiteRun(
GetSuiteRunRequest.builder()
    .suiteDefinitionId("your-suite-definition-id")
    .suiteRunId("your-suite-run-id")
.build()) 

Stop a test suite run

To stop a test suite run that is still in progress, you can call the StopSuiteRun API operation. After you call the StopSuiteRun operation, the service starts the cleanup process. While the service runs the cleanup process, the test suite run status updates to Stopping. The cleanup process can take several minutes. Once the process is complete, the test suite run status updates to Stopped. After a test run has completely stopped, you n start another test suite run. You can periodically check the suite run status using the GetSuiteRun API operation, as shown in the previous section.

SDK example:

// Using the SDK, call the StopSuiteRun API.

response = iotDeviceAdvisorClient.StopSuiteRun(
StopSuiteRun.builder()
    .suiteDefinitionId("your-suite-definition-id")
    .suiteRunId("your-suite-run-id")
.build())

Get a qualification report for a successful qualification test suite run

If you run a qualification test suite that completes successfully, you can retrieve a qualification report with the GetSuiteRunReport API operation. You use this qualification report to qualify your device with the AWS IoT Core qualification program. To determine whether your test suite is a qualification test suite, check whether the intendedForQualification parameter is set to true. After you call the GetSuiteRunReport API operation, you can download the report from the returned URL for up to 90 seconds. If more than 90 seconds elapse from the previous time you called the GetSuiteRunReport operation, call the operation again to retrieve a new, valid URL.

SDK example:

// Using the SDK, call the getSuiteRunReport API. 

response = iotDeviceAdvisorClient.getSuiteRunReport( 
    GetSuiteRunReportRequest.builder() 
        .suiteDefinitionId("your-suite-definition-id")
        .suiteRunId("your-suite-run-id")
        .build()
)