Troubleshoot errors

Each test suite execution has a unique execution ID that is used to create a folder named results/execution-id in the results directory. Individual test group logs are under the results/execution-id/logs directory. Use the IDT for FreeRTOS console output to find the execution id, test case id, and test group id of the test case that failed and then open the log file for that test case named results/execution-id/logs/test_group_id__test_case_id.log. The information in this file includes:

We recommend the following workflow for troubleshooting:

Troubleshoot device configuration

When you use IDT for FreeRTOS, you must get the correct configuration files in place before you execute the binary. If you're getting parsing and configuration errors, your first step should be to locate and use a configuration template appropriate for your environment. These templates are located in the IDT_ROOT/configs directory.

If you are still having issues, see the following debugging process.

Where do I look?

Start by reading the console output to find information, such as the execution UUID, which is referenced as execution-id in this documentation.

Next, look in the FRQ_Report.xml file in the /results/execution-id directory. This file contains all of the test cases that were run and error snippets for each failure. To get all of the execution logs, look for the file /results/execution-id/logs/test_group_id__test_case_id.log for each test case.

IDT error codes

The following table explains the error codes generated by IDT for FreeRTOS:

Debug config file parsing errors

Occasionally, a typo in a JSON configuration can lead to parsing errors. Most of the time, the issue is a result of omitting a bracket, comma, or quote from your JSON file. IDT for FreeRTOS performs JSON validation and prints debugging information. It prints the line where the error occurred, the line number, and the column number of the syntax error. This information should be enough to help you fix the error, but if you are still having issues locating the error, you can perform validation manually in your IDE, a text editor such as Atom or Sublime, or through an online tool like JSONLint.

Debug test results parse errors

While running a test group from FreeRTOS-Libraries-Integration-Tests, such as FullTransportInterfaceTLS, FullPKCS11_Core, FullPKCS11_Onboard_ECC, FullPKCS11_Onboard_RSA, FullPKCS11_PreProvisioned_ECC, FullPKCS11_PreProvisioned_RSA, or OTACore, IDT for FreeRTOS parses the test results from the test device with the serial connection. Sometimes, extra serial outputs on the device may interfere with the parsing of the test results.

In the above mentioned case, strange test case failure reasons like strings originating from unrelated device outputs are output. The IDT for FreeRTOS test case log file (which includes all the serial output IDT for FreeRTOS has received during the test) may show the following:

<unrelated device output>
TEST(Full_PKCS11_Capabilities, PKCS11_Capabilities)<unrelated device output>
<unrelated device output>
 PASS 
        

In the above example, the unrelated device output prevents IDT for FreeRTOS from detecting the test result which is PASS.

Check the following to ensure optimal testing.

An IDT for FreeRTOS test case log would ideally show an uninterrupted test results output like below:

---------STARTING TESTS---------
TEST(Full_OTA_PAL, otaPal_CloseFile_ValidSignature) PASS
TEST(Full_OTA_PAL, otaPal_CloseFile_InvalidSignatureBlockWritten) PASS
-----------------------
2 Tests 0 Failures 0 Ignored        
      

Debug integrity check failures

If using FRQ 1.x.x version of FreeRTOS the following integrity checks apply.

When you run the FreeRTOSIntegrity test group and you encounter failures, first make sure that you haven't modified any of the freertos directory files. If you haven’t, and are still seeing issues, make sure you are using the correct branch. If you run IDT's list-supported-products command, you can find which tagged branch of the freertos repo you should be using.

If you cloned the correct tagged branch of the freertos repo and still have issues, make sure you have also run the submodule update command. The clone workflow for the freertos repo is as follows.

git clone --branch version-number https://github.com/aws/amazon-freertos.git
cd amazon-freertos
git submodule update --checkout —init —recursive

The list of files the integrity checker looks for are in the checksums.json file in your freertos directory. To qualify a FreeRTOS port without any modifications to files and the folder structure, make sure that none of the files listed in the 'exhaustive' and 'minimal' sections of the checksums.json file have been modified. To run with an SDK configured, verify that none of the files under the 'minimal' section have been modified.

If you run IDT with an SDK and have modified some files in your freertos directory, then make sure you correctly configure your SDK in your userdata file. Otherwise, the Integrity checker will verify all files in the freertos directory.

Debug FullWiFi test group failures

If you are using FRQ 1.x.x and encounter failures in the FullWiFi test group, and the "AFQP_WiFiConnectMultipleAP" test fails, this could be because both access points aren't in the same subnet as the host computer running IDT. Make sure that both access points are in the same subnet as the host computer running IDT.

Debug "required parameter missing" errors

Because new features are being added to IDT for FreeRTOS, changes to the configuration files might be introduced. Using an old configuration file might break your configuration. If this happens, the test_group_id__test_case_id.log file under the results/execution-id/logs directory explicitly lists all missing parameters. IDT for FreeRTOS validates your JSON configuration file schemas to ensure that the latest supported version has been used.

Debug "test could not start" errors

You might see errors that point to failures during test start. Because there are several possible causes, check the following areas for correctness:

Debug "unable to find start of test results" errors

You might see errors when IDT attempts to parse the results output by the device under test. There are several possible causes, so check the following areas for correctness:

Debug a "Test failure:expected __ results but saw ___" errors

You might see errors that point to a test failure during testing. The test expects a certain number of results, and does not see it during testing. Some FreeRTOS tests run before IDT sees the output from the device. If you see this error, you can try increasing the value of testStartDelayms in your userdata configuration. For more information, see Configure build, flash, and test settings.

Debug a "________ was unselected due to ConditionalTests constraints" errors

This means that you are running a test on a device pool that is incompatible with the test. This may happen with the OTA E2E tests. For example, while running the OTADataplaneMQTT test group and in your device.json config file, you have chosen OTA as No or OTADataPlaneProtocol as HTTP. The test group chosen to run must match your device.json capability selections.

Debug an IDT timeout during device output monitoring

IDT can timeout due to a number of reasons. If a timeout happens during the device output monitoring phase of a test, and you can see the results inside of the IDT test case log, it means that the results were incorrectly parsed by IDT. One reason could be the interleaved log messages in the middle of the test results. If this is the case, please refer to the FreeRTOS Porting Guide for further details on how the UNITY logs should be setup.

Another reason for a timeout during device output monitoring could be a device rebooting after a single TLS test case failure. The device then runs the flashed image and causes an infinite loop which is seen in the logs. If this happens, make sure your device does not reboot after a test failure.

Debug a "not authorized to access resource" error

You might see the error "user/role is not authorized to access this resource" in the terminal output or in the test_manager.log file under /results/execution-id/logs. To resolve this issue, attach the AWSIoTDeviceTesterForFreeRTOSFullAccess managed policy to your test user. For more information, see Create and configure an AWS account.

Debug network test errors

For network-based tests, IDT starts an echo server that binds to a non-reserved port on the host machine. If you are running into errors due to timeouts or unavailable connections in the WiFi or secure sockets tests, make sure that your network is configured to allow traffic to configured ports in the 1024 - 49151 range.

The secure sockets test uses ports 33333 and 33334 by default. The WiFi tests uses port 33335 by default. If these three ports are in use or blocked by firewall or network, you can choose to use different ports in userdata.json for testing. For more information, see Configure build, flash, and test settings. You can use the following commands to check whether a specific port is in use:

OTA update failures due to same version payload

If OTA test cases are failing due to the same version being on the device after an OTA was performed, it may be due to your build system (e.g. cmake) not noticing IDT's changes to the FreeRTOS source code and not building an updated binary. This causes OTA to be performed with the same binary that is currently on the device, and the test to fail. To troubleshoot OTA update failures, start by making sure that you are using the latest supported version of your build system.

OTA test failure on PresignedUrlExpired test case

One prerequisite of this test is that the OTA update time should be more than 60 seconds, otherwise the test would fail. If this occurs, the following error message is found in the log: "Test takes less than 60 seconds (url expired time) to finish. Please reach out to us."

Debug device interface and port errors

This section contains information about the device interfaces IDT uses to connect to your devices.

Supported platforms

IDT supports Linux, macOS, and Windows. All three platforms have different naming schemes for serial devices that are attached to them:

To check your device port:

To verify which device is connected to a port:

Device interfaces

Each embedded device is different, which means that they can have one or more serial ports. It is common for devices to have two ports when connected to a machine:

For Windows, follow the same logic.

Reading device data

IDT for FreeRTOS uses individual device build and flash tooling to specify port configuration. If you are testing your device and don't get output, try the following default settings:

These settings are handled by IDT for FreeRTOS. You do not have to set them. However, you can use the same method to manually read device output. On Linux or macOS, you can do this with the screen command. On Windows, you can use a program such as TeraTerm.

Screen: screen /dev/cu.usbserial 115200

TeraTerm: Use the above-provided settings to set the fields explicitly in the GUI.

Development toolchain problems

This section discusses problems that can occur with your toolchain.

Code Composer Studio on Ubuntu

Newer versions of Ubuntu (17.10 and 18.04) have a version of the glibc package that is not compatible with Code Composer Studio 7.x versions. We recommended that you install Code Composer Studio version 8.2 or later.

Symptoms of incompatibility might include:

Logging

IDT for FreeRTOS logs are placed in a single location. From the root IDT directory, these files are available under results/execution-id/:

FRQ_Report.xml and logs/test_group_id__test_case_id.log are the most important logs to examine. FRQ_Report.xml contains information about which test cases failed with a specific error message. You can then use logs/test_group_id__test_case_id.log to dig further into the problem to get better context.

Console errors

When AWS IoT Device Tester is run, failures are reported to the console with brief messages. Look in results/execution-id/logs/test_group_id__test_case_id.log to learn more about the error.

Log errors

Each test suite execution has a unique execution ID that is used to create a folder named results/execution-id. Individual test case logs are under the results/execution-id/logs directory. Use the output of the IDT for FreeRTOS console to find the execution id, test case id, and test group id of the test case that failed. Then use this information to find and open the log file for that test case named results/execution-id/logs/test_group_id__test_case_id.log The information in this file includes the full build and flash command output, test execution output, and more verbose AWS IoT Device Tester console output.

S3 bucket issues

If you press CTRL+C while running IDT, IDT will start a clean up process. Part of that clean up is to remove Amazon S3 resources that have been created as a part of the IDT tests. If the clean up can't finish, you might run into an issue where you have too many Amazon S3 buckets that have been created. This means the next time that you run IDT the tests will start to fail.

If you press CTRL+C to stop IDT, you must let it finish the clean up process to avoid this issue. You can also delete the Amazon S3 buckets from your account that were created manually.

Troubleshoot timeout errors

If you see timeout errors while running a test suite, increase the timeout by specifying a timeout multiplier factor. This factor is applied to the default timeout value. Any value configured for this flag must be greater than or equal to 1.0. To use the timeout multiplier, use the flag --timeout-multiplier when running the test suite.

Cellular feature and AWS charges

When the Cellular feature is set to Yes in your device.JSON file, FullSecureSockets will use t.micro EC2 instances for running tests and this may incur additional costs to your AWS account. For more information, see Amazon EC2 pricing.

Qualification report generation policy

Qualification reports are only generated by AWS IoT Device Tester (IDT) versions that support FreeRTOS versions released within the last two years. If you have questions about the support policy, please contact AWS Support.