# Analyzing test results in AWS Device Farm In the standard test environment, you can use the Device Farm console to view reports for each test in your test run. Vewing the reports helps you understand which tests passed or failed, and provides you with details on the performance and behavior of your app across different device configurations. Device Farm also gathers other artifacts such as files, logs, and images that you can download when your test run is complete. This information can help you analyze how your app is behaving on real devices, identify issues or bugs, and diagnose problems. **Topics** + [Viewing test reports in Device Farm](how-to-use-reports.md) + [Downloading artifacts in Device Farm](artifacts.md) # Viewing test reports in Device Farm Use the Device Farm console to view your test reports. For more information, see [Reports in AWS Device FarmReports](reports.md). **Topics** + [Prerequisites](#how-to-use-reports-prerequisites) + [View reports](#how-to-use-reports-viewing-reports) + [Device Farm test result statuses](how-to-use-reports-displaying-results.md) ## Prerequisites Set up a test run and verify that it is complete. 1. To create a run, see [Creating a test run in Device Farm](how-to-create-test-run.md), and then return to this page. 1. Verify that the run is complete. During your test run, the Device Farm console displays a pending icon ![\[Device Farm scheduled a job.\]](http://docs.aws.amazon.com/devicefarm/latest/developerguide/images/df-run-calendar.png) for runs that are in progress. Each device in the run will also start with the pending icon, then switch to the running ![\[Device Farm progress indicator.\]](http://docs.aws.amazon.com/devicefarm/latest/developerguide/images/df-run-progress.png) icon when the test begins. As each test finishes, a test result icon is displayed next to the device name. When all tests have been completed, the pending icon next to the run changes to a test result icon. For more information, see [Device Farm test result statuses](how-to-use-reports-displaying-results.md). ## View reports You can view the results of your test in the Device Farm console. **Topics** + [View the test run summary page](#how-to-use-reports-console-summary) + [View unique problem reports](#how-to-use-reports-console-unique-problems) + [View device reports](#how-to-use-reports-console-by-device) + [View test suite reports](#how-to-use-reports-console-by-suite) + [View test reports](#how-to-use-reports-console-by-test) + [View log information for a problem, device, suite, or test in a report](#how-to-use-reports-console-log) ### View the test run summary page 1. Sign in to the Device Farm console at [https://console.aws.amazon.com/devicefarm](https://console.aws.amazon.com/devicefarm). 1. In the navigation pane, choose **Mobile Device Testing**, and then choose **Projects**. 1. In the list of projects, choose the project for the run. **Tip** To filter the project list by name, use the search bar. 1. Choose a completed run to view its summary report page. 1. The test run summary page displays an overview of your test results. + The **Unique problems** section lists unique warnings and failures. To view unique problems, follow the instructions in [View unique problem reports](#how-to-use-reports-console-unique-problems). + The **Devices** section displays the total number of tests, by outcome, for each device. ![\[Device Farm device summary results.\]](http://docs.aws.amazon.com/devicefarm/latest/developerguide/images/df-run-device-summary-results-bar.png) In this example, there are several devices. In the first table entry, the Google Pixel 4 XL device running Android version 10 reports three successful tests that took 02:36 minutes to run. To view the results by device, follow the instructions in [View device reports](#how-to-use-reports-console-by-device). + The **Screenshots** section displays a list of any screenshots that Device Farm captured during the run, grouped by device. + In the **Parsing result** section, you can download the parsing result. ### View unique problem reports 1. In **Unique problems**, choose the problem that you want to view. 1. Choose the device. The report displays information about the problem. The **Video** section displays a downloadable video recording of the test. The **Result** section displays the result of the test. The status is represented as a result icon. For more information, see [Statuses of an individual test](how-to-use-reports-displaying-results.md#how-to-use-reports-displaying-results-individual). The **Logs** section displays any information that Device Farm logged during the test. To view this information, follow the instructions in [View log information for a problem, device, suite, or test in a report](#how-to-use-reports-console-log). The **Files** tab displays a list of any of the test's associated files (such as log files) that you can download. To download a file, choose the file's link in the list. The **Screenshots** tab displays a list of any screenshots that Device Farm captured during the test. ### View device reports + In the **Devices** section, choose the device. The **Video** section displays a downloadable video recording of the test. The **Suites** section displays a table containing information about the suites for the device. In this table, the **Test results** column summarizes the number of tests by outcome for each of the test suites that have run on the device. This data also has a graphical component. For more information, see [Statuses for multiple tests](how-to-use-reports-displaying-results.md#how-to-use-reports-displaying-results-summary). To view the full results by suite, follow the instructions in [View test suite reports](#how-to-use-reports-console-by-suite). The **Logs** section displays any information that Device Farm logged for the device during the run. To view this information, follow the instructions in [View log information for a problem, device, suite, or test in a report](#how-to-use-reports-console-log). The **Files** section displays a list of suites for the device and any associated files (such as log files) that you can download. To download a file, choose the file's link in the list. The **Screenshots** section displays a list of any screenshots that Device Farm captured during the run for the device, grouped by suite. ### View test suite reports 1. In the **Devices** section, choose the device. 1. In the **Suites** section, choose the suite from the table. The **Video** section displays a downloadable video recording of the test. The **Tests** section displays a table containing information about the tests in the suite. In the table, the **Test results** column displays the result. This data also has a graphical component. For more information, see [Statuses for multiple tests](how-to-use-reports-displaying-results.md#how-to-use-reports-displaying-results-summary). To view the full results by test, follow the instructions in [View test reports](#how-to-use-reports-console-by-test). The **Logs** section displays any information that Device Farm logged during the run for the suite. To view this information, follow the instructions in [View log information for a problem, device, suite, or test in a report](#how-to-use-reports-console-log). The **Files** section displays a list of tests for the suite and any associated files (such as log files) that you can download. To download a file, choose the file's link in the list. The **Screenshots** section displays a list of any screenshots that Device Farm captured during the run for the suite, grouped by test. ### View test reports 1. In the **Devices** section, choose the device. 1. In the **Suites** section, choose the suite. 1. In the **Tests** section, choose the test. 1. The **Video** section displays a downloadable video recording of the test. The **Result** section displays the result of the test. The status is represented as a result icon. For more information, see [Statuses of an individual test](how-to-use-reports-displaying-results.md#how-to-use-reports-displaying-results-individual). The **Logs** section displays any information that Device Farm logged during the test. To view this information, follow the instructions in [View log information for a problem, device, suite, or test in a report](#how-to-use-reports-console-log). The **Files** tab displays a list of any of the test's associated files (such as log files) that you can download. To download a file, choose the file's link in the list. The **Screenshots** tab displays a list of any screenshots that Device Farm captured during the test. ### View log information for a problem, device, suite, or test in a report The **Logs** section displays the following information: + **Source** represents the source of a log entry. Possible values include: + **Harness** represents a log entry that Device Farm created. These log entries are typically created during start and stop events. + **Device** represents a log entry that the device created. For Android, these log entries are logcat-compatible. For iOS, these log entries are syslog-compatible. + **Test** represents a log entry that either a test or its test framework created. + **Time** represents the elapsed time between the first log entry and this log entry. The time is expressed in *MM:SS.SSS* format, where *M* represents minutes and *S* represents seconds. + **PID** represents the process identifier (PID) that created the log entry. All log entries created by an app on a device have the same PID. + **Level** represents the logging level for the log entry. For example, `Logger.debug("This is a message!")` logs a **Level** of `Debug`. These are the possible values: + **Alert** + **Critical** + **Debug** + **Emergency** + **Error** + **Errored** + **Failed** + **Info** + **Internal** + **Notice** + **Passed** + **Skipped** + **Stopped** + **Verbose** + **Warned** + **Warning** + **Tag** represents arbitrary metadata for the log entry. For example, Android logcat can use this to describe which part of the system created the log entry (for example, `ActivityManager`). + **Message** represents the message or data for the log entry. For example, `Logger.debug("Hello, World!")` logs a **Message** of `"Hello, World!"`. To display only a portion of the information: + To show all log entries that match a value for a specific column, enter the value into the search bar. For example, to show all log entries with a **Source** value of `Harness`, enter **Harness** in the search bar. + To remove all of the characters from a column header box, choose the **X** in that column header box. Removing all of the characters from a column header box is the same as entering **\$1** in that column header box. To download all of the log information for the device, including all of the suites and tests that you ran, choose **Download logs**. # Device Farm test result statuses The Device Farm console displays icons that help you quickly assess the state of your completed test run. For more information about tests in Device Farm, see [Reports in AWS Device FarmReports](reports.md). **Topics** + [Statuses of an individual test](#how-to-use-reports-displaying-results-individual) + [Statuses for multiple tests](#how-to-use-reports-displaying-results-summary) ## Statuses of an individual test For reports that describe an individual test, Device Farm displays an icon representing the test result status: | Description | Icon | | --- | --- | | The test succeeded. | ![\[The test succeeded.\]](http://docs.aws.amazon.com/devicefarm/latest/developerguide/images/df-run-success.png) | | The test failed. | ![\[The test failed.\]](http://docs.aws.amazon.com/devicefarm/latest/developerguide/images/df-run-failure.png) | | Device Farm skipped the test. | ![\[The test was skipped.\]](http://docs.aws.amazon.com/devicefarm/latest/developerguide/images/df-run-skipped.png) | | The test stopped. | ![\[The test was stopped.\]](http://docs.aws.amazon.com/devicefarm/latest/developerguide/images/df-run-stopped.png) | | Device Farm returned a warning. | ![\[Device Farm returned a warning.\]](http://docs.aws.amazon.com/devicefarm/latest/developerguide/images/df-run-warning.png) | | Device Farm returned an error. | ![\[Device Farm returned an error.\]](http://docs.aws.amazon.com/devicefarm/latest/developerguide/images/df-run-error.png) | ## Statuses for multiple tests If you choose a finished run, Device Farm displays a summary graph showing the percentage of tests in various states. ![\[Device Farm test results summary graph.\]](http://docs.aws.amazon.com/devicefarm/latest/developerguide/images/df-summary-results-graph.png) For example, this test run results graph shows that the run had 4 stopped tests, 1 failed test, and 10 successful tests. Graphs are always color coded and labeled. # Downloading artifacts in Device Farm Device Farm gathers artifacts such as reports, log files, and images for each test in the run. You can download artifacts created during your test run: **Files** Files generated during the test run including Device Farm reports. For more information, see [Viewing test reports in Device Farm](how-to-use-reports.md). **Logs** Output from each test in the test run. **Screenshots** Screen images recorded for each test in the test run. ![\[Workflow diagram showing Project, Run, Job (device), Test suite, and Test stages in AWS Cloud.\]](http://docs.aws.amazon.com/devicefarm/latest/developerguide/images/hierarchy.png) ## Download artifacts (console) 1. On the test run report page, from **Devices**, choose a mobile device. 1. To download a file, choose one from **Files**. 1. To download the logs from your test run, from **Logs**, choose **Download logs**. 1. To download a screenshot, choose a screenshot from **Screenshots**. For more information about downloading artifacts in a custom test environment, see [Downloading artifacts in a custom test environment](using-artifacts-custom.md). ## Download artifacts (AWS CLI) You can use the AWS CLI to list your test run artifacts. **Topics** + [Step 1: Get your Amazon Resource Names (ARN)](#artifacts-cli-step1) + [Step 2: List your artifacts](#artifacts-cli-step2) + [Step 3: Download your artifacts](#artifacts-cli-step3) ### Step 1: Get your Amazon Resource Names (ARN) You can list your artifacts by run, job, test suite, or test. You need the corresponding ARN. This table shows the input ARN for each of the AWS CLI list commands: | AWS CLI List Command | Required ARN | | --- | --- | | list-projects | This command returns all projects and does not require an ARN. | | list-runs | project | | list-jobs | run | | list-suites | job | | list-tests | suite | For example, to find a test ARN, run **list-tests** using your test suite ARN as an input parameter. Example: ``` aws devicefarm list-tests –-arn arn:MyTestSuiteARN ``` The response includes a test ARN for each test in the test suite. ``` { "tests": [ { "status": "COMPLETED", "name": "Tests.FixturesTest.testExample", "created": 1537563725.116, "deviceMinutes": { "unmetered": 0.0, "total": 1.89, "metered": 1.89 }, "result": "PASSED", "message": "testExample passed", "arn": "arn:aws:devicefarm:us-west-2:123456789101:test:5e01a8c7-c861-4c0a-b1d5-12345EXAMPLE", "counters": { "skipped": 0, "warned": 0, "failed": 0, "stopped": 0, "passed": 1, "errored": 0, "total": 1 } } ] } ``` ### Step 2: List your artifacts The AWS CLI [list-artifacts](https://docs.aws.amazon.com/cli/latest/reference/devicefarm/list-artifacts.html) command returns a list of artifacts, such as files, screenshots, and logs. Each artifact has a URL so you can download the file. + Call **list-artifacts** specifying a run, job, test suite, or test ARN. Specify a type of FILE, LOG, or SCREENSHOT. This example returns a download URL for each artifact available for an individual test: ``` aws devicefarm list-artifacts --arn arn:MyTestARN --type "FILE" ``` The response contains a download URL for each artifact. ``` { "artifacts": [ { "url": "https://prod-us-west-2-uploads.s3-us-west-2.amazonaws.com/ExampleURL", "extension": "txt", "type": "APPIUM_JAVA_OUTPUT", "name": "Appium Java Output", "arn": "arn:aws:devicefarm:us-west-2:123456789101:artifact:5e01a8c7-c861-4c0a-b1d5-12345EXAMPLE", } ] } ``` ### Step 3: Download your artifacts + Download your artifact using the URL from the previous step. This example uses **curl** to download an Android Appium Java output file: ``` curl "https://prod-us-west-2-uploads.s3-us-west-2.amazonaws.com/ExampleURL" > MyArtifactName.txt ``` ## Download artifacts (API) The Device Farm API [ListArtifacts](https://docs.aws.amazon.com/devicefarm/latest/APIReference/API_ListArtifacts.html) method returns a list of artifacts, such as files, screenshots, and logs. Each artifact has a URL so you can download the file. # Downloading artifacts in a custom test environment In a custom test environment, Device Farm gathers artifacts such as custom reports, log files, and images. These artifacts are available for each device in the test run. You can download these artifacts created during your test run: **Test spec output** The output from running the commands in the test spec YAML file. **Customer artifacts** A zipped file that contains the artifacts from the test run. It is configured in the **artifacts:** section of your test spec YAML file. **Test spec shell script** An intermediate shell script file created from your YAML file. Because it is used in the test run, the shell script file can be used for debugging the YAML file. **Test spec file** The YAML file used in the test run. For more information, see [Downloading artifacts in Device Farm](artifacts.md). ![\[Device Farm hierarchy of operations in a custom test environment\]](http://docs.aws.amazon.com/devicefarm/latest/developerguide/images/hierarchy.png)