# Test runs in AWS Device Farm
A run in Device Farm represents a specific build of your app, with a specific set of tests, to be run on a specific set of devices. A run produces a report that contains information about the results of the run. A run contains one or more jobs. For more information, see [Runs](test-runs.md).
You can use the AWS Device Farm console, AWS Command Line Interface (AWS CLI), or AWS Device Farm API to work with test runs.
**Topics**
+ [Creating a test run in Device Farm](how-to-create-test-run.md)
+ [Setting the execution timeout for test runs in AWS Device Farm](how-to-set-default-timeout-for-test-runs.md)
+ [Simulating network connections and conditions for your AWS Device Farm runs](how-to-simulate-network-connections-and-conditions.md)
+ [Stopping a run in AWS Device Farm](how-to-stop-test-runs.md)
+ [Viewing a list of runs in AWS Device Farm](how-to-view-runs-list.md)
+ [Creating a device pool in AWS Device Farm](how-to-create-device-pool.md)
+ [Analyzing test results in AWS Device Farm](analyzing-results.md)
# Creating a test run in Device Farm
You can use the Device Farm console, AWS CLI, or Device Farm API to create a test run. You can also use a supported plugin, such as the Jenkins or Gradle plugins for Device Farm. For more information about plugins, see [Tools and plugins](aws-device-farm-tools-plugins.md). For information about runs, see [Runs](test-runs.md).
**Topics**
+ [Prerequisites](#how-to-create-test-run-prerequisites)
+ [Create a test run (console)](#how-to-create-test-run-console)
+ [Create a test run (AWS CLI)](#how-to-create-test-run-cli)
+ [Create a test run (API)](#how-to-create-test-run-api)
+ [Next steps](#how-to-create-test-run-console-next-steps)
## Prerequisites
You must have a project in Device Farm. Follow the instructions in [Creating a project in AWS Device Farm](how-to-create-project.md), and then return to this page.
## Create a test run (console)
1. Sign in to the Device Farm console at [https://console.aws.amazon.com/devicefarm](https://console.aws.amazon.com/devicefarm).
1. In the navigation pane, choose **Mobile Device Testing**, and then choose **Projects**.
1. If you already have a project, you can upload your tests to it. Otherwise, choose **New project**, enter a **Project Name**, and then choose **Create**.
1. Open your project, and then choose **Create run**.
1. (Optional) Under **Run settings**, in the **Run name** section, enter a name for your run. If no name is provided, the Device Farm console will name your run 'My Device Farm run' by default.
1. (Optional) Under **Run settings**, in the **Job timeout** section, you can specify the execution timeout for your test run. If you're using unlimited testing slots, confirm that **Unmetered** is selected under **Billing method**.
1. Under **Run settings**, in the **Run type** section, select your run type. Select **Android app** if you do not have an app ready for testing, or if you are testing an android (.apk) app. Select **iOS app** if you are testing an iOS (.ipa) app. Select **Web app** if you want to test web applications.
1. Under **Select app**, in the **App selection options** section, choose **Select sample app provided by Device Farm** if you do not have an app available for testing. If you are bringing your own app, select **Upload own app**, and choose your application file. If you're uploading an iOS app, be sure to choose **iOS device**, as opposed to a simulator.
1. Under **Configure test**, choose one of the available test frameworks.
**Note**
If you don't have any tests available, choose **Built-in: Fuzz** to run a standard, built-in test suite. If you choose **Built-in: Fuzz**, and the **Event count**, **Event throttle**, and **Randomizer seed** boxes appear, you can change or keep the values.
For information about the available test suites, see [Test frameworks and built-in tests in AWS Device Farm](test-types.md).
1. If you didn't choose **Built-in: Fuzz**, select **Choose File** under **Select test package**. Browse to and choose the file that contains your tests.
1. For your testing environment, choose **Run your test in our standard environment** or **Run your test in a custom environment**. For more information, see [Test environments in AWS Device Farm](test-environments.md).
1. If you're using a custom test environment, you can optionally do the following:
+ If you want to edit the default test spec in a custom test environment, choose **Edit** to update the default YAML specification.
+ If you changed the test spec, choose **Save as New** to update it.
+ You may configure envirnonment variables. Variables supplied here will take precedence over any that may be configured on the parent project.
1. Under **Select devices**, do one of the following:
+ To choose a built-in device pool to run the tests against, for **Device pool**, choose **Top Devices**.
+ To create your own device pool to run the tests against, follow the instructions in [Creating a device pool](how-to-create-device-pool.md), and then return to this page.
+ If you created your own device pool earlier, for **Device pool**, choose your device pool.
+ Select **Manually select devices** and choose the desired devices you want to run against. This configuration will not be saved.
For more information, see [Device support in AWS Device FarmDevices](devices.md).
1. (Optional) To add additional configuration, open the **Additional configuration** dropdown. In this section, you can do any of the following:
+ To provide an execution role ARN, or override one configured on the parent project, use the Exectuion role ARN field.
+ To provide other data for Device Farm to use during the run, next to **Add extra data**, choose **Choose File**, and then browse to and choose the .zip file that contains the data.
+ To install an additional app for Device Farm to use during the run, next to **Install other apps**, choose **Choose File**, and then browse to and choose the .apk or .ipa file that contains the app. Repeat this for other apps you want to install. You can change the installation order by dragging and dropping the apps after you upload them.
+ To specify whether Wi-Fi, Bluetooth, GPS, or NFC is enabled during the run, next to **Set radio states**, select the appropriate boxes.
+ To preset the device latitude and longitude for the run, next to **Device location**, enter the coordinates.
+ To preset the device locale for the run, in **Device locale**, choose the locale.
+ Select **Enable video recording** to record video during testing.
+ Select **Enable app performance data capture** to capture performance data from the device.
**Note**
Setting the device radio state and locale are options only available for Android native tests at this time.
**Note**
If you have private devices, configuration specific to private devices is also displayed.
1. At the bottom of the page, choose **Create run** to schedule the run.
Device Farm starts the run as soon as devices are available, typically within a few minutes. During your test run, the Device Farm console displays a pending icon ![\[Device Farm scheduled a job.\]](http://docs.aws.amazon.com/devicefarm/latest/developerguide/images/df-run-calendar.png) in the run table. Each device in the run will also start with the pending icon, then switch to the running icon ![\[Device Farm progress indicator.\]](http://docs.aws.amazon.com/devicefarm/latest/developerguide/images/df-run-progress.png) when the test begins. As each test finishes, a test result icon is displayed next to the device name. When all tests have been completed, the pending icon next to the run changes to a test result icon.
If you want to stop the test run, see [Stopping a run in AWS Device Farm](how-to-stop-test-runs.md).
## Create a test run (AWS CLI)
You can use the AWS CLI to create a test run.
**Topics**
+ [Step 1: Choose a project](#how-to-create-test-run-cli-step1)
+ [Step 2: Choose a device pool](#how-to-create-test-run-cli-step2)
+ [Step 3: Upload your application file](#how-to-create-test-run-cli-step3)
+ [Step 4: Upload your test scripts package](#how-to-create-test-run-cli-step4)
+ [Step 5: (Optional) Upload your custom test spec](#how-to-create-test-run-cli-step5)
+ [Step 6: Schedule a test run](#how-to-create-test-run-cli-step6)
### Step 1: Choose a project
You must associate your test run with a Device Farm project.
1. To list your Device Farm projects, run **list-projects**. If you do not have a project, see [Creating a project in AWS Device Farm](how-to-create-project.md).
Example:
```
aws devicefarm list-projects
```
The response includes a list of your Device Farm projects.
```
{
"projects": [
{
"name": "MyProject",
"arn": "arn:aws:devicefarm:us-west-2:123456789101:project:5e01a8c7-c861-4c0a-b1d5-12345EXAMPLE",
"created": 1503612890.057
}
]
}
```
1. Choose a project to associate with your test run, and make a note of its Amazon Resource Name (ARN).
### Step 2: Choose a device pool
You must choose a device pool to associate with your test run.
1. To view your device pools, run **list-device-pools**, specifying your project ARN.
Example:
```
aws devicefarm list-device-pools --arn arn:MyProjectARN
```
The response includes the built-in Device Farm device pools, such as **Top Devices**, and any device pools previously created for this project:
```
{
"devicePools": [
{
"rules": [
{
"attribute": "ARN",
"operator": "IN",
"value": "[\"arn:aws:devicefarm:us-west-2::device:example1\",\"arn:aws:devicefarm:us-west-2::device:example2\",\"arn:aws:devicefarm:us-west-2::device:example3\"]"
}
],
"type": "CURATED",
"name": "Top Devices",
"arn": "arn:aws:devicefarm:us-west-2::devicepool:example",
"description": "Top devices"
},
{
"rules": [
{
"attribute": "PLATFORM",
"operator": "EQUALS",
"value": "\"ANDROID\""
}
],
"type": "PRIVATE",
"name": "MyAndroidDevices",
"arn": "arn:aws:devicefarm:us-west-2:605403973111:devicepool:example2"
}
]
}
```
1. Choose a device pool, and make a note of its ARN.
You can also create a device pool, and then return to this step. For more information, see [Create a device pool (AWS CLI)](how-to-create-device-pool.md#how-to-create-device-pool-cli).
### Step 3: Upload your application file
To create your upload request and get an Amazon Simple Storage Service (Amazon S3) presigned upload URL, you need:
+ Your project ARN.
+ The name of your app file.
+ The type of the upload.
For more information, see [https://docs.aws.amazon.com/cli/latest/reference/devicefarm/create-upload.html](https://docs.aws.amazon.com/cli/latest/reference/devicefarm/create-upload.html).
1. To upload a file, run **create-upload** with the `–-project-arn`, `--name`, and `--type` parameters.
This example creates an upload for an Android app:
```
aws devicefarm create-upload -–project-arn arn:MyProjectArn -–name MyAndroid.apk -–type ANDROID_APP
```
The response includes your app upload ARN and a presigned URL.
```
{
"upload": {
"status": "INITIALIZED",
"name": "MyAndroid.apk",
"created": 1535732625.964,
"url": "https://prod-us-west-2-uploads.s3-us-west-2.amazonaws.com/ExampleURL",
"type": "ANDROID_APP",
"arn": "arn:aws:devicefarm:us-west-2:123456789101:upload:5e01a8c7-c861-4c0a-b1d5-12345EXAMPLE"
}
}
```
1. Make a note of the app upload ARN and the presigned URL.
1. Upload your app file using the Amazon S3 presigned URL. This example uses **curl** to upload an Android .apk file:
```
curl -T MyAndroid.apk "https://prod-us-west-2-uploads.s3-us-west-2.amazonaws.com/ExampleURL"
```
For more information, see [Uploading objects using presigned URLs](https://docs.aws.amazon.com/AmazonS3/latest/userguide/PresignedUrlUploadObject.html) in the *Amazon Simple Storage Service User Guide*.
1. To check the status of your app upload, run **get-upload** and specify the ARN of the app upload.
```
aws devicefarm get-upload –-arn arn:MyAppUploadARN
```
Wait until the status in the response is **SUCCEEDED** before you upload your test scripts package.
```
{
"upload": {
"status": "SUCCEEDED",
"name": "MyAndroid.apk",
"created": 1535732625.964,
"url": "https://prod-us-west-2-uploads.s3-us-west-2.amazonaws.com/ExampleURL",
"type": "ANDROID_APP",
"arn": "arn:aws:devicefarm:us-west-2:123456789101:upload:5e01a8c7-c861-4c0a-b1d5-12345EXAMPLE",
"metadata": "{"valid": true}"
}
}
```
### Step 4: Upload your test scripts package
Next, you upload your test scripts package.
1. To create your upload request and get an Amazon S3 presigned upload URL, run **create-upload** with the `–-project-arn`, `--name`, and `--type` parameters.
This example creates an Appium Java TestNG test package upload:
```
aws devicefarm create-upload –-project-arn arn:MyProjectARN -–name MyTests.zip –-type APPIUM_JAVA_TESTNG_TEST_PACKAGE
```
The response includes your test package upload ARN and a presigned URL.
```
{
"upload": {
"status": "INITIALIZED",
"name": "MyTests.zip",
"created": 1535738627.195,
"url": "https://prod-us-west-2-uploads.s3-us-west-2.amazonaws.com/ExampleURL",
"type": "APPIUM_JAVA_TESTNG_TEST_PACKAGE",
"arn": "arn:aws:devicefarm:us-west-2:123456789101:upload:5e01a8c7-c861-4c0a-b1d5-12345EXAMPLE"
}
}
```
1. Make a note of the ARN of the test package upload and the presigned URL.
1. Upload your test scripts package file using the Amazon S3 presigned URL. This example uses **curl** to upload a zipped Appium TestNG scripts file:
```
curl -T MyTests.zip "https://prod-us-west-2-uploads.s3-us-west-2.amazonaws.com/ExampleURL"
```
1. To check the status of your test scripts package upload, run **get-upload** and specify the ARN of the test package upload from step 1.
```
aws devicefarm get-upload –-arn arn:MyTestsUploadARN
```
Wait until the status in the response is **SUCCEEDED** before you continue to the next, optional step.
```
{
"upload": {
"status": "SUCCEEDED",
"name": "MyTests.zip",
"created": 1535738627.195,
"url": "https://prod-us-west-2-uploads.s3-us-west-2.amazonaws.com/ExampleURL",
"type": "APPIUM_JAVA_TESTNG_TEST_PACKAGE",
"arn": "arn:aws:devicefarm:us-west-2:123456789101:upload:5e01a8c7-c861-4c0a-b1d5-12345EXAMPLE",
"metadata": "{"valid": true}"
}
}
```
### Step 5: (Optional) Upload your custom test spec
If you're running your tests in a standard test environment, skip this step.
Device Farm maintains a default test spec file for each supported test type. Next, you download your default test spec and use it to create a custom test spec upload for running your tests in a custom test environment. For more information, see [Test environments in AWS Device Farm](test-environments.md).
1. To find the upload ARN for your default test spec, run **list-uploads** and specify your project ARN.
```
aws devicefarm list-uploads --arn arn:MyProjectARN
```
The response contains an entry for each default test spec:
```
{
"uploads": [
{
{
"status": "SUCCEEDED",
"name": "Default TestSpec for Android Appium Java TestNG",
"created": 1529498177.474,
"url": "https://prod-us-west-2-uploads.s3-us-west-2.amazonaws.com/ExampleURL",
"type": "APPIUM_JAVA_TESTNG_TEST_SPEC",
"arn": "arn:aws:devicefarm:us-west-2:123456789101:upload:5e01a8c7-c861-4c0a-b1d5-12345EXAMPLE"
}
}
]
}
```
1. Choose your default test spec from the list. Make a note of its upload ARN.
1. To download your default test spec, run **get-upload** and specify the upload ARN.
Example:
```
aws devicefarm get-upload –-arn arn:MyDefaultTestSpecARN
```
The response contains a presigned URL where you can download your default test spec.
1. This example uses **curl** to download the default test spec and save it as `MyTestSpec.yml`:
```
curl "https://prod-us-west-2-uploads.s3-us-west-2.amazonaws.com/ExampleURL" > MyTestSpec.yml
```
1. You can edit the default test spec to meet your testing requirements, and then use your modified test spec in future test runs. Skip this step to use the default test spec as-is in a custom test environment.
1. To create an upload of your custom test spec, run **create-upload**, specifying your test spec name, test spec type, and project ARN.
This example creates an upload for an Appium Java TestNG custom test spec:
```
aws devicefarm create-upload --name MyTestSpec.yml --type APPIUM_JAVA_TESTNG_TEST_SPEC --project-arn arn:MyProjectARN
```
The response includes the test spec upload ARN and presigned URL:
```
{
"upload": {
"status": "INITIALIZED",
"category": "PRIVATE",
"name": "MyTestSpec.yml",
"created": 1535751101.221,
"url": "https://prod-us-west-2-uploads.s3-us-west-2.amazonaws.com/ExampleURL",
"type": "APPIUM_JAVA_TESTNG_TEST_SPEC",
"arn": "arn:aws:devicefarm:us-west-2:123456789101:upload:5e01a8c7-c861-4c0a-b1d5-12345EXAMPLE"
}
}
```
1. Make a note of the ARN for the test spec upload and the presigned URL.
1. Upload your test spec file using the Amazon S3 presigned URL. This example uses **curl** to upload an Appium JavaTestNG test spec:
```
curl -T MyTestSpec.yml "https://prod-us-west-2-uploads.s3-us-west-2.amazonaws.com/ExampleURL"
```
1. To check the status of your test spec upload, run **get-upload** and specify the upload ARN.
```
aws devicefarm get-upload –-arn arn:MyTestSpecUploadARN
```
Wait until the status in the response is **SUCCEEDED** before you schedule your test run.
```
{
"upload": {
"status": "SUCCEEDED",
"name": "MyTestSpec.yml",
"created": 1535732625.964,
"url": "https://prod-us-west-2-uploads.s3-us-west-2.amazonaws.com/ExampleURL",
"type": "APPIUM_JAVA_TESTNG_TEST_SPEC",
"arn": "arn:aws:devicefarm:us-west-2:123456789101:upload:5e01a8c7-c861-4c0a-b1d5-12345EXAMPLE",
"metadata": "{"valid": true}"
}
}
```
To update your custom test spec, run **update-upload**, specifying the upload ARN for the test spec. For more information, see [https://docs.aws.amazon.com/cli/latest/reference/devicefarm/update-upload.html](https://docs.aws.amazon.com/cli/latest/reference/devicefarm/update-upload.html).
### Step 6: Schedule a test run
To schedule a test run with the AWS CLI, run **schedule-run**, specifying:
+ The project ARN from [step 1](#how-to-create-test-run-cli-step1).
+ The device pool ARN from [step 2](#how-to-create-test-run-cli-step2).
+ The app upload ARN from [step 3](#how-to-create-test-run-cli-step3).
+ The test package upload ARN from [step 4](#how-to-create-test-run-cli-step4).
If you are running tests in a custom test environment, you also need your test spec ARN from [step 5](#how-to-create-test-run-cli-step5).
**To schedule a run in a standard test environment**
+ Run **schedule-run**, specifying your project ARN, device pool ARN, application upload ARN, and test package information.
Example:
```
aws devicefarm schedule-run --project-arn arn:MyProjectARN --app-arn arn:MyAppUploadARN --device-pool-arn arn:MyDevicePoolARN --name MyTestRun --test type=APPIUM_JAVA_TESTNG,testPackageArn=arn:MyTestPackageARN
```
The response contains a run ARN that you can use to check the status of your test run.
```
{
"run": {
"status": "SCHEDULING",
"appUpload": "arn:aws:devicefarm:us-west-2:123456789101:upload:5e01a8c7-c861-4c0a-b1d5-12345appEXAMPLE",
"name": "MyTestRun",
"radios": {
"gps": true,
"wifi": true,
"nfc": true,
"bluetooth": true
},
"created": 1535756712.946,
"totalJobs": 179,
"completedJobs": 0,
"platform": "ANDROID_APP",
"result": "PENDING",
"devicePoolArn": "arn:aws:devicefarm:us-west-2:123456789101:devicepool:5e01a8c7-c861-4c0a-b1d5-12345devicepoolEXAMPLE",
"jobTimeoutMinutes": 150,
"billingMethod": "METERED",
"type": "APPIUM_JAVA_TESTNG",
"testSpecArn": "arn:aws:devicefarm:us-west-2:123456789101:upload:5e01a8c7-c861-4c0a-b1d5-12345specEXAMPLE",
"arn": "arn:aws:devicefarm:us-west-2:123456789101:run:5e01a8c7-c861-4c0a-b1d5-12345runEXAMPLE",
"counters": {
"skipped": 0,
"warned": 0,
"failed": 0,
"stopped": 0,
"passed": 0,
"errored": 0,
"total": 0
}
}
}
```
For more information, see [https://docs.aws.amazon.com/cli/latest/reference/devicefarm/schedule-run.html](https://docs.aws.amazon.com/cli/latest/reference/devicefarm/schedule-run.html).
**To schedule a run in a custom test environment**
+ The steps are the almost the same as those for the standard test environment, with an additional `testSpecArn` attribute in the `--test` parameter.
Example:
```
aws devicefarm schedule-run --project-arn arn:MyProjectARN --app-arn arn:MyAppUploadARN --device-pool-arn arn:MyDevicePoolARN --name MyTestRun --test testSpecArn=arn:MyTestSpecUploadARN,type=APPIUM_JAVA_TESTNG,testPackageArn=arn:MyTestPackageARN
```
**To check the status of your test run**
+ Use the **get-run** command and specify the run ARN:
```
aws devicefarm get-run --arn arn:aws:devicefarm:us-west-2:111122223333:run:5e01a8c7-c861-4c0a-b1d5-12345runEXAMPLE
```
For more information, see [https://docs.aws.amazon.com/cli/latest/reference/devicefarm/get-run.html](https://docs.aws.amazon.com/cli/latest/reference/devicefarm/get-run.html). For information about using Device Farm with the AWS CLI, see [AWS CLI reference](cli-ref.md).
## Create a test run (API)
The steps are the same as those described in the AWS CLI section. See [Create a test run (AWS CLI)](#how-to-create-test-run-cli).
You need this information to call the [https://docs.aws.amazon.com/devicefarm/latest/APIReference/API_ScheduleRun.html](https://docs.aws.amazon.com/devicefarm/latest/APIReference/API_ScheduleRun.html) API:
+ A project ARN. See [Create a project (API)](how-to-create-project.md#how-to-create-project-api) and [https://docs.aws.amazon.com/devicefarm/latest/APIReference/API_CreateProject.html](https://docs.aws.amazon.com/devicefarm/latest/APIReference/API_CreateProject.html).
+ An application upload ARN. See [https://docs.aws.amazon.com/devicefarm/latest/APIReference/API_CreateUpload.html](https://docs.aws.amazon.com/devicefarm/latest/APIReference/API_CreateUpload.html).
+ A test package upload ARN. See [https://docs.aws.amazon.com/devicefarm/latest/APIReference/API_CreateUpload.html](https://docs.aws.amazon.com/devicefarm/latest/APIReference/API_CreateUpload.html).
+ A device pool ARN. See [Creating a device pool](how-to-create-device-pool.md) and [https://docs.aws.amazon.com/devicefarm/latest/APIReference/API_CreateDevicePool.html](https://docs.aws.amazon.com/devicefarm/latest/APIReference/API_CreateDevicePool.html).
**Note**
If you're running tests in a custom test environment, you also need your test spec upload ARN. For more information, see [Step 5: (Optional) Upload your custom test spec](#how-to-create-test-run-cli-step5) and [https://docs.aws.amazon.com/devicefarm/latest/APIReference/API_CreateUpload.html](https://docs.aws.amazon.com/devicefarm/latest/APIReference/API_CreateUpload.html).
For information about using the Device Farm API, see [Automating Device Farm](api-ref.md).
## Next steps
In the Device Farm console, the clock icon ![\[Device Farm scheduled a job.\]](http://docs.aws.amazon.com/devicefarm/latest/developerguide/images/df-run-calendar.png) changes to a result icon such as success ![\[The test succeeded.\]](http://docs.aws.amazon.com/devicefarm/latest/developerguide/images/df-run-success.png) when the run is complete. A report for the run appears as soon as tests are complete. For more information, see [Reports in AWS Device FarmReports](reports.md).
To use the report, follow the instructions in [Viewing test reports in Device Farm](how-to-use-reports.md).
# Setting the execution timeout for test runs in AWS Device Farm
You can set a value for how long a test run should execute before you stop each device from running a test. The default execution timeout is 150 minutes per device, but you can set a value as low as 5 minutes. You can use the AWS Device Farm console, AWS CLI, or AWS Device Farm API to set the execution timeout.
**Important**
The execution timeout option should be set to the *maximum duration* for a test run, along with some buffer. For example, if your tests take 20 minutes per device, you should choose a timeout of 30 minutes per device.
If the execution exceeds your timeout, the execution on that device is forcibly stopped. Partial results are available, if possible. You are billed for execution up to that point, if you're using the metered billing option. For more information about pricing, see [Device Farm Pricing](https://aws.amazon.com/device-farm/pricing/).
You might want to use this feature if you know how long a test run is supposed to take to execute on each device. When you specify an execution timeout for a test run, you can avoid the situation where a test run is stuck for some reason and you are being billed for device minutes when no tests are being executed. In other words, using the execution timeout feature lets you stop that run if it's taking longer than expected.
You can set the execution timeout in two places, at the project level and the test run level.
## Prerequisites
1. Complete the steps in [Setting up](setting-up.md).
1. Create a project in Device Farm. Follow the instructions in [Creating a project in AWS Device Farm](how-to-create-project.md), and then return to this page.
## Set the execution timeout for a project
1. Sign in to the Device Farm console at [https://console.aws.amazon.com/devicefarm](https://console.aws.amazon.com/devicefarm).
1. On the Device Farm navigation panel, choose **Mobile Device Testing**, then choose **Projects**.
1. If you already have a project, choose it from the list. Otherwise, choose **New project**, enter a name for your project, then choose **Submit**.
1. Choose **Project settings**.
1. On the **General** tab, for **Execution timeout**, enter a value or use the slider bar.
1. Choose **Save**.
All the test runs in your project now use the execution timeout value that you specified, unless you override the timeout value when you schedule a run.
## Set the execution timeout for a test run
1. Sign in to the Device Farm console at [https://console.aws.amazon.com/devicefarm](https://console.aws.amazon.com/devicefarm).
1. On the Device Farm navigation panel, choose **Mobile Device Testing**, then choose **Projects**.
1. If you already have a project, choose it from the list. Otherwise, choose **New project**, enter a name for your project, then choose **Submit**.
1. Choose **Create a new run**.
1. Follow the steps to choose an application, configure your test, select your devices, and specify a device state.
1. On **Review and start run**, for **Set execution timeout**, enter a value or use the slider bar.
1. Choose **Confirm and start run**.
# Simulating network connections and conditions for your AWS Device Farm runs
You can use network shaping to simulate network connections and conditions while testing your Android, iOS and web apps in Device Farm. For example, you can simulate lossy or intermittent internet connectivity.
When you create a run using the default network settings, each device has a full, unhindered Wi-Fi connection with internet connectivity. When you use network shaping, you can change the Wi-Fi connection to specify a network profile like **3G** or **Lossy WiFi** that controls throughput, delay, jitter, and loss for both inbound and outbound traffic.
**Topics**
+ [Set up network shaping when scheduling a test run](#network-shaping-how-to-choose-a-curated-profile-when-scheduling-a-test-run)
+ [Create a network profile](#network-shaping-how-to-create-a-network-profile)
+ [Change network conditions during your test](#change-network-conditions-during-test)
## Set up network shaping when scheduling a test run
When you schedule a run, you can choose from any of the Device Farm-curated profiles, or you can create and manage your own.
1. From any Device Farm project, choose **Create a new run**.
If you don't have a project yet, see [Creating a project in AWS Device Farm](how-to-create-project.md).
1. Choose your application, and then choose **Next**.
1. Configure your test, and then choose **Next**.
1. Select your devices, and then choose **Next**.
1. In the **Location and network settings** section, choose a network profile or choose **Create network profile** to create your own.
![\[Network profile for a test run\]](http://docs.aws.amazon.com/devicefarm/latest/developerguide/images/aws-device-farm-set-up-network-profile.png)
1. Choose **Next**.
1. Review and start your test run.
## Create a network profile
When you create a test run, you can create a network profile.
1. Choose **Create network profile**.
![\[Create a new network profile\]](http://docs.aws.amazon.com/devicefarm/latest/developerguide/images/aws-device-farm-create-a-new-network-profile.png)
1. Enter a name and settings for your network profile.
1. Choose **Create**.
1. Finish creating your test run and start the run.
After you have created a network profile, you'll be able to see and manage it on the **Project settings** page.
![\[Network profiles in Project settings\]](http://docs.aws.amazon.com/devicefarm/latest/developerguide/images/aws-device-farm-network-profiles-in-project-settings.png)
## Change network conditions during your test
You can call an API from your device host using a framework like Appium to simulate dynamic network conditions such as reduced bandwidth during your test run. For more information, see [CreateNetworkProfile](https://docs.aws.amazon.com/devicefarm/latest/APIReference/API_CreateNetworkProfile.html).
# Stopping a run in AWS Device Farm
You might want to stop a run after you have started it. For example, if you notice an issue while your tests are running you might want to restart the run with an updated test script.
You can use the Device Farm console, AWS CLI, or API to stop a run.
**Topics**
+ [Stop a run (console)](#how-to-stop-run-console)
+ [Stop a run (AWS CLI)](#how-to-stop-test-run-cli)
+ [Stop a run (API)](#how-to-stop-test-run-api)
## Stop a run (console)
1. Sign in to the Device Farm console at [https://console.aws.amazon.com/devicefarm](https://console.aws.amazon.com/devicefarm).
1. On the Device Farm navigation panel, choose **Mobile Device Testing**, then choose **Projects**.
1. Choose the project where you have an active test run.
1. On the **Automated tests** page, choose the test run.
The pending or running icon should appear to the left of the device name.
![\[Device Farm - Stop a test run\]](http://docs.aws.amazon.com/devicefarm/latest/developerguide/images/aws-device-farm-stop-run.png)
1. Choose **Stop run**.
After a short time, an icon with a red circle with a minus inside it appears next to the device name. When the run has been stopped, the icon color changes from red to black.
**Important**
If a test has already been run, Device Farm cannot stop it. If a test is in progress, Device Farm stops the test. The total minutes for which you will be billed appears in the **Devices** section. In addition, you will also be billed for the total minutes that Device Farm takes to run the setup suite and the teardown suite. For more information, see [Device Farm Pricing](http://aws.amazon.com/device-farm/faq/#pricing).
The following image shows an example **Devices** section after a test run was successfully stopped.
![\[Device Farm - Details page of a stopped run\]](http://docs.aws.amazon.com/devicefarm/latest/developerguide/images/aws-device-farm-stop-run-view-details.png)
## Stop a run (AWS CLI)
You can run the following command to stop the specified test run, where *myARN* is the Amazon Resource Name (ARN) of the test run.
```
$ aws devicefarm stop-run --arn myARN
```
You should see output similar to the following:
```
{
"run": {
"status": "STOPPING",
"name": "Name of your run",
"created": 1458329687.951,
"totalJobs": 7,
"completedJobs": 5,
"deviceMinutes": {
"unmetered": 0.0,
"total": 0.0,
"metered": 0.0
},
"platform": "ANDROID_APP",
"result": "PENDING",
"billingMethod": "METERED",
"type": "BUILTIN_EXPLORER",
"arn": "myARN",
"counters": {
"skipped": 0,
"warned": 0,
"failed": 0,
"stopped": 0,
"passed": 0,
"errored": 0,
"total": 0
}
}
}
```
To get the ARN of your run, use the `list-runs` command. The output should be similar to the following:
```
{
"runs": [
{
"status": "RUNNING",
"name": "Name of your run",
"created": 1458329687.951,
"totalJobs": 7,
"completedJobs": 5,
"deviceMinutes": {
"unmetered": 0.0,
"total": 0.0,
"metered": 0.0
},
"platform": "ANDROID_APP",
"result": "PENDING",
"billingMethod": "METERED",
"type": "BUILTIN_EXPLORER",
"arn": "Your ARN will be here",
"counters": {
"skipped": 0,
"warned": 0,
"failed": 0,
"stopped": 0,
"passed": 0,
"errored": 0,
"total": 0
}
}
]
}
```
For information about using Device Farm with the AWS CLI, see [AWS CLI reference](cli-ref.md).
## Stop a run (API)
+ Call the [StopRun](../../latest/APIReference/API_StopRun.html) operation to the test run.
For information about using the Device Farm API, see [Automating Device Farm](api-ref.md).
# Viewing a list of runs in AWS Device Farm
You can use the Device Farm console, AWS CLI, or API to view a list of runs for a project.
**Topics**
+ [View a list of runs (console)](#how-to-view-runs-list-console)
+ [View a list of runs (AWS CLI)](#how-to-view-runs-list-cli)
+ [View a list of runs (API)](#how-to-view-runs-list-api)
## View a list of runs (console)
1. Sign in to the Device Farm console at [https://console.aws.amazon.com/devicefarm](https://console.aws.amazon.com/devicefarm).
1. On the Device Farm navigation panel, choose **Mobile Device Testing**, then choose **Projects**.
1. In the list of projects, choose the project that corresponds to the list you want to view.
**Tip**
You can use the search bar to filter the project list by name.
## View a list of runs (AWS CLI)
+ Run the [https://docs.aws.amazon.com/cli/latest/reference/devicefarm/list-runs.html](https://docs.aws.amazon.com/cli/latest/reference/devicefarm/list-runs.html) command.
To view information about a single run, run the [https://docs.aws.amazon.com/cli/latest/reference/devicefarm/get-run.html](https://docs.aws.amazon.com/cli/latest/reference/devicefarm/get-run.html) command.
For information about using Device Farm with the AWS CLI, see [AWS CLI reference](cli-ref.md).
## View a list of runs (API)
+ Call the [https://docs.aws.amazon.com/devicefarm/latest/APIReference/API_ListRuns.html](https://docs.aws.amazon.com/devicefarm/latest/APIReference/API_ListRuns.html) API.
To view information about a single run, call the [https://docs.aws.amazon.com/devicefarm/latest/APIReference/API_GetRun.html](https://docs.aws.amazon.com/devicefarm/latest/APIReference/API_GetRun.html) API.
For information about the Device Farm API, see [Automating Device Farm](api-ref.md).
# Creating a device pool in AWS Device Farm
You can use the Device Farm console, AWS CLI, or API to create a device pool.
**Topics**
+ [Prerequisites](#how-to-create-device-pool-prerequisites)
+ [Create a device pool (console)](#how-to-create-device-pool-console)
+ [Create a device pool (AWS CLI)](#how-to-create-device-pool-cli)
+ [Create a device pool (API)](#how-to-create-device-pool-api)
## Prerequisites
+ Create a run in the Device Farm console. Follow the instructions in [Creating a test run in Device Farm](how-to-create-test-run.md). When you get to the **Select devices** page, continue with the instructions in this section.
## Create a device pool (console)
1. On the **Projects** page, choose your project. In the **Project details** page, choose **Project settings**. In the **Device pools** tab, choose **Create device pool**.
1. For **Name**, enter a name that makes this device pool easy to identify.
1. For **Description**, enter a description that makes this device pool easy to identify.
1. If you want to use one or more selection criteria for the devices in this device pool, do the following:
1. Choose **Create dynamic device pool**.
1. Choose **Add a rule**.
1. For **Field** (first drop-down list), choose one of the following:
+ To include devices by their manufacturer name, choose **Device Manufacturer**.
+ To include devices by their form factor (tablet or phone), choose **Form Factor**.
+ To include devices by their availability status based on load, choose **Availability**.
+ To include only public or private devices, choose **Fleet Type**.
+ To include devices by their operating system, choose **Platform**.
+ Some devices have an additional label tag or description about the device. You can find devices based on their label contents by choosing **Instance labels**.
+ To include devices by their operating system version, choose **OS Version**.
+ To include devices by their model, choose **Model**.
1. For **Operator** (second drop-down list), choose a logical operation (EQUALS, CONTAINS, etc.) to include devices based on the query. For example, you could choose *Availability EQUALS AVAILABLE* to include devices that currently have the `Available` status.
1. For **Value** (third drop-down list), enter or choose the value you want to specify for the **Field** and **Operator** values. Values are limited based on your **Field** choice. For example, if you choose **Platform** for **Field**, the only available selections are **ANDROID** and **IOS**. Similarly, if you choose **Form Factor** for **Field**, the only available selections are **PHONE** and **TABLET**.
1. To add another rule, choose **Add a rule**.
After you create the first rule, in the list of devices, the box next to each device that matches the rule is selected. After you create or change rules, in the list of devices, the box next to each device that matches those combined rules is selected. Devices with selected boxes are included in the device pool. Devices with cleared boxes are excluded.
1. Under **Max devices**, enter the number of devices you want to use in your device pool. If you do not enter the max number of devices, Device Farm will pick all devices in the fleet that match the rule(s) you created. To avoid additional charges, set this number to an amount that matches your actual parallel execution and device variety requirements.
1. To delete a rule, choose **Remove rule**.
1. If you want to manually include or exclude individual devices, do the following:
1. Choose **Create static device pool**.
1. Select or clear the box next to each device. You can select or clear the boxes only if you do not have any rules specified.
1. If you want to include or exclude all displayed devices, select or clear the box in the column header row of the list. If you want to view private device instances only, choose **See private device instances only**.
**Important**
Although you can use the boxes in the column header row to change the list of displayed devices, this does not mean that the remaining displayed devices are the only ones included or excluded. To confirm which devices are included or excluded, be sure to clear the contents of all of the boxes in the column header row, and then browse the boxes.
1. Choose **Create**.
## Create a device pool (AWS CLI)
**Tip**
If you do not enter the max number of devices, Device Farm will pick all devices in the fleet that match the rule(s) you created. To avoid additional charges, set this number to an amount that matches your actual parallel execution and device variety requirements.
+ Run the [https://docs.aws.amazon.com/cli/latest/reference/devicefarm/create-device-pool.html](https://docs.aws.amazon.com/cli/latest/reference/devicefarm/create-device-pool.html) command.
For information about using Device Farm with the AWS CLI, see [AWS CLI reference](cli-ref.md).
## Create a device pool (API)
**Tip**
If you do not enter the max number of devices, Device Farm will pick all devices in the fleet that match the rule(s) you created. To avoid additional charges, set this number to an amount that matches your actual parallel execution and device variety requirements.
+ Call the [https://docs.aws.amazon.com/devicefarm/latest/APIReference/API_CreateDevicePool.html](https://docs.aws.amazon.com/devicefarm/latest/APIReference/API_CreateDevicePool.html) API.
For information about using the Device Farm API, see [Automating Device Farm](api-ref.md).
# Analyzing test results in AWS Device Farm
In the standard test environment, you can use the Device Farm console to view reports for each test in your test run. Vewing the reports helps you understand which tests passed or failed, and provides you with details on the performance and behavior of your app across different device configurations.
Device Farm also gathers other artifacts such as files, logs, and images that you can download when your test run is complete. This information can help you analyze how your app is behaving on real devices, identify issues or bugs, and diagnose problems.
**Topics**
+ [Viewing test reports in Device Farm](how-to-use-reports.md)
+ [Downloading artifacts in Device Farm](artifacts.md)
# Viewing test reports in Device Farm
Use the Device Farm console to view your test reports. For more information, see [Reports in AWS Device FarmReports](reports.md).
**Topics**
+ [Prerequisites](#how-to-use-reports-prerequisites)
+ [View reports](#how-to-use-reports-viewing-reports)
+ [Device Farm test result statuses](how-to-use-reports-displaying-results.md)
## Prerequisites
Set up a test run and verify that it is complete.
1. To create a run, see [Creating a test run in Device Farm](how-to-create-test-run.md), and then return to this page.
1. Verify that the run is complete. During your test run, the Device Farm console displays a pending icon ![\[Device Farm scheduled a job.\]](http://docs.aws.amazon.com/devicefarm/latest/developerguide/images/df-run-calendar.png) for runs that are in progress. Each device in the run will also start with the pending icon, then switch to the running ![\[Device Farm progress indicator.\]](http://docs.aws.amazon.com/devicefarm/latest/developerguide/images/df-run-progress.png) icon when the test begins. As each test finishes, a test result icon is displayed next to the device name. When all tests have been completed, the pending icon next to the run changes to a test result icon. For more information, see [Device Farm test result statuses](how-to-use-reports-displaying-results.md).
## View reports
You can view the results of your test in the Device Farm console.
**Topics**
+ [View the test run summary page](#how-to-use-reports-console-summary)
+ [View unique problem reports](#how-to-use-reports-console-unique-problems)
+ [View device reports](#how-to-use-reports-console-by-device)
+ [View test suite reports](#how-to-use-reports-console-by-suite)
+ [View test reports](#how-to-use-reports-console-by-test)
+ [View log information for a problem, device, suite, or test in a report](#how-to-use-reports-console-log)
### View the test run summary page
1. Sign in to the Device Farm console at [https://console.aws.amazon.com/devicefarm](https://console.aws.amazon.com/devicefarm).
1. In the navigation pane, choose **Mobile Device Testing**, and then choose **Projects**.
1. In the list of projects, choose the project for the run.
**Tip**
To filter the project list by name, use the search bar.
1. Choose a completed run to view its summary report page.
1. The test run summary page displays an overview of your test results.
+ The **Unique problems** section lists unique warnings and failures. To view unique problems, follow the instructions in [View unique problem reports](#how-to-use-reports-console-unique-problems).
+ The **Devices** section displays the total number of tests, by outcome, for each device.
![\[Device Farm device summary results.\]](http://docs.aws.amazon.com/devicefarm/latest/developerguide/images/df-run-device-summary-results-bar.png)
In this example, there are several devices. In the first table entry, the Google Pixel 4 XL device running Android version 10 reports three successful tests that took 02:36 minutes to run.
To view the results by device, follow the instructions in [View device reports](#how-to-use-reports-console-by-device).
+ The **Screenshots** section displays a list of any screenshots that Device Farm captured during the run, grouped by device.
+ In the **Parsing result** section, you can download the parsing result.
### View unique problem reports
1. In **Unique problems**, choose the problem that you want to view.
1. Choose the device. The report displays information about the problem.
The **Video** section displays a downloadable video recording of the test.
The **Result** section displays the result of the test. The status is represented as a result icon. For more information, see [Statuses of an individual test](how-to-use-reports-displaying-results.md#how-to-use-reports-displaying-results-individual).
The **Logs** section displays any information that Device Farm logged during the test. To view this information, follow the instructions in [View log information for a problem, device, suite, or test in a report](#how-to-use-reports-console-log).
The **Files** tab displays a list of any of the test's associated files (such as log files) that you can download. To download a file, choose the file's link in the list.
The **Screenshots** tab displays a list of any screenshots that Device Farm captured during the test.
### View device reports
+ In the **Devices** section, choose the device.
The **Video** section displays a downloadable video recording of the test.
The **Suites** section displays a table containing information about the suites for the device.
In this table, the **Test results** column summarizes the number of tests by outcome for each of the test suites that have run on the device. This data also has a graphical component. For more information, see [Statuses for multiple tests](how-to-use-reports-displaying-results.md#how-to-use-reports-displaying-results-summary).
To view the full results by suite, follow the instructions in [View test suite reports](#how-to-use-reports-console-by-suite).
The **Logs** section displays any information that Device Farm logged for the device during the run. To view this information, follow the instructions in [View log information for a problem, device, suite, or test in a report](#how-to-use-reports-console-log).
The **Files** section displays a list of suites for the device and any associated files (such as log files) that you can download. To download a file, choose the file's link in the list.
The **Screenshots** section displays a list of any screenshots that Device Farm captured during the run for the device, grouped by suite.
### View test suite reports
1. In the **Devices** section, choose the device.
1. In the **Suites** section, choose the suite from the table.
The **Video** section displays a downloadable video recording of the test.
The **Tests** section displays a table containing information about the tests in the suite.
In the table, the **Test results** column displays the result. This data also has a graphical component. For more information, see [Statuses for multiple tests](how-to-use-reports-displaying-results.md#how-to-use-reports-displaying-results-summary).
To view the full results by test, follow the instructions in [View test reports](#how-to-use-reports-console-by-test).
The **Logs** section displays any information that Device Farm logged during the run for the suite. To view this information, follow the instructions in [View log information for a problem, device, suite, or test in a report](#how-to-use-reports-console-log).
The **Files** section displays a list of tests for the suite and any associated files (such as log files) that you can download. To download a file, choose the file's link in the list.
The **Screenshots** section displays a list of any screenshots that Device Farm captured during the run for the suite, grouped by test.
### View test reports
1. In the **Devices** section, choose the device.
1. In the **Suites** section, choose the suite.
1. In the **Tests** section, choose the test.
1. The **Video** section displays a downloadable video recording of the test.
The **Result** section displays the result of the test. The status is represented as a result icon. For more information, see [Statuses of an individual test](how-to-use-reports-displaying-results.md#how-to-use-reports-displaying-results-individual).
The **Logs** section displays any information that Device Farm logged during the test. To view this information, follow the instructions in [View log information for a problem, device, suite, or test in a report](#how-to-use-reports-console-log).
The **Files** tab displays a list of any of the test's associated files (such as log files) that you can download. To download a file, choose the file's link in the list.
The **Screenshots** tab displays a list of any screenshots that Device Farm captured during the test.
### View log information for a problem, device, suite, or test in a report
The **Logs** section displays the following information:
+ **Source** represents the source of a log entry. Possible values include:
+ **Harness** represents a log entry that Device Farm created. These log entries are typically created during start and stop events.
+ **Device** represents a log entry that the device created. For Android, these log entries are logcat-compatible. For iOS, these log entries are syslog-compatible.
+ **Test** represents a log entry that either a test or its test framework created.
+ **Time** represents the elapsed time between the first log entry and this log entry. The time is expressed in *MM:SS.SSS* format, where *M* represents minutes and *S* represents seconds.
+ **PID** represents the process identifier (PID) that created the log entry. All log entries created by an app on a device have the same PID.
+ **Level** represents the logging level for the log entry. For example, `Logger.debug("This is a message!")` logs a **Level** of `Debug`. These are the possible values:
+ **Alert**
+ **Critical**
+ **Debug**
+ **Emergency**
+ **Error**
+ **Errored**
+ **Failed**
+ **Info**
+ **Internal**
+ **Notice**
+ **Passed**
+ **Skipped**
+ **Stopped**
+ **Verbose**
+ **Warned**
+ **Warning**
+ **Tag** represents arbitrary metadata for the log entry. For example, Android logcat can use this to describe which part of the system created the log entry (for example, `ActivityManager`).
+ **Message** represents the message or data for the log entry. For example, `Logger.debug("Hello, World!")` logs a **Message** of `"Hello, World!"`.
To display only a portion of the information:
+ To show all log entries that match a value for a specific column, enter the value into the search bar. For example, to show all log entries with a **Source** value of `Harness`, enter **Harness** in the search bar.
+ To remove all of the characters from a column header box, choose the **X** in that column header box. Removing all of the characters from a column header box is the same as entering **\$1** in that column header box.
To download all of the log information for the device, including all of the suites and tests that you ran, choose **Download logs**.
# Device Farm test result statuses
The Device Farm console displays icons that help you quickly assess the state of your completed test run. For more information about tests in Device Farm, see [Reports in AWS Device FarmReports](reports.md).
**Topics**
+ [Statuses of an individual test](#how-to-use-reports-displaying-results-individual)
+ [Statuses for multiple tests](#how-to-use-reports-displaying-results-summary)
## Statuses of an individual test
For reports that describe an individual test, Device Farm displays an icon representing the test result status:
| Description | Icon |
| --- | --- |
| The test succeeded. | ![\[The test succeeded.\]](http://docs.aws.amazon.com/devicefarm/latest/developerguide/images/df-run-success.png) |
| The test failed. | ![\[The test failed.\]](http://docs.aws.amazon.com/devicefarm/latest/developerguide/images/df-run-failure.png) |
| Device Farm skipped the test. | ![\[The test was skipped.\]](http://docs.aws.amazon.com/devicefarm/latest/developerguide/images/df-run-skipped.png) |
| The test stopped. | ![\[The test was stopped.\]](http://docs.aws.amazon.com/devicefarm/latest/developerguide/images/df-run-stopped.png) |
| Device Farm returned a warning. | ![\[Device Farm returned a warning.\]](http://docs.aws.amazon.com/devicefarm/latest/developerguide/images/df-run-warning.png) |
| Device Farm returned an error. | ![\[Device Farm returned an error.\]](http://docs.aws.amazon.com/devicefarm/latest/developerguide/images/df-run-error.png) |
## Statuses for multiple tests
If you choose a finished run, Device Farm displays a summary graph showing the percentage of tests in various states.
![\[Device Farm test results summary graph.\]](http://docs.aws.amazon.com/devicefarm/latest/developerguide/images/df-summary-results-graph.png)
For example, this test run results graph shows that the run had 4 stopped tests, 1 failed test, and 10 successful tests.
Graphs are always color coded and labeled.
# Downloading artifacts in Device Farm
Device Farm gathers artifacts such as reports, log files, and images for each test in the run.
You can download artifacts created during your test run:
**Files**
Files generated during the test run including Device Farm reports. For more information, see [Viewing test reports in Device Farm](how-to-use-reports.md).
**Logs**
Output from each test in the test run.
**Screenshots**
Screen images recorded for each test in the test run.
![\[Workflow diagram showing Project, Run, Job (device), Test suite, and Test stages in AWS Cloud.\]](http://docs.aws.amazon.com/devicefarm/latest/developerguide/images/hierarchy.png)
## Download artifacts (console)
1. On the test run report page, from **Devices**, choose a mobile device.
1. To download a file, choose one from **Files**.
1. To download the logs from your test run, from **Logs**, choose **Download logs**.
1. To download a screenshot, choose a screenshot from **Screenshots**.
For more information about downloading artifacts in a custom test environment, see [Downloading artifacts in a custom test environment](using-artifacts-custom.md).
## Download artifacts (AWS CLI)
You can use the AWS CLI to list your test run artifacts.
**Topics**
+ [Step 1: Get your Amazon Resource Names (ARN)](#artifacts-cli-step1)
+ [Step 2: List your artifacts](#artifacts-cli-step2)
+ [Step 3: Download your artifacts](#artifacts-cli-step3)
### Step 1: Get your Amazon Resource Names (ARN)
You can list your artifacts by run, job, test suite, or test. You need the corresponding ARN. This table shows the input ARN for each of the AWS CLI list commands:
| AWS CLI List Command | Required ARN |
| --- | --- |
| list-projects | This command returns all projects and does not require an ARN. |
| list-runs | project |
| list-jobs | run |
| list-suites | job |
| list-tests | suite |
For example, to find a test ARN, run **list-tests** using your test suite ARN as an input parameter.
Example:
```
aws devicefarm list-tests –-arn arn:MyTestSuiteARN
```
The response includes a test ARN for each test in the test suite.
```
{
"tests": [
{
"status": "COMPLETED",
"name": "Tests.FixturesTest.testExample",
"created": 1537563725.116,
"deviceMinutes": {
"unmetered": 0.0,
"total": 1.89,
"metered": 1.89
},
"result": "PASSED",
"message": "testExample passed",
"arn": "arn:aws:devicefarm:us-west-2:123456789101:test:5e01a8c7-c861-4c0a-b1d5-12345EXAMPLE",
"counters": {
"skipped": 0,
"warned": 0,
"failed": 0,
"stopped": 0,
"passed": 1,
"errored": 0,
"total": 1
}
}
]
}
```
### Step 2: List your artifacts
The AWS CLI [list-artifacts](https://docs.aws.amazon.com/cli/latest/reference/devicefarm/list-artifacts.html) command returns a list of artifacts, such as files, screenshots, and logs. Each artifact has a URL so you can download the file.
+ Call **list-artifacts** specifying a run, job, test suite, or test ARN. Specify a type of FILE, LOG, or SCREENSHOT.
This example returns a download URL for each artifact available for an individual test:
```
aws devicefarm list-artifacts --arn arn:MyTestARN --type "FILE"
```
The response contains a download URL for each artifact.
```
{
"artifacts": [
{
"url": "https://prod-us-west-2-uploads.s3-us-west-2.amazonaws.com/ExampleURL",
"extension": "txt",
"type": "APPIUM_JAVA_OUTPUT",
"name": "Appium Java Output",
"arn": "arn:aws:devicefarm:us-west-2:123456789101:artifact:5e01a8c7-c861-4c0a-b1d5-12345EXAMPLE",
}
]
}
```
### Step 3: Download your artifacts
+ Download your artifact using the URL from the previous step. This example uses **curl** to download an Android Appium Java output file:
```
curl "https://prod-us-west-2-uploads.s3-us-west-2.amazonaws.com/ExampleURL" > MyArtifactName.txt
```
## Download artifacts (API)
The Device Farm API [ListArtifacts](https://docs.aws.amazon.com/devicefarm/latest/APIReference/API_ListArtifacts.html) method returns a list of artifacts, such as files, screenshots, and logs. Each artifact has a URL so you can download the file.
# Downloading artifacts in a custom test environment
In a custom test environment, Device Farm gathers artifacts such as custom reports, log files, and images. These artifacts are available for each device in the test run.
You can download these artifacts created during your test run:
**Test spec output**
The output from running the commands in the test spec YAML file.
**Customer artifacts**
A zipped file that contains the artifacts from the test run. It is configured in the **artifacts:** section of your test spec YAML file.
**Test spec shell script**
An intermediate shell script file created from your YAML file. Because it is used in the test run, the shell script file can be used for debugging the YAML file.
**Test spec file**
The YAML file used in the test run.
For more information, see [Downloading artifacts in Device Farm](artifacts.md).
![\[Device Farm hierarchy of operations in a custom test environment\]](http://docs.aws.amazon.com/devicefarm/latest/developerguide/images/hierarchy.png)