# Custom test environments in AWS Device Farm AWS Device Farm enables configuring a custom environment for automated testing (custom mode), which is the recommended approach for all Device Farm users. To learn more about environments in Device Farm, see [Test environments](https://docs.aws.amazon.com/devicefarm/latest/developerguide/test-environments.html). Benefits of the Custom Mode as opposed to the Standard Mode include: + **Faster end-to-end test execution**: The test package isn't parsed to detect every test in the suite, avoiding preprocessing/postprocessing overhead. + **Live log and video streaming**: Your client-side test logs and video are live streamed when using Custom Mode. This feature isn't available in the standard mode. + **Captures all artifacts**: On the host and device, Custom Mode allows you to capture all test artifacts. This may not be possible in the standard mode. + **More consistent and replicable local environment**: When in Standard Mode, artifacts will be provided for each individual test separately, which can be beneficial under certain circumstances. However, your local test environment may deviate from the original configuration as Device Farm handles each executed test differently. In contrast, Custom Mode enables you to make your Device Farm test execution environment consistently in line with with your local test environment. Custom environments are configured using a YAML-formatted test specification (test spec) file. Device Farm provides a default test spec file for each supported test type that can be used as is or customized; customizations like test filters or config files can be added to the test spec. Edited test specs can be saved for future test runs. For more information, see [Uploading a Custom Test Spec Using the AWS CLI](https://docs.aws.amazon.com/devicefarm/latest/developerguide/how-to-create-test-run.html#how-to-create-test-run-cli-step5) and [Creating a test run in Device Farm](how-to-create-test-run.md). **Topics** + [ # Test spec reference and syntax ](custom-test-environment-test-spec.md) + [ # Hosts for custom test environments ](custom-test-environments-hosts.md) + [ # Access AWS resources using an IAM Execution Role ](custom-test-environments-iam-roles.md) + [ # Environment variables for custom test environments ](custom-test-environment-variables.md) + [ # Best practices for custom test environment execution ](custom-test-environments-best-practices.md) + [ # Migrating tests from a standard to custom test environment ](custom-test-environment-migration.md) + [ # Extending custom test environments in Device Farm ](custom-test-environments-extending.md) # Test spec reference and syntax The test spec (test specification) is a file that you use to define custom test environments in Device Farm. ## Test spec workflow The Device Farm test spec executes phases and their commands in a pre-determined order, letting you customize the way your environment is prepared and executed. When each phase is executed, its commands are run in the order listed within the test spec file. Phases are executed in the following sequence 1. `install` - This is where actions like downloading, installing, and setting up tooling should be defined. 1. `pre_test` - This is where pre-test actions like starting background processes should be defined. 1. `test` - This is where the command that invokes your test should be defined. 1. `post_test` - This is where any final tasks that need to be run after your test concludes should be defined, such as test report generation and artifact file aggregation. ## Test spec syntax The following is the YAML schema for a test spec file ``` version: 0.1 android_test_host: "string" ios_test_host: "string" phases: install: commands: - "string" - "string" pre_test: commands: - "string" - "string" test: commands: - "string" - "string" post_test: commands: - "string" - "string" artifacts: - "string" - "string" ``` ** `version` ** *(Required, number)* Reflects the Device Farm supported test spec version. The current version number is ` 0.1`. ** `android_test_host` ** *(Optional, string)* The test host that will be selected for test runs performed on Android devices. This field is required for test runs on Android devices. For more information, see [Available test hosts for custom test environments](custom-test-environments-hosts.md#custom-test-environments-hosts-available). ** `ios_test_host` ** *(Optional, string)* The test host that will be selected for test runs performed on iOS devices. This field is required for test runs on iOS devices with a major version greater than 26. For more information, see [Available test hosts for custom test environments](custom-test-environments-hosts.md#custom-test-environments-hosts-available). ** `phases` ** This section contains groups of commands executed during a test run, where each phase is optional The allowed test phase names are: `install`, `pre_test` , `test`, and `post_test`. + `install` - Default dependencies for testing frameworks supported by Device Farm are already installed. This phase contains additional commands, if any, that Device Farm runs during installation. + `pre_test` - The commands, if any, executed before your automated test. + `test` - The commands executed during your automated test run. If any command in the test phase fails (meaning it returns a non-zero exit code), the test is marked as failed + `post_test` - The commands, if any, executed after your automated test run. This will be executed whether or not your test in the `test` phase succeeds or fails. ** `commands` ** *(Optional, List[string])* A list of strings to execute as a shell command during the phase. ** `artifacts` ** *(Optional, List[string])* Device Farm gathers artifacts such as custom reports, log files, and images from a location specified here. Wildcard characters are not supported as part of an artifact location, so you must specify a valid path for each location. These test artifacts are available for each device in your test run. For information about retrieving your test artifacts, see [Downloading artifacts in a custom test environment](using-artifacts-custom.md). **Important** A test spec must be formatted as a valid YAML file. If the indenting or spacing in your test spec are invalid, your test run can fail. Tabs are not allowed in YAML files. You can use a YAML validator to test whether your test spec is a valid YAML file. For more information, see the [YAML website](http://yaml.org/spec/1.2/spec.html). ## Test spec examples The following examples show test specs that can be executed on Device Farm. ------ #### [ Simple Demo ] The following is an example test spec file that simply logs `Hello world!` as a test run artifact. ``` version: 0.1 android_test_host: amazon_linux_2 ios_test_host: macos_sequoia phases: install: commands: # Setup your environment by installing and/or validating software - devicefarm-cli use python 3.11 - python --version pre_test: commands: # Setup your tests by starting background tasks or setting up # additional environment variables. - OUTPUT_FILE="/tmp/hello.log" test: commands: # Run your tests within this phase. - python -c 'print("Hello world!")' &> $OUTPUT_FILE post_test: commands: # Perform any remaining tasks within this phase, such as copying # artifacts to the DEVICEFARM_LOG_DIR for upload - cp $OUTPUT_FILE $DEVICEFARM_LOG_DIR artifacts: # By default, Device Farm will collect your artifacts from the $DEVICEFARM_LOG_DIR directory. - $DEVICEFARM_LOG_DIR ``` ------ #### [ Appium Android ] The following is an example test spec file that configures an Appium Java TestNG test run on Android.. ``` version: 0.1 # The following fields(s) allow you to select which Device Farm test host is used for your test run. android_test_host: amazon_linux_2 phases: # The install phase contains commands for installing dependencies to run your tests. # Certain frequently used dependencies are preinstalled on the test host to accelerate and # simplify your test setup. To find these dependencies, versions supported and additional # software installation please see: # https://docs.aws.amazon.com/devicefarm/latest/developerguide/custom-test-environments-hosts-software.html install: commands: # The Appium server is written using Node.js. In order to run your desired version of Appium, # you first need to set up a Node.js environment that is compatible with your version of Appium. - devicefarm-cli use node 20 - node --version # Use the devicefarm-cli to select a preinstalled major version of Appium. - devicefarm-cli use appium 2 - appium --version # The Device Farm service periodically updates the preinstalled Appium versions over time to # incorporate the latest minor and patch versions for each major version. If you wish to # select a specific version of Appium, you can use NPM to install it. # - npm install -g appium@2.19.0 # When running Android tests with Appium version 2, the uiautomator2 driver is preinstalled using driver # version 2.44.1 for Appium 2.5.1 If you want to install a different version of the driver, # you can use the Appium extension CLI to uninstall the existing uiautomator2 driver # and install your desired version: # - |- # if [ $DEVICEFARM_DEVICE_PLATFORM_NAME = "Android" ]; # then # appium driver uninstall uiautomator2; # appium driver install uiautomator2@2.34.0; # fi; # Based on Appium framework's recommendation, we recommend setting the Appium server's # base path explicitly for accepting commands. If you prefer the legacy base path of /wd/hub, # please set it here. - export APPIUM_BASE_PATH= # Use the devicefarm-cli to setup a Java environment, with which you can run your test suite. - devicefarm-cli use java 17 - java -version # The pre-test phase contains commands for setting up your test environment. pre_test: commands: # Setup the CLASSPATH so that Java knows where to find your test classes. - export CLASSPATH=$CLASSPATH:$DEVICEFARM_TEST_PACKAGE_PATH/* - export CLASSPATH=$CLASSPATH:$DEVICEFARM_TEST_PACKAGE_PATH/dependency-jars/* # We recommend starting the Appium server process in the background using the command below. # The Appium server log will be written to the $DEVICEFARM_LOG_DIR directory. # The environment variables passed as capabilities to the server will be automatically assigned # during your test run based on your test's specific device. # For more information about which environment variables are set and how they're set, please see # https://docs.aws.amazon.com/devicefarm/latest/developerguide/custom-test-environment-variables.html - |- appium --base-path=$APPIUM_BASE_PATH --log-timestamp \ --log-no-colors --relaxed-security --default-capabilities \ "{\"appium:deviceName\": \"$DEVICEFARM_DEVICE_NAME\", \ \"platformName\": \"$DEVICEFARM_DEVICE_PLATFORM_NAME\", \ \"appium:udid\":\"$DEVICEFARM_DEVICE_UDID\", \ \"appium:platformVersion\": \"$DEVICEFARM_DEVICE_OS_VERSION\", \ \"appium:chromedriverExecutableDir\": \"$DEVICEFARM_CHROMEDRIVER_EXECUTABLE_DIR\", \ \"appium:automationName\": \"UiAutomator2\"}" \ >> $DEVICEFARM_LOG_DIR/appium.log 2>&1 &; # This code snippet is to wait until the Appium server starts. - |- appium_initialization_time=0; until curl --silent --fail "http://0.0.0.0:4723${APPIUM_BASE_PATH}/status"; do if [[ $appium_initialization_time -gt 30 ]]; then echo "Appium did not start within 30 seconds. Exiting..."; exit 1; fi; appium_initialization_time=$((appium_initialization_time + 1)); echo "Waiting for Appium to start on port 4723..."; sleep 1; done; # The test phase contains commands for running your tests. test: commands: # Your test package is downloaded and unpackaged into the $DEVICEFARM_TEST_PACKAGE_PATH directory. - echo "Navigate to test package directory" - cd $DEVICEFARM_TEST_PACKAGE_PATH - echo "Starting the Appium TestNG test" # The following command runs your Appium Java TestNG test. # For more information, please see TestNG's documentation here: # https://testng.org/#_running_testng - |- java -Dappium.screenshots.dir=$DEVICEFARM_SCREENSHOT_PATH org.testng.TestNG -testjar *-tests.jar \ -d $DEVICEFARM_LOG_DIR/test-output -verbose 10 # To run your tests with a testng.xml file that is a part of your test package, # use the following commands instead: # - echo "Unzipping the tests JAR file" # - unzip *-tests.jar # - |- # java -Dappium.screenshots.dir=$DEVICEFARM_SCREENSHOT_PATH org.testng.TestNG -testjar *-tests.jar \ # testng.xml -d $DEVICEFARM_LOG_DIR/test-output -verbose 10 # The post-test phase contains commands that are run after your tests have completed. # If you need to run any commands to generating logs and reports on how your test performed, # we recommend adding them to this section. post_test: commands: # Artifacts are a list of paths on the filesystem where you can store test output and reports. # All files in these paths will be collected by Device Farm, with certain limits (see limit details # here: https://docs.aws.amazon.com/devicefarm/latest/developerguide/limits.html#file-limits). # These files will be available through the ListArtifacts API as your "Customer Artifacts". artifacts: # By default, Device Farm will collect your artifacts from the $DEVICEFARM_LOG_DIR directory. - $DEVICEFARM_LOG_DIR ``` ------ #### [ Appium iOS ] The following is an example test spec file that configures an Appium Java TestNG test run on iOS. ``` version: 0.1 # The following fields(s) allow you to select which Device Farm test host is used for your test run. ios_test_host: macos_sequoia phases: # The install phase contains commands for installing dependencies to run your tests. # Certain frequently used dependencies are preinstalled on the test host to accelerate and # simplify your test setup. To find these dependencies, versions supported and additional # software installation please see: # https://docs.aws.amazon.com/devicefarm/latest/developerguide/custom-test-environments-hosts-software.html install: commands: # The Appium server is written using Node.js. In order to run your desired version of Appium, # you first need to set up a Node.js environment that is compatible with your version of Appium. - devicefarm-cli use node 20 - node --version # Use the devicefarm-cli to select a preinstalled major version of Appium. - devicefarm-cli use appium 2 - appium --version # The Device Farm service periodically updates the preinstalled Appium versions over time to # incorporate the latest minor and patch versions for each major version. If you wish to # select a specific version of Appium, you can use NPM to install it. # - npm install -g appium@2.19.0 # When running iOS tests with Appium version 2, the XCUITest driver is preinstalled using driver # version 9.10.5 for Appium 2.5.4. If you want to install a different version of the driver, # you can use the Appium extension CLI to uninstall the existing XCUITest driver # and install your desired version: # - |- # if [ $DEVICEFARM_DEVICE_PLATFORM_NAME = "iOS" ]; # then # appium driver uninstall xcuitest; # appium driver install xcuitest@10.0.0; # fi; # Based on Appium framework's recommendation, we recommend setting the Appium server's # base path explicitly for accepting commands. If you prefer the legacy base path of /wd/hub, # please set it here. - export APPIUM_BASE_PATH= # Use the devicefarm-cli to setup a Java environment, with which you can run your test suite. - devicefarm-cli use java 17 - java -version # The pre-test phase contains commands for setting up your test environment. pre_test: commands: # Setup the CLASSPATH so that Java knows where to find your test classes. - export CLASSPATH=$CLASSPATH:$DEVICEFARM_TEST_PACKAGE_PATH/* - export CLASSPATH=$CLASSPATH:$DEVICEFARM_TEST_PACKAGE_PATH/dependency-jars/* # Device Farm provides multiple pre-built versions of WebDriverAgent (WDA), a required # Appium dependency for iOS, where each version corresponds to the XCUITest driver version selected. # If Device Farm cannot find a corresponding version of WDA for your XCUITest driver, # the latest available version is selected by default. - |- APPIUM_DRIVER_VERSION=$(appium driver list --installed --json | jq -r ".xcuitest.version" | cut -d "." -f 1); CORRESPONDING_APPIUM_WDA=$(env | grep "DEVICEFARM_APPIUM_WDA_DERIVED_DATA_PATH_V${APPIUM_DRIVER_VERSION}") if [[ ! -z "$APPIUM_DRIVER_VERSION" ]] && [[ ! -z "$CORRESPONDING_APPIUM_WDA" ]]; then echo "Using Device Farm's prebuilt WDA version ${APPIUM_DRIVER_VERSION}.x, which corresponds with your driver"; DEVICEFARM_APPIUM_WDA_DERIVED_DATA_PATH=$(echo $CORRESPONDING_APPIUM_WDA | cut -d "=" -f2) else LATEST_SUPPORTED_WDA_VERSION=$(env | grep "DEVICEFARM_APPIUM_WDA_DERIVED_DATA_PATH_V" | sort -V -r | head -n 1) echo "Unknown driver version $APPIUM_DRIVER_VERSION; falling back to the Device Farm default version of $LATEST_SUPPORTED_WDA_VERSION"; DEVICEFARM_APPIUM_WDA_DERIVED_DATA_PATH=$(echo $LATEST_SUPPORTED_WDA_VERSION | cut -d "=" -f2) fi; # For iOS versions 16 and below only, the device unique identifier (UDID) needs to modified for Appium tests # on Device Farm to remove the hypens. - |- if [ $DEVICEFARM_DEVICE_PLATFORM_NAME = "iOS" ]; then DEVICEFARM_DEVICE_UDID_FOR_APPIUM=$DEVICEFARM_DEVICE_UDID; if [ $(echo $DEVICEFARM_DEVICE_OS_VERSION | cut -d "." -f 1) -le 16 ]; then DEVICEFARM_DEVICE_UDID_FOR_APPIUM=$(echo $DEVICEFARM_DEVICE_UDID | tr -d "-"); fi; fi; # We recommend starting the Appium server process in the background using the command below. # The Appium server log will be written to the $DEVICEFARM_LOG_DIR directory. # The environment variables passed as capabilities to the server will be automatically assigned # during your test run based on your test's specific device. # For more information about which environment variables are set and how they're set, please see # https://docs.aws.amazon.com/devicefarm/latest/developerguide/custom-test-environment-variables.html - |- appium --base-path=$APPIUM_BASE_PATH --log-timestamp \ --log-no-colors --relaxed-security --default-capabilities \ "{\"appium:deviceName\": \"$DEVICEFARM_DEVICE_NAME\", \ \"platformName\": \"$DEVICEFARM_DEVICE_PLATFORM_NAME\", \ \"appium:app\": \"$DEVICEFARM_APP_PATH\", \ \"appium:udid\":\"$DEVICEFARM_DEVICE_UDID_FOR_APPIUM\", \ \"appium:platformVersion\": \"$DEVICEFARM_DEVICE_OS_VERSION\", \ \"appium:derivedDataPath\": \"$DEVICEFARM_APPIUM_WDA_DERIVED_DATA_PATH\", \ \"appium:usePrebuiltWDA\": true, \ \"appium:automationName\": \"XCUITest\"}" \ >> $DEVICEFARM_LOG_DIR/appium.log 2>&1 & # This code snippet is to wait until the Appium server starts. - |- appium_initialization_time=0; until curl --silent --fail "http://0.0.0.0:4723${APPIUM_BASE_PATH}/status"; do if [[ $appium_initialization_time -gt 30 ]]; then echo "Appium did not start within 30 seconds. Exiting..."; exit 1; fi; appium_initialization_time=$((appium_initialization_time + 1)); echo "Waiting for Appium to start on port 4723..."; sleep 1; done; # The test phase contains commands for running your tests. test: commands: # Your test package is downloaded and unpackaged into the $DEVICEFARM_TEST_PACKAGE_PATH directory. - echo "Navigate to test package directory" - cd $DEVICEFARM_TEST_PACKAGE_PATH - echo "Starting the Appium TestNG test" # The following command runs your Appium Java TestNG test. # For more information, please see TestNG's documentation here: # https://testng.org/#_running_testng - |- java -Dappium.screenshots.dir=$DEVICEFARM_SCREENSHOT_PATH org.testng.TestNG -testjar *-tests.jar \ -d $DEVICEFARM_LOG_DIR/test-output -verbose 10 # To run your tests with a testng.xml file that is a part of your test package, # use the following commands instead: # - echo "Unzipping the tests JAR file" # - unzip *-tests.jar # - |- # java -Dappium.screenshots.dir=$DEVICEFARM_SCREENSHOT_PATH org.testng.TestNG -testjar *-tests.jar \ # testng.xml -d $DEVICEFARM_LOG_DIR/test-output -verbose 10 # The post-test phase contains commands that are run after your tests have completed. # If you need to run any commands to generating logs and reports on how your test performed, # we recommend adding them to this section. post_test: commands: # Artifacts are a list of paths on the filesystem where you can store test output and reports. # All files in these paths will be collected by Device Farm, with certain limits (see limit details # here: https://docs.aws.amazon.com/devicefarm/latest/developerguide/limits.html#file-limits). # These files will be available through the ListArtifacts API as your "Customer Artifacts". artifacts: # By default, Device Farm will collect your artifacts from the $DEVICEFARM_LOG_DIR directory. - $DEVICEFARM_LOG_DIR ``` ------ #### [ Appium (Both Platforms) ] The following is an example test spec file that configures an Appium Java TestNG test run on both Android and iOS. ``` version: 0.1 # The following fields(s) allow you to select which Device Farm test host is used for your test run. android_test_host: amazon_linux_2 ios_test_host: macos_sequoia phases: # The install phase contains commands for installing dependencies to run your tests. # Certain frequently used dependencies are preinstalled on the test host to accelerate and # simplify your test setup. To find these dependencies, versions supported and additional # software installation please see: # https://docs.aws.amazon.com/devicefarm/latest/developerguide/custom-test-environments-hosts-software.html install: commands: # The Appium server is written using Node.js. In order to run your desired version of Appium, # you first need to set up a Node.js environment that is compatible with your version of Appium. - devicefarm-cli use node 20 - node --version # Use the devicefarm-cli to select a preinstalled major version of Appium. - devicefarm-cli use appium 2 - appium --version # The Device Farm service periodically updates the preinstalled Appium versions over time to # incorporate the latest minor and patch versions for each major version. If you wish to # select a specific version of Appium, you can use NPM to install it. # - npm install -g appium@2.19.0 # When running Android tests with Appium version 2, the uiautomator2 driver is preinstalled using driver # version 2.44.1 for Appium 2.5.1 If you want to install a different version of the driver, # you can use the Appium extension CLI to uninstall the existing uiautomator2 driver # and install your desired version: # - |- # if [ $DEVICEFARM_DEVICE_PLATFORM_NAME = "Android" ]; # then # appium driver uninstall uiautomator2; # appium driver install uiautomator2@2.34.0; # fi; # When running iOS tests with Appium version 2, the XCUITest driver is preinstalled using driver # version 9.10.5 for Appium 2.5.4. If you want to install a different version of the driver, # you can use the Appium extension CLI to uninstall the existing XCUITest driver # and install your desired version: # - |- # if [ $DEVICEFARM_DEVICE_PLATFORM_NAME = "iOS" ]; # then # appium driver uninstall xcuitest; # appium driver install xcuitest@10.0.0; # fi; # Based on Appium framework's recommendation, we recommend setting the Appium server's # base path explicitly for accepting commands. If you prefer the legacy base path of /wd/hub, # please set it here. - export APPIUM_BASE_PATH= # Use the devicefarm-cli to setup a Java environment, with which you can run your test suite. - devicefarm-cli use java 17 - java -version # The pre-test phase contains commands for setting up your test environment. pre_test: commands: # Setup the CLASSPATH so that Java knows where to find your test classes. - export CLASSPATH=$CLASSPATH:$DEVICEFARM_TEST_PACKAGE_PATH/* - export CLASSPATH=$CLASSPATH:$DEVICEFARM_TEST_PACKAGE_PATH/dependency-jars/* # Device Farm provides multiple pre-built versions of WebDriverAgent (WDA), a required # Appium dependency for iOS, where each version corresponds to the XCUITest driver version selected. # If Device Farm cannot find a corresponding version of WDA for your XCUITest driver, # the latest available version is selected by default. - |- if [ $DEVICEFARM_DEVICE_PLATFORM_NAME = "iOS" ]; then APPIUM_DRIVER_VERSION=$(appium driver list --installed --json | jq -r ".xcuitest.version" | cut -d "." -f 1); CORRESPONDING_APPIUM_WDA=$(env | grep "DEVICEFARM_APPIUM_WDA_DERIVED_DATA_PATH_V${APPIUM_DRIVER_VERSION}") if [[ ! -z "$APPIUM_DRIVER_VERSION" ]] && [[ ! -z "$CORRESPONDING_APPIUM_WDA" ]]; then echo "Using Device Farm's prebuilt WDA version ${APPIUM_DRIVER_VERSION}.x, which corresponds with your driver"; DEVICEFARM_APPIUM_WDA_DERIVED_DATA_PATH=$(echo $CORRESPONDING_APPIUM_WDA | cut -d "=" -f2) else LATEST_SUPPORTED_WDA_VERSION=$(env | grep "DEVICEFARM_APPIUM_WDA_DERIVED_DATA_PATH_V" | sort -V -r | head -n 1) echo "Unknown driver version $APPIUM_DRIVER_VERSION; falling back to the Device Farm default version of $LATEST_SUPPORTED_WDA_VERSION"; DEVICEFARM_APPIUM_WDA_DERIVED_DATA_PATH=$(echo $LATEST_SUPPORTED_WDA_VERSION | cut -d "=" -f2) fi; fi; # For iOS versions 16 and below only, the device unique identifier (UDID) needs to modified for Appium tests # on Device Farm to remove the hypens. - |- if [ $DEVICEFARM_DEVICE_PLATFORM_NAME = "iOS" ]; then DEVICEFARM_DEVICE_UDID_FOR_APPIUM=$DEVICEFARM_DEVICE_UDID; if [ $(echo $DEVICEFARM_DEVICE_OS_VERSION | cut -d "." -f 1) -le 16 ]; then DEVICEFARM_DEVICE_UDID_FOR_APPIUM=$(echo $DEVICEFARM_DEVICE_UDID | tr -d "-"); fi; fi; # We recommend starting the Appium server process in the background using the command below. # The Appium server log will be written to the $DEVICEFARM_LOG_DIR directory. # The environment variables passed as capabilities to the server will be automatically assigned # during your test run based on your test's specific device. # For more information about which environment variables are set and how they're set, please see # https://docs.aws.amazon.com/devicefarm/latest/developerguide/custom-test-environment-variables.html - |- if [ $DEVICEFARM_DEVICE_PLATFORM_NAME = "Android" ]; then appium --base-path=$APPIUM_BASE_PATH --log-timestamp \ --log-no-colors --relaxed-security --default-capabilities \ "{\"appium:deviceName\": \"$DEVICEFARM_DEVICE_NAME\", \ \"platformName\": \"$DEVICEFARM_DEVICE_PLATFORM_NAME\", \ \"appium:udid\":\"$DEVICEFARM_DEVICE_UDID\", \ \"appium:platformVersion\": \"$DEVICEFARM_DEVICE_OS_VERSION\", \ \"appium:chromedriverExecutableDir\": \"$DEVICEFARM_CHROMEDRIVER_EXECUTABLE_DIR\", \ \"appium:automationName\": \"UiAutomator2\"}" \ >> $DEVICEFARM_LOG_DIR/appium.log 2>&1 & else appium --base-path=$APPIUM_BASE_PATH --log-timestamp \ --log-no-colors --relaxed-security --default-capabilities \ "{\"appium:deviceName\": \"$DEVICEFARM_DEVICE_NAME\", \ \"platformName\": \"$DEVICEFARM_DEVICE_PLATFORM_NAME\", \ \"appium:udid\":\"$DEVICEFARM_DEVICE_UDID_FOR_APPIUM\", \ \"appium:platformVersion\": \"$DEVICEFARM_DEVICE_OS_VERSION\", \ \"appium:derivedDataPath\": \"$DEVICEFARM_WDA_DERIVED_DATA_PATH\", \ \"appium:usePrebuiltWDA\": true, \ \"appium:automationName\": \"XCUITest\"}" \ >> $DEVICEFARM_LOG_DIR/appium.log 2>&1 & fi; # This code snippet is to wait until the Appium server starts. - |- appium_initialization_time=0; until curl --silent --fail "http://0.0.0.0:4723${APPIUM_BASE_PATH}/status"; do if [[ $appium_initialization_time -gt 30 ]]; then echo "Appium did not start within 30 seconds. Exiting..."; exit 1; fi; appium_initialization_time=$((appium_initialization_time + 1)); echo "Waiting for Appium to start on port 4723..."; sleep 1; done; # The test phase contains commands for running your tests. test: commands: # Your test package is downloaded and unpackaged into the $DEVICEFARM_TEST_PACKAGE_PATH directory. - echo "Navigate to test package directory" - cd $DEVICEFARM_TEST_PACKAGE_PATH - echo "Starting the Appium TestNG test" # The following command runs your Appium Java TestNG test. # For more information, please see TestNG's documentation here: # https://testng.org/#_running_testng - |- java -Dappium.screenshots.dir=$DEVICEFARM_SCREENSHOT_PATH org.testng.TestNG -testjar *-tests.jar \ -d $DEVICEFARM_LOG_DIR/test-output -verbose 10 # To run your tests with a testng.xml file that is a part of your test package, # use the following commands instead: # - echo "Unzipping the tests JAR file" # - unzip *-tests.jar # - |- # java -Dappium.screenshots.dir=$DEVICEFARM_SCREENSHOT_PATH org.testng.TestNG -testjar *-tests.jar \ # testng.xml -d $DEVICEFARM_LOG_DIR/test-output -verbose 10 # The post-test phase contains commands that are run after your tests have completed. # If you need to run any commands to generating logs and reports on how your test performed, # we recommend adding them to this section. post_test: commands: # Artifacts are a list of paths on the filesystem where you can store test output and reports. # All files in these paths will be collected by Device Farm, with certain limits (see limit details # here: https://docs.aws.amazon.com/devicefarm/latest/developerguide/limits.html#file-limits). # These files will be available through the ListArtifacts API as your "Customer Artifacts". artifacts: # By default, Device Farm will collect your artifacts from the $DEVICEFARM_LOG_DIR directory. - $DEVICEFARM_LOG_DIR ``` ------ # Hosts for custom test environments Device Farm supports a set of operating systems with pre-configured software through the use of a test host environment. During test execution, Device Farm utilizes Amazon-managed instances (hosts) that dynamically connect to the selected device under test. This instance is fully cleaned up and not re-used between runs, and is terminated with its generated artifacts after the test run concludes. **Topics** + [ ## Available test hosts for custom test environments ](#custom-test-environments-hosts-available) + [ ## Selecting a test host for custom test environments ](#test-host-selection) + [ # Supported software within custom test environments ](custom-test-environments-hosts-software.md) + [ # Test environment for Android devices ](custom-test-environments-hosts-android.md) + [ # Test environment for iOS devices ](custom-test-environments-hosts-ios.md) ## Available test hosts for custom test environments The test hosts are fully managed by Device Farm. The following table lists the currently available and supported Device Farm test hosts for custom test environments. | Device Platform | Test Host | Operating System | Architecture(s) | Supported Devices | | --- | --- | --- | --- | --- | | Android | amazon\$1linux\$12 | Amazon Linux 2 | x86\$164 | Android 6 and above | | iOS | macos\$1sequoia | macOS Sequoia (version 15) | arm64 | iOS 15 to 26 | **Note** Periodically, Device Farm adds new test hosts for a device platform to support newer device OS versions and its dependencies. When this occurs, older test hosts for the respective device platform are subject to end of support. ### Operating system version Each available test host uses a specific version of the operating system supported on Device Farm at the time. Although we try to be on the latest OS version, this may might not be the latest publicly distributed version available. Device Farm will periodically update the operating system with minor version updates and security patches. To know the specific version (including the minor version) of the operating system in use during your test run, you can add the following snippet of code to any of your test spec file's phases. **Example** ``` phases: install: commands: # The following example prints the instance's operating system version details - |- if [[ "Darwin" == "$(uname)" ]]; then echo "$(sw_vers --productName) $(sw_vers --productVersion) ($(sw_vers --buildVersion))"; else echo "$(. /etc/os-release && echo $PRETTY_NAME) ($(uname -r))"; fi ``` ## Selecting a test host for custom test environments You can specify the Android and iOS test host in the appropriate `android_test_host` and `ios_test_host` variables of your [test spec file](custom-test-environment-test-spec.md#custom-test-environment-test-spec-syntax). If you do not specify a test host selection for the given device platform, tests will run on the test host that Device Farm has set as the default for the specified device and test configuration. **Important** When testing on iOS 18 and below, a legacy test host will be used when a host is not selected. For more information, see the topic on the [Legacy iOS test host](custom-test-environments-hosts-ios.md#legacy-ios-host). As an example, review the following code snippet: **Example** ``` version: 0.1 android_test_host: amazon_linux_2 ios_test_host: macos_sequoia phases: # ... ``` # Supported software within custom test environments Device Farm uses host machines that are pre-installed with many of the necessary software libraries to run test frameworks supported on our service, providing a ready testing environment on launch. Device Farm supports multiple languages through the use of our software selection mechanism, and will periodically update the versions of the languages included in the environment. For any other required software, you can modify the test spec file to install from your test package, download from the internet, or access private sources within your VPC (see [VPC ENI](https://docs.aws.amazon.com//devicefarm/latest/developerguide/vpc-eni.html) for more information). For more information, see [Test spec examples](custom-test-environment-test-spec.md#custom-test-environment-test-spec-example). ## Pre-configured software In order to facilitate device testing on each platform, the following tooling is provided on the test host: | Tools | Device Platform(s) | | --- | --- | | Android SDK Build-Tools | Android | | Android SDK Platform-Tools (includes `adb`) | Android | | Xcode | iOS | ## Selectable software In addition to the pre-configured software on the host, Device Farm offers a way to select certain versions of supported software via the `devicefarm-cli` tooling. The following table contains the selectable software and the test hosts that contain them. | Software / Tool | Hosts that support this software | Command to use in your test spec | | --- | --- | --- | | Java 17 | amazon\$1linux\$12 macos\$1sequoia | `devicefarm-cli use java 17` | | Java 11 | amazon\$1linux\$12 macos\$1sequoia | `devicefarm-cli use java 11` | | Java 8 | amazon\$1linux\$12 macos\$1sequoia | `devicefarm-cli use java 8` | | Node.js 22 | amazon\$1linux\$12 macos\$1sequoia | `devicefarm-cli use node 22` | | Node.js 20 | amazon\$1linux\$12 macos\$1sequoia | `devicefarm-cli use node 20` | | Node.js 18 | amazon\$1linux\$12 macos\$1sequoia | `devicefarm-cli use node 18` | | Node.js 16 | amazon\$1linux\$12 | `devicefarm-cli use node 16` | | Python 3.12 | amazon\$1linux\$12 macos\$1sequoia | `devicefarm-cli use python 3.12` | | Python 3.11 | amazon\$1linux\$12 macos\$1sequoia | `devicefarm-cli use python 3.11` | | Python 3.10 | amazon\$1linux\$12 macos\$1sequoia | `devicefarm-cli use python 3.10` | | Python 3.9 | amazon\$1linux\$12 macos\$1sequoia | `devicefarm-cli use python 3.9` | | Python 3.8 | amazon\$1linux\$12 | `devicefarm-cli use python 3.8` | | Ruby 3.2 | amazon\$1linux\$12 macos\$1sequoia | `devicefarm-cli use ruby 3.2` | | Ruby 2.7 | amazon\$1linux\$12 | `devicefarm-cli use ruby 2.7` | | Appium 3 | amazon\$1linux\$12 macos\$1sequoia | `devicefarm-cli use appium 3` | | Appium 2 | amazon\$1linux\$12 macos\$1sequoia | `devicefarm-cli use appium 2` | | Appium 1 | amazon\$1linux\$12 | `devicefarm-cli use appium 1` | | Xcode 26 | macos\$1sequoia | `devicefarm-cli use xcode 26` | | Xcode 16 | macos\$1sequoia | `devicefarm-cli use xcode 16` | The test host also includes commonly used supporting tools for each software version, such as the `pip` and `npm` package managers (included with Python and Node.js respectively) and dependencies (such as the Appium UIAutomator2 Driver) for tools like Appium. This ensures you have the tools needed to work with the supported test frameworks. # Using the devicefarm-cli tool in custom test environments The test host uses a standardized version management tool called ` devicefarm-cli` to select software versions. This tool is separate from the AWS CLI and only available on the Device Farm test host. With `devicefarm-cli`, you can switch to any pre-installed software version on the test host. This provides a straightforward way to maintain your Device Farm test spec file over time and gives you a predictable mechanism to upgrade software versions in the future. **Important** This command line tool is not available on legacy iOS hosts. For more information, see the topic on the [Legacy iOS test host](custom-test-environments-hosts-ios.md#legacy-ios-host). The snippet below shows the `help` page of `devicefarm-cli`: ``` $ devicefarm-cli help Usage: devicefarm-cli COMMAND [ARGS] Commands: help Prints this usage message. list Lists all versions of software configurable via this CLI. use Configures the software for usage within the current shell's environment. ``` Let's review a couple of examples using `devicefarm-cli`. To use the tool to change the Python version from *3.10* to *3.9* in your test spec file, run the following commands: ``` $ python --version Python 3.10.12 $ devicefarm-cli use python 3.9 $ python --version Python 3.9.17 ``` To change the Appium version from *1* to *2* : ``` $ appium --version 1.22.3 $ devicefarm-cli use appium 2 $ appium --version 2.1.2 ``` **Tip** Note that when you select a software version, `devicefarm-cli` also switches the supporting tools for those languages, such as `pip` for Python and `npm` for NodeJS. For more information about the preinstalled software on the test host, see [Supported software within custom test environments](custom-test-environments-hosts-software.md). # Test environment for Android devices AWS Device Farm utilizes Amazon Elastic Compute Cloud (EC2) host machines running Amazon Linux 2 to execute Android tests. When you schedule a test run, Device Farm allocates a dedicated host for each device to independently run tests. The host machines terminate after the test run along with any generated artifacts. The Amazon Linux 2 host provides several advantages: + **Faster, more reliable testing**: Compared to the legacy host, the new test host significantly improves test speed, especially reducing test start times. The Amazon Linux 2 host also demonstrates greater stability and reliability during testing. + **Enhanced Remote Access for manual testing**: Upgrades to the latest test host and improvements lead to lower latency and better video performance for Android manual testing. + **Standard software version selection**: Device Farm now standardizes major programming language support on the test host as well as Appium framework versions. For supported languages (currently Java, Python, Node.js, and Ruby) and Appium, the new test host provides long-term stable releases soon after launch. Centralized version management through the `devicefarm-cli` tool enables test spec file development with a consistent experience across frameworks. **Topics** + [ # Supported IP ranges for the Amazon Linux 2 test environment in Device Farm ](amazon-linux-2-ip-ranges.md) # Supported IP ranges for the Amazon Linux 2 test environment in Device Farm Customers often need to know the IP range from which Device Farm's traffic originates, particularly for configuring their firewalls and security settings. For Amazon EC2 test hosts, the IP range encompasses the entire `us-west-2` region. For Amazon Linux 2 test hosts, which is the default option for new Android runs, the ranges have been restricted. The traffic now originates from a specific set of NAT gateways, restricting the IP range to the following addresses: **** | IP Ranges | | --- | | **44.236.137.143** | | **52.13.151.244** | | **52.35.189.191** | | **54.201.250.26** | For more information about Android test environments in Device Farm, see [Test environment for Android devices](custom-test-environments-hosts-android.md). # Test environment for iOS devices Device Farm utilizes Amazon-managed macOS instances (hosts) that dynamically connect to the iOS device during the test run. Each host is pre-configured with software that enables device testing on various popular test platforms, such as XCTestUI and Appium. The current iteration of the iOS test host has improved upon the testing experience when compared to previous versions, including: + ** Consistent host OS and tooling experience for iOS 15 to iOS 26 ** Before, the test host was determined by the device in use, leading to a fragmented software environment when executing on multiple iOS versions. The current experience allows simple host selection to enable a consistent environment across devices. This will enable the same macOS version and tooling (such as Xcode) to be available across each iOS device. + ** Performance improvements for iOS 15 and 16 tests ** Using updated infrastructure, setup time has improved substantially for iOS 15 and 16 tests. + ** Standardized selectable software versions for supported dependencies ** We now have the `devicefarm-cli` software selection system on both iOS and Android test hosts, enabling you to select your preferred version of our supported dependencies. For supported dependencies (such as Java, Python, Node.js, Ruby, and Appium), versions will be selectable via the test spec. For an idea of how this feature works, please see the topic on [Supported software within custom test environments](custom-test-environments-hosts-software.md). **Important** If executing on iOS 18 and below, your tests will execute on legacy test hosts by default. See the topic below on how to migrate away from legacy hosts. ## Legacy iOS test host For existing tests on iOS 18 and below, the legacy test hosts are selected by default for custom test environments. The following table contains the test host version that is executed with by the iOS device version. | Operating System | Architecture(s) | Default for Devices | | --- | --- | --- | | macOS Sonoma (version 14) | arm64 | iOS 18 | | macOS Ventura (version 13) | arm64 | iOS 17 | | macOS Monterey (version 12) | x86\$164 | iOS 16 and below | In order to select the newer test hosts, see the topic regarding [Migrating your custom test environments to the new iOS test hosts](ios-host-migration.md). ## Supported software for iOS devices In order to support iOS device testing, Device Farm test hosts for iOS devices come pre-configured with Xcode and its associated command line tooling. For other available software, please review the topic regarding [Supported software within custom test environments](custom-test-environments-hosts-software.md). # Migrating your custom test environments to the new iOS test hosts To migrate existing tests from the legacy host to the new macOS test host, you will need to develop new test spec files based on your pre-existing ones. The recommended approach is to start with the example test spec file for your desired test types, then migrate relevant commands from your old test spec file to the new one. This lets you leverage new features and optimizations of the example test spec for the new host while reusing snippets your existing code. **Topics** + [ ## Tutorial: Migrating iOS test spec files with the console ](#ios-host-migration-console-tutorial) + [ ## Differences between the new and legacy test hosts ](#ios-host-migration-differences) ## Tutorial: Migrating iOS test spec files with the console In this example, the Device Farm console will be used to onboard an existing iOS device test spec to use the new test host. ### Step 1: Creating a new test spec files with the console 1. Sign in to the [AWS Device Farm console](https://console.aws.amazon.com/devicefarm). 1. Navigate to the Device Farm project containing your automation tests. 1. Download a copy of the existing test spec your wish to onboard with. 1. Click the "Project Settings" option and navigate to the **Uploads** tab. 1. Navigate to the test spec file that you wish to onboard with. 1. Click the **Download** button to make a local copy of this file. 1. Navigate back to the Project page and click **Create run**. 1. Fill in the options on the wizard as if you were to start a new run, but stop at the **Select test spec** option. 1. Using the iOS test spec selected by default, click the **Create a test spec** button. 1. Modify the test specification that was selected by *default* in the text editor. 1. If not already present, modify the test spec file to select the new host using: ``` ios_test_host: macos_sequoia ``` 1. From the copy of your test spec downloaded in a prior step, review each ` phase`. 1. Copy commands from the old test spec's phases into each respective phase in the new test spec, ignoring commands related to installing or selecting Java, Python, Node.js, Ruby, Appium, or Xcode. 1. Enter a new file name in the **Save as** text box. 1. Click the **Save as new** button to save your changes. For an example of a test spec file you can use as a reference, see the example provided in [Test spec examples](custom-test-environment-test-spec.md#custom-test-environment-test-spec-example). ### Step 2: Selecting software pre-installed software In the new test host, pre-installed software versions are selected using a new standardized version management tool called `devicefarm-cli`. This tooling is now the recommended approach for using the various software we provide on the test hosts. As an example, you would add the following line to use a different JDK 17 your test environment: ``` - devicefarm-cli use java 17 ``` For more information on the software supported available, please review: [Supported software within custom test environments](custom-test-environments-hosts-software.md). ### Step 3: Using Appium and its dependencies via the software selection tooling The new test host only supports Appium 2.x and above. Please explicitly select the Appium version using the `devicefarm-cli`, while removing legacy tooling such as ` avm`. For example: ``` # This line using 'avm' should be removed # - avm 2.3.1 # And the following lines should be added - devicefarm-cli use appium 2 # Selects the version - appium --version # Prints the version ``` The Appium version selected with `devicefarm-cli` comes preinstalled with a compatible version of the XCUITest driver for iOS. Additionally, you will need to update your test spec to use ` DEVICEFARM_APPIUM_WDA_DERIVED_DATA_PATH_V9` instead of ` DEVICEFARM_WDA_DERIVED_DATA_PATH`. The new environment variable points to a pre-built version of WebDriverAgent 9.x, which is the latest supported version for Appium 2 tests. For more information, review [Selecting a WebDriverAgent version for iOS tests](test-types-appium.md#test-types-appium-select-wda) and [Environment variables for Appium tests](custom-test-environment-variables.md#custom-test-environment-variables-appium). ## Differences between the new and legacy test hosts When you're editing your test spec file to use the new iOS test host and transitioning your tests from the legacy test host, be aware of these key environment differences: + ** Xcode versions: ** In the legacy test host environment, the Xcode version available was based on the iOS version of the device used for testing. For example, tests on iOS 18 devices used Xcode 16 in the legacy host, whereas tests on iOS 17 used Xcode 15. In the new host environment, all devices can access the same versions of Xcode, allowing a consistent environment for tests on devices with different versions. For a list of currently available Xcode versions, see [Supported software](custom-test-environments-hosts-software.md). + ** Selecting software versions: ** In many instances, the default software versions have changed, so if you weren't explicitly selecting your software version in the legacy test host before, you may want to specify it now in the new test host using [`devicefarm-cli`](custom-test-environments-hosts-software-cli.md). In the vast majority of use cases, we recommend that customers explicitly select the versions of software they use. By selecting a software version with `devicefarm-cli` you'll have a predictable and consistent experience with it and receive ample amounts of warnings if Device Farm plans to remove that version from the test host. Moreover, software selection tools like `nvm`, `pyenv`, ` avm`, and `rvm` have been removed in favor of the new ` devicefarm-cli` software selection system. + ** Available software versions: ** Many versions of previously pre-installed software have been removed, and many new versions have been added. So, ensure that when using the `devicefarm-cli` to select your software versions, you select versions which are in the [supported version list](custom-test-environments-hosts-software.md). + ** The `libimobiledevice` suite of tools have been removed ** in favor of newer / first party tooling to track current iOS device testing and industry standards. For iOS 17 and above, you can migrate most commands to use similar Xcode tooling, called `devicectl`. For information on `devicectl`, you can run `xcrun devicectl help` from a machine with Xcode installed. + ** File paths that are hard-coded ** in your legacy host test spec file as absolute paths will most likely not work as expected in the new test host, and they're generally not recommended for test spec file use. We recommend that you use relative paths and environment variables for all test spec file code. For more information, review the topic on [Best practices for custom test environment execution](custom-test-environments-best-practices.md). + ** Operating system version and architecture: ** The legacy test hosts were using a variety of macOS versions and CPU architectures based on the assigned device. As a result, users may notice some differences in the available system libraries available in the environment. For more information on the previous host OS version, review [Legacy iOS test host](custom-test-environments-hosts-ios.md#legacy-ios-host). + **For Appium** users, the way to select the WebDriverAgent has changed to a use environment variable prefix ` DEVICEFARM_APPIUM_WDA_DERIVED_DATA_PATH_V` instead of the old ` DEVICEFARM_WDA_DERIVED_DATA_PATH_V` prefix. For more information on the updated variable, review [Environment variables for Appium tests](custom-test-environment-variables.md#custom-test-environment-variables-appium). + **For Appium Java** users, the new test host does not contain any pre-installed JAR files in its class path, whereas the previous host contained one for the TestNG framework (via an environment variable `$DEVICEFARM_TESTNG_JAR`). We recommend that customers package the necessary JAR files for their test frameworks inside their test package and remove instances of the `$DEVICEFARM_TESTNG_JAR` variable from their test spec files. We recommend reaching out to the service team through a support case if you have any feedback or questions about the differences between the test hosts from a software perspective. # Access AWS resources using an IAM Execution Role Device Farm supports specifying an IAM role that will be assumed by the custom test runtime environment during test execution. This feature allows your tests to securely access AWS resources in your account, such as Amazon S3 buckets, DynamoDB tables, or other AWS services on which your application depends. **Topics** + [ ## Overview ](#iam-execution-role-overview) + [ ## IAM role requirements ](#iam-role-requirements) + [ ## Configuring an IAM execution role ](#configuring-iam-execution-role) + [ ## Best practices ](#iam-role-best-practices) + [ ## Troubleshooting ](#troubleshooting-iam-roles) ## Overview When you specify an IAM execution role, Device Farm assumes this role during test execution, allowing your tests to interact with AWS services using the permissions defined in the role. Common use cases for IAM execution roles include: + Accessing test data stored in Amazon S3 buckets + Pushing test artifacts to Amazon S3 buckets + Retrieving application configuration from AWS AppConfig + Writing test logs and metrics to Amazon CloudWatch + Sending test results or status messages to Amazon SQS queues + Calling AWS Lambda functions as part of test workflows ## IAM role requirements To use an IAM execution role with Device Farm, your role must meet the following requirements: + **Trust relationship**: The Device Farm service principal must be trusted to assume the role. The trust policy must include `devicefarm.amazonaws.com` as a trusted entity. + **Permissions**: The role must have the necessary permissions to access the AWS resources your tests require. + **Session duration**: The role's maximum session duration must be at least as long as your Device Farm project's job timeout setting. By default, Device Farm projects have a job timeout of 150 minutes, so your role must support a session duration of at least 150 minutes. + **Same account requirement**: The IAM role must be in the same AWS account as the one used to call Device Farm. Cross-account role assumption is not supported. + **PassRole permission**: The caller must be authorized to pass the IAM role by a policy allowing the `iam:PassRole` action on the specified execution role. ### Example trust policy The following example shows a trust policy that allows Device Farm to assume your execution role. This trust policy should only be attached to the specific IAM role you intend to use with Device Farm, not to other roles in your account: **Example** ``` { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "devicefarm.amazonaws.com" }, "Action": "sts:AssumeRole" } ] } ``` ### Example permissions policy The following example shows a permissions policy that grants access to common AWS services used in testing: **Example** ``` { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:GetObject", "s3:PutObject", "s3:ListBucket" ], "Resource": [ "arn:aws:s3:::my-test-bucket", "arn:aws:s3:::my-test-bucket/*" ] }, { "Effect": "Allow", "Action": [ "appconfig:GetConfiguration", "appconfig:StartConfigurationSession" ], "Resource": "*" }, { "Effect": "Allow", "Action": [ "logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents" ], "Resource": "arn:aws:logs:*:*:log-group:/devicefarm/test-*" }, { "Effect": "Allow", "Action": [ "sqs:SendMessage", "sqs:GetQueueUrl" ], "Resource": "arn:aws:sqs:*:*:test-results-*" } ] } ``` ## Configuring an IAM execution role You can specify an IAM execution role at the project level or for individual test runs. When configured at the project level, all runs within that project will inherit the execution role. An execution role configured on a run will supersede any configured on its parent project. For detailed instructions on configuring execution roles, see: + [Creating a project in AWS Device Farm](how-to-create-project.md) - for configuring execution roles at the project level + [Creating a test run in Device Farm](how-to-create-test-run.md) - for configuring execution roles for individual runs You can also configure execution roles using the Device Farm API. For more information, see the [Device Farm API Reference](https://docs.aws.amazon.com/devicefarm/latest/APIReference/). ## Best practices Follow these best practices when configuring IAM execution roles for your Device Farm tests: + **Principle of least privilege**: Grant only the minimum permissions necessary for your tests to function. Avoid using overly broad permissions like `*` actions or resources. + **Use resource-specific permissions**: When possible, limit permissions to specific resources (e.g., specific S3 buckets or DynamoDB tables) rather than all resources of a type. + **Separate test and production resources**: Use dedicated test resources and roles to avoid accidentally affecting production systems during testing. + **Regular role review**: Periodically review and update your execution roles to ensure they still meet your testing needs and follow security best practices. + **Use condition keys**: Consider using IAM condition keys to further restrict when and how the role can be used. ## Troubleshooting If you encounter issues with IAM execution roles, check the following: + **Trust relationship**: Verify that the role's trust policy includes `devicefarm.amazonaws.com` as a trusted service. + **Permissions**: Check that the role has the necessary permissions for the AWS services your tests are trying to access. + **Test logs**: Review the test execution logs for specific error messages related to AWS API calls or permission denials. # Environment variables for custom test environments Device Farm dynamically configures several environment variables for use as part of your custom test environment run. **Topics** + [ ## Custom environment variables ](#custom-test-environment-variables-custom) + [ ## Common environment variables ](#custom-test-environment-variables-common) + [ ## Environment variables for Appium tests ](#custom-test-environment-variables-appium) + [ ## Environment variables for XCUITest tests ](#custom-test-environment-variables-xcuitest) ## Custom environment variables Device Farm supports the configuration of key-value pairs that are applied as environment variables on the test host. These may be configured on a Device Farm project or during run creation; any variables configured on a run will supersede any that may be configured on its parent project. The following restrictions apply: + Custom environment variables are not supported on legacy iOS test hosts. For more information, see [Legacy iOS test host](custom-test-environments-hosts-ios.md#legacy-ios-host). + Variable names beginning with `$DEVICEFARM_` are reserved for internal service use. + Custom environment variables may not be used to configure test host compute selection in your test spec. ## Common environment variables This section describes environment variables common to all tests in Device Farm. ** `$DEVICEFARM_DEVICE_NAME` ** The device on which your tests run. It represents the unique device identifier (UDID) of the device. ** `$DEVICEFARM_DEVICE_UDID` ** The device's unique identifier. ** `$DEVICEFARM_DEVICE_PLATFORM_NAME` ** The device's platform name. It is either `Android` or `iOS`. ** `$DEVICEFARM_DEVICE_OS_VERSION` ** The device's OS version. ** `$DEVICEFARM_APP_PATH` ** *(mobile app tests)* The path to the mobile app on the host machine where the tests are being executed. This variable is not available during web tests. ** `$DEVICEFARM_LOG_DIR` ** The path to the default directory where customer logs, artifacts, and other wanted files will be stored for later retrieval. Using an [example test spec](custom-test-environment-test-spec.md#custom-test-environment-test-spec-example), files in this directory are archived in a ZIP file and made available as an artifact after your test run. ** `$DEVICEFARM_SCREENSHOT_PATH` ** The path to the screenshots, if any, captured during the test run. ** `$DEVICEFARM_PROJECT_ARN` ** The ARN of the job's parent project. ** `$DEVICEFARM_RUN_ARN` ** The ARN of the job's parent run. ** `$DEVICEFARM_DEVICE_ARN` ** The ARN of the device under test. ** `$DEVICEFARM_TOTAL_JOBS` ** The total number of jobs associated with its parent Device Farm run. ** `$DEVICEFARM_JOB_NUMBER` ** This job's number within `$DEVICEFARM_TOTAL_JOBS`. For example, a run may contain 5 jobs, and each will have a unique `$DEVICEFARM_JOB_NUMBER` ranging from 0 to 4. ** `$AWS_REGION` ** The AWS region. The service will set this to match the region in which the device under test is located. It can be overridden by a custom environment variable if needed. ** `$ANDROID_HOME` ** *(Android only)* The path to the Android SDK installation directory. ## Environment variables for Appium tests This section describes environment variables used by any Appium test in a custom test environment in Device Farm. ** `$DEVICEFARM_CHROMEDRIVER_EXECUTABLE_DIR` ** *(Android only)* The location of a directory which contains the necessary ChromeDriver executables for use in Appium web and hybrid tests. ** `$DEVICEFARM_APPIUM_WDA_DERIVED_DATA_PATH_V` ** *(iOS only)* The derived data path of a version of WebDriverAgent built to run on Device Farm. The numbering on the variable will correspond to the major version of the WebDriverAgent. As an example, `DEVICEFARM_APPIUM_WDA_DERIVED_DATA_PATH_V9` will point to the a WebDriverAgent version of 9.x. For more information, see [Selecting a WebDriverAgent version for iOS tests](test-types-appium.md#test-types-appium-select-wda). The `$DEVICEFARM_APPIUM_WDA_DERIVED_DATA_PATH_V` environment variables are only present on non-legacy iOS hosts. For more information, see [Legacy iOS test host](custom-test-environments-hosts-ios.md#legacy-ios-host). ** `$DEVICEFARM_WDA_DERIVED_DATA_PATH_V9` ** *(iOS only, deprecated)* The derived data path of a version of WebDriverAgent built to run on Device Farm. Refer to `$DEVICEFARM_APPIUM_WDA_DERIVED_DATA_PATH_V` for the replacement naming scheme. ## Environment variables for XCUITest tests This section describes environment variables used by the XCUITest test in a custom test environment in Device Farm. ** `$DEVICEFARM_XCUITESTRUN_FILE` ** The path to the Device Farm `.xctestun` file. It is generated from your app and test packages. ** `$DEVICEFARM_DERIVED_DATA_PATH` ** Expected path of Device Farm xcodebuild output. ** `$DEVICEFARM_XCTEST_BUILD_DIRECTORY` ** The path to the unzipped contents of the test package file. # Best practices for custom test environment execution The following topics cover recommended best practices for using custom test execution with Device Farm. **Run configuration** + ** Rely on Device Farm managed software and API features for run configuration ** wherever possible, as opposed to applying similar configurations via shell commands in the test spec file. This includes the configuration of the test host and the device, as this will be more sustainable and consistent across test hosts and devices. While Device Farm encourages you to customize your test spec file as greatly as you need in order to run your tests, the test spec file can become difficult to maintain over time as more customized commands are added to it. Using Device Farm managed software (through tools like ` devicefarm-cli` and the default available tools in the `$PATH`), and using managed features (like the [https://docs.aws.amazon.com/devicefarm/latest/APIReference/API_ScheduleRunConfiguration.html#devicefarm-Type-ScheduleRunConfiguration-deviceProxy](https://docs.aws.amazon.com/devicefarm/latest/APIReference/API_ScheduleRunConfiguration.html#devicefarm-Type-ScheduleRunConfiguration-deviceProxy) request parameter) to simplify the test spec file by shifting the responsibility of maintenance to Device Farm itself. **Test spec and test package code** + **Do not use absolute paths or rely on specific minor versions ** in your test spec file or test package code. Device Farm applies routine updates to the selected test host and its included software versions. Using specific or absolute paths (such as ` /usr/local/bin/python` instead of `python`) or requiring specific minor versions (such as Node.js `20.3.1` instead of just ` 20`) may lead to your tests failing to locate the required executable / file. As part of the custom test execution, Device Farm sets up various environment variables and the `$PATH` variable to ensure tests to have a consistent experience within our dynamic environments. See [Environment variables for custom test environments](custom-test-environment-variables.md) and [Supported software within custom test environments](custom-test-environments-hosts-software.md) for more information. + **Save generated or copied files within the temp directory during the test run. ** Today, we make sure that the temp directory (`/tmp`) will be accessible to the user during the test execution (besides managed directories, such as the `$DEVICEFARM_LOG_DIR`). Other directories that the user has access to may change over time due to the needs of the service or the operating system in use. + **Save your test execution logs to `$DEVICEFARM_LOG_DIR`**. This is the default artifact directory provided for your execution to add execution logs / artifacts into. The [example test specs](custom-test-environment-test-spec.md#custom-test-environment-test-spec-example) we provide each uses this directory for artifacts by default. + **Ensure your commands return a non-zero code on failure** during the `test` phase of your test spec. We determine if your execution failed by checking for a non-zero exit code of each shell command invoked during the `test` phase. You should ensure your logic or test framework will return a non-zero exit code for all desired scenarios, which may require additional configuration. For example, certain test frameworks (such as JUnit5) do not consider zero tests run to be a failure, which will cause your tests to be detected to have run successfully even if nothing was executed. Using JUnit5 as the example, you would need to specify the command line option `--fail-if-no-tests` to ensure this scenario exits with a non-zero exit code. + **Review the compatibility of software** with the device OS version and test host version you will be using for the test run. As an example, there are certain features in testing software frameworks (ie: Appium) that may not work as intended on all OS versions of the device being tested. **Security** + ** Avoid storing or logging sensitive variables (like AWS keys) in your test spec file. ** Test spec files, the test spec generated scripts, and the test spec script's logs are all provided as downloadable artifacts at the end of the test execution. This may lead to the unintended exposure of secrets for other users in your account with read access to your test run. # Migrating tests from a standard to custom test environment You can switch from a standard test execution mode to a custom execution mode in AWS Device Farm. Migration primarily involves two different forms of execution: 1. **Standard mode**: This test execution mode is primarily built to provide customers with granular reporting and a fully-managed environment. 1. **Custom mode**: This test execution mode is built for different use cases that require faster test runs, the ability to lift and shift and achieve parity with their local environment, and live video streaming. For more information about standard and custom modes in Device Farm, see [Test environments in AWS Device Farm](test-environments.md) and [Custom test environments in AWS Device Farm](custom-test-environments.md). ## Considerations when migrating This section lists some of the prominent use cases to consider when migrating to the custom mode: 1. **Speed**: In the standard mode of execution, Device Farm parses the metadata of the tests that you have packaged and uploaded using the packaging instructions for your particular framework. Parsing detects the number of tests in your package. Thereafter, Device Farm runs each test separately and presents the logs, videos, and other result artifacts individually for each test. However, this steadily adds to the total end-to-end test execution time as there are the pre- and post-processing of tests and result artifacts on the service end. By contrast, the custom mode of execution doesn't parse your test package; this means no pre-processing and minimal post-processing for tests or result artifacts. This results in total end-to-end execution times close to your local setup. The tests are executed in the same format as they would be if they ran on your local machine(s). The results of the tests are the same as what you get locally and are available for download at the end of the job execution. 1. **Customization or Flexibility**: The standard mode of execution parses your test package to detect the number of tests and then runs each test separately. Note that there is no guarantee that the tests will run in the order you specified. As a result, tests requiring a particular sequence of execution may not work as expected. In addition, there is no way to customize the host machine environment or pass config files that may be needed to run your tests a certain way. By contrast, custom mode lets you configure the host machine environment including the ability to install additional software, pass filters to your tests, pass config files, and control the test execution setup. It achieves this via a yaml file (also called the testspec file) that you can modify by adding shell commands to it. This yaml file gets converted to a shell script which gets executed on the test host machine. You can save multiple yaml files and choose one dynamically as per your requirements when you schedule a run. 1. **Live video and logging**: Both standard and custom modes of execution provide you with videos and logs for your tests. However, in standard mode, you get the video and predefined logs of your tests only after your tests are completed. By contrast, custom mode gives you a live stream of the video and client-side logs of your tests. In addition, you are able to download the video and other artifacts at the end of the test(s). **Tip** If your use case involves at least one of the factors above, we strongly recommend switching to the custom mode of execution. ## Migration steps To migrate from standard to custom mode, do the following: 1. Sign in to the AWS Management Console and open the Device Farm console at [https://console.aws.amazon.com/devicefarm/](https://console.aws.amazon.com/devicefarm/). 1. Choose your project and then start a new automation run. 1. Upload your app (or select `web app`), choose your test framework type, upload your test package, then under the `Choose your execution environment` parameter, choose the option to `Run your test in a custom environment`. 1. By default, Device Farm's example test spec file will appear for you to view and edit. This example file can be used as a starting place to try your tests out in [custom environment mode](https://docs.aws.amazon.com/devicefarm/latest/developerguide/custom-test-environments.html). Then, once you've verified that your tests are working properly from the console, you can then alter any of your API, CLI, and pipeline integrations with Device Farm to use this test spec file as a parameter when scheduling test runs. For information on how to add a test spec file as a parameter for your runs, see the `testSpecArn` parameter section for the `ScheduleRun` API in our [API guide](https://docs.aws.amazon.com/devicefarm/latest/APIReference/API_ScheduleRun.html). ## Appium framework In a custom test environnment, Device Farm does not insert or override any Appium capabilities in your Appium framework tests. You must specify your test's Appium capabilities in either the test spec YAML file or your test code. ## Android instrumentation You do not need to make changes to move your Android instrumentation tests to a custom test environment. ## iOS XCUITest You do not need to make changes to move your iOS XCUITest tests to a custom test environment. # Extending custom test environments in Device Farm AWS Device Farm enables configuring a custom environment for automated testing (custom mode), which is the recommended approach for all Device Farm users. The Device Farm custom mode enables you to run more than just your test suite. In this section, you learn how to extend your test suite and optimize your tests. For more information about custom test environments in Device Farm, see [Custom test environments in AWS Device Farm](custom-test-environments.md). **Topics** + [ # Setting a device PIN when running tests in Device Farm ](custom-test-environments-extending-set-pin.md) + [ # Speeding up Appium-based tests in Device Farm through desired capabilities ](custom-test-environments-extending-speed.md) + [ # Using Webhooks and other APIs after your tests run in Device Farm ](custom-test-environments-extending-webhooks.md) + [ # Adding extra files to your test package in Device Farm ](custom-test-environments-extending-files.md) # Setting a device PIN when running tests in Device Farm Some applications require that you set a PIN on the device. Device Farm does not support setting a PIN on devices natively. However, this is possible with the following caveats: + The device must be running Android 8 or above. + The PIN must be removed after the test is complete. To set the PIN in your tests, use the `pre_test` and `post_test` phases to set and remove the PIN, as shown following: ``` phases: pre_test: - # ... among your pre_test commands - DEVICE_PIN_CODE="1234" - adb shell locksettings set-pin "$DEVICE_PIN_CODE" post_test: - # ... Among your post_test commands - adb shell locksettings clear --old "$DEVICE_PIN_CODE" ``` When your test suite begins, the PIN 1234 is set. After your test suite exits, the PIN is removed. **Warning** If you don't remove the PIN from the device after the test is complete, the device and your account will be quarantined. For more ways to extend your test suite and optimize your tests, see [Extending custom test environments in Device Farm](custom-test-environments-extending.md). # Speeding up Appium-based tests in Device Farm through desired capabilities When using Appium, you might find that the standard mode test suite is very slow. This is because Device Farm applies the default settings and doesn't make any assumptions about how you want to use the Appium environment. While these defaults are built around industry best practices, they might not apply to your situation. To fine-tune the parameters of the Appium server, you can adjust the default Appium capabilities in your test spec. For example, the following sets the `usePrebuildWDA` capability to `true` in an iOS test suite to speed up initial start time: ``` phases: pre_test: - # ... Start up Appium - >- appium --log-timestamp --default-capabilities "{\"usePrebuiltWDA\": true, \"derivedDataPath\":\"$DEVICEFARM_WDA_DERIVED_DATA_PATH\", \"deviceName\": \"$DEVICEFARM_DEVICE_NAME\", \"platformName\":\"$DEVICEFARM_DEVICE_PLATFORM_NAME\", \"app\":\"$DEVICEFARM_APP_PATH\", \"automationName\":\"XCUITest\", \"udid\":\"$DEVICEFARM_DEVICE_UDID_FOR_APPIUM\", \"platformVersion\":\"$DEVICEFARM_DEVICE_OS_VERSION\"}" >> $DEVICEFARM_LOG_DIR/appiumlog.txt 2>&1 & ``` Appium capabilities must be a shell-escaped, quoted JSON structure. The following Appium capabilities are common sources of performance improvements: `noReset` and `fullReset` These two capabilities, which are mutually exclusive, describe the behavior of Appium after each session is complete. When `noReset` is set to `true`, the Appium server doesn't remove data from your application when an Appium session ends, effectively doing no cleanup whatsoever. `fullReset` uninstalls and clears all application data from the device after the session has closed. For more information, see [Reset Strategies](http://appium.io/docs/en/writing-running-appium/other/reset-strategies/) in the Appium documentation. `ignoreUnimportantViews` (Android only) Instructs Appium to compress the Android UI hierarchy only to *relevant* views for the test, speeding up certain element lookups. However, this can break some XPath-based test suites because the hierarchy of the UI layout has been changed. `skipUnlock` (Android only) Informs Appium that there is no PIN code currently set, which speeds up tests after a screen off event or other lock event. `webDriverAgentUrl` (iOS only) Instructs Appium to assume that an essential iOS dependency, `webDriverAgent`, is already running and available to accept HTTP requests at the specified URL. If `webDriverAgent` isn't already up and running, it can take Appium some time at the beginning of a test suite to start the `webDriverAgent`. If you start `webDriverAgent` yourself and set `webDriverAgentUrl` to `http://localhost:8100` when starting Appium, you can boot up your test suite faster. Note that this capability should never be used alongside the `useNewWDA` capability. You can use the following code to start `webDriverAgent` from your test spec file on the device's local port `8100`, then forward it to the test host's local port `8100` (this allows you to set `webDriverAgentUrl`'s value to `http://localhost:8100`). This code should be run during the install phase after any code for setting up the Appium and `webDriverAgent` environment variables has been defined: ``` # Start WebDriverAgent and iProxy - >- xcodebuild test-without-building -project /usr/local/avm/versions/$APPIUM_VERSION/node_modules/appium/node_modules/appium-webdriveragent/WebDriverAgent.xcodeproj -scheme WebDriverAgentRunner -derivedDataPath $DEVICEFARM_WDA_DERIVED_DATA_PATH -destination id=$DEVICEFARM_DEVICE_UDID_FOR_APPIUM IPHONEOS_DEPLOYMENT_TARGET=$DEVICEFARM_DEVICE_OS_VERSION GCC_TREAT_WARNINGS_AS_ERRORS=0 COMPILER_INDEX_STORE_ENABLE=NO >> $DEVICEFARM_LOG_DIR/webdriveragent_log.txt 2>&1 & iproxy 8100 8100 >> $DEVICEFARM_LOG_DIR/iproxy_log.txt 2>&1 & ``` Then, you can add the following code to your test spec file to ensure that `webDriverAgent` started successfully. This code should be run at the end of the pre-test phase after ensuring that Appium started successfully: ``` # Wait for WebDriverAgent to start - >- start_wda_timeout=0; while [ true ]; do if [ $start_wda_timeout -gt 60 ]; then echo "WebDriverAgent server never started in 60 seconds."; exit 1; fi; grep -i "ServerURLHere" $DEVICEFARM_LOG_DIR/webdriveragent_log.txt >> /dev/null 2>&1; if [ $? -eq 0 ]; then echo "WebDriverAgent REST http interface listener started"; break; else echo "Waiting for WebDriverAgent server to start. Sleeping for 1 seconds"; sleep 1; start_wda_timeout=$((start_wda_timeout+1)); fi; done; ``` For more information on the capabilities that Appium supports, see [Appium Desired Capabilities](http://appium.io/docs/en/writing-running-appium/caps/) in the Appium documentation. For more ways to extend your test suite and optimize your tests, see [Extending custom test environments in Device Farm](custom-test-environments-extending.md). # Using Webhooks and other APIs after your tests run in Device Farm You can have Device Farm call a webhook after every test suite finishes using **curl**. The process to do this varies with the destination and formatting. For your specific webhook, see the documentation for that webhook. The following example posts a message each time a test suite has finished to a Slack webhook: ``` phases: post_test: - curl -X POST -H 'Content-type: application/json' --data '{"text":"Tests on '$DEVICEFARM_DEVICE_NAME' have finished!"}' https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX ``` For more information on using webhooks with Slack, see [Sending your first Slack message using Webhook](https://api.slack.com/tutorials/slack-apps-hello-world) in the Slack API reference. For more ways to extend your test suite and optimize your tests, see [Extending custom test environments in Device Farm](custom-test-environments-extending.md). You are not limited to using **curl** to call webhooks. Test packages can include extra scripts and tools, as long as they are compatible with the Device Farm execution environment. For example, your test package may include auxiliary scripts that make requests to other APIs. Make sure that any required packages are installed alongside your test suite's requirements. To add a script that runs after your test suite is complete, include the script in your test package and add the following to your test spec: ``` phases: post_test: - python post_test.py ``` **Note** Maintaining any API keys or other authentication tokens used in your test package is your responsibility. We recommend that you keep any form of security credential out of source control, use credentials with the fewest possible privileges, and use revokable, short-lived tokens whenever possible. To verify security requirements, see the documentation for the third-party APIs that you use. If you plan on using AWS services as a part of your test execution suite, you should use IAM temporary credentials, generated outside of your test suite and included in your test package. These credentials should have the fewest granted permissions and shortest lifespan possible. For more information on creating temporary credentials, see [Requesting temporary security credentials](https://docs.aws.amazon.com//IAM/latest/UserGuide/id_credentials_temp_request.html) in the *IAM User Guide*. For more ways to extend your test suite and optimize your tests, see [Extending custom test environments in Device Farm](custom-test-environments-extending.md). # Adding extra files to your test package in Device Farm You may want to use additional files as a part of your tests either as extra configuration files or additional test data. You can add these additional files to your test package before uploading it to AWS Device Farm, then access them from the custom environment mode. Fundamentally, all test package upload formats (ZIP, IPA, APK, JAR, etc.) are package archive formats that support standard ZIP operations. You can add files to your test archive before uploading it to AWS Device Farm by using the following command: ``` $ zip zip-with-dependencies.zip extra_file ``` For a directory of extra files: ``` $ zip -r zip-with-dependencies.zip extra_files/ ``` These commands work as expected for all test package upload formats except for IPA files. For IPA files, especially when used with XCUITests, we recommend that you put any extra files in a slightly different location due to how AWS Device Farm resigns iOS test packages. When building your iOS test, the test application directory will be located inside of another directory named *Payload*. For example, this is how one such iOS test directory may look: ``` $ tree . └── Payload └── ADFiOSReferenceAppUITests-Runner.app ├── ADFiOSReferenceAppUITests-Runner ├── Frameworks │ ├── XCTAutomationSupport.framework │ │ ├── Info.plist │ │ ├── XCTAutomationSupport │ │ ├── _CodeSignature │ │ │ └── CodeResources │ │ └── version.plist │ └── XCTest.framework │ ├── Info.plist │ ├── XCTest │ ├── _CodeSignature │ │ └── CodeResources │ ├── en.lproj │ │ └── InfoPlist.strings │ └── version.plist ├── Info.plist ├── PkgInfo ├── PlugIns │ ├── ADFiOSReferenceAppUITests.xctest │ │ ├── ADFiOSReferenceAppUITests │ │ ├── Info.plist │ │ └── _CodeSignature │ │ └── CodeResources │ └── ADFiOSReferenceAppUITests.xctest.dSYM │ └── Contents │ ├── Info.plist │ └── Resources │ └── DWARF │ └── ADFiOSReferenceAppUITests ├── _CodeSignature │ └── CodeResources └── embedded.mobileprovision ``` For these XCUITest packages, add any extra files to the directory ending in *.app* inside of the *Payload* directory. For example, the following commands show how you can add a file to this test package: ``` $ mv extra_file Payload/*.app/ $ zip -r my_xcui_tests.ipa Payload/ ``` When you add a file to your test package, you can expect slightly different interaction behavior in AWS Device Farm based on its upload format. If the upload used the ZIP file extension, AWS Device Farm will automatically unzip the upload before your test and leave the unzipped files at the location with the *\$1DEVICEFARM\$1TEST\$1PACKAGE\$1PATH* environment variable. (This means that if you added a file called *extra\$1file* to the root of the archive as in the first example, it would be located at *\$1DEVICEFARM\$1TEST\$1PACKAGE\$1PATH/extra\$1file* during the test). To use a more practical example, if you’re an Appium TestNG user who wants to include a *testng.xml* file with your test, you can include it in your archive using the following command: ``` $ zip zip-with-dependencies.zip testng.xml ``` Then, you can change your test command in the custom environment mode to the following: ``` java -D appium.screenshots.dir=$DEVICEFARM_SCREENSHOT_PATH org.testng.TestNG -testjar *-tests.jar -d $DEVICEFARM_LOG_DIR/test-output $DEVICEFARM_TEST_PACKAGE_PATH/testng.xml ``` If your test package upload extension isn't ZIP (e.g., APK, IPA, or JAR file), the uploaded package file itself is found at *\$1DEVICEFARM\$1TEST\$1PACKAGE\$1PATH*. Because these are still archive format files, you can unzip the file in order to access the additional files from within. For example, the following command will unzip the contents of the test package (for APK, IPA, or JAR files) to the */tmp* directory: ``` unzip $DEVICEFARM_TEST_PACKAGE_PATH -d /tmp ``` In the case of an APK or JAR file, you would find your extra files unzipped to the */tmp* directory (e.g., */tmp/extra\$1file*). In the case of an IPA file, as explained before, extra files would be in a slightly different location inside the folder ending in *.app*, which is inside of the *Payload* directory. For example, based on the IPA example above, the file would be found at the location */tmp/Payload/ADFiOSReferenceAppUITests-Runner.app/extra\$1file* (referenceable as */tmp/Payload/\$1.app/extra\$1file*). For more ways to extend your test suite and optimize your tests, see [Extending custom test environments in Device Farm](custom-test-environments-extending.md).