# How Image Builder uses the AWS Task Orchestrator and Executor application to manage components EC2 Image Builder uses the AWS Task Orchestrator and Executor (AWSTOE) application to orchestrate complex workflows, modify system configurations, and test your images without the need for additional devops scripts or code. This application manages and runs components that implement its declarative document schema. AWSTOE is a standalone application that Image Builder installs on its build and test instances when you create an image. You can also install it manually on EC2 instances to create your own custom components. It doesn't require any additional setup, and can also run on premises. **Topics** + [AWSTOE downloads](#toe-downloads) + [Supported Regions](#toe-supported-regions) + [AWSTOE command reference](#toe-commands) + [Manual set up to develop custom components with AWSTOE](toe-get-started.md) + [Use the AWSTOE component document framework for custom components](toe-use-documents.md) + [Action modules supported by AWSTOE component manager](toe-action-modules.md) + [Configure input for the AWSTOE run command](toe-run-config-input.md) ## AWSTOE downloads To install AWSTOE, choose the download link for your architecture and platform. If you attach to a VPC endpoint for your service (Image Builder, for example), it must have a custom endpoint policy attached that includes access to the S3 bucket for AWSTOE downloads. Otherwise, your build and test instances will not be able to download the bootstrap script (`bootstrap.sh`) and install the AWSTOE application. For more information see [Create a VPC endpoint policy for Image Builder](vpc-interface-endpoints.md#vpc-endpoint-policy). **Important** AWS is phasing out support for TLS versions 1.0 and 1.1. To access the S3 bucket for AWSTOE downloads, your client software must use TLS version 1.2 or later. For more information, see this [AWS Security Blog post](https://aws.amazon.com/blogs/security/tls-1-2-required-for-aws-endpoints/). | Architecture | Platform | Download link | Example | | --- | --- | --- | --- | | 386 | AL 2 and 2023 RHEL 7, 8, and 9 Ubuntu 16.04, 18.04, 20.04, 22.04, and 24.04 CentOS 7 and 8 SUSE 12 and 15 | `https://awstoe-.s3..amazonaws.com/latest/linux/386/awstoe` | [https://awstoe-us-east-1.s3.us-east-1.amazonaws.com/latest/linux/386/awstoe](https://awstoe-us-east-1.s3.us-east-1.amazonaws.com/latest/linux/386/awstoe) | | AMD64 | AL 2 and 2023 RHEL 7, 8, and 9 Ubuntu 16.04, 18.04, 20.04, 22.04, and 24.04 CentOS 7 and 8 CentOS Stream 8 SUSE 12 and 15 | https://awstoe-.s3..amazonaws.com/latest/linux/amd64/awstoe | [https://awstoe-us-east-1.s3.us-east-1.amazonaws.com/latest/linux/amd64/awstoe](https://awstoe-us-east-1.s3.us-east-1.amazonaws.com/latest/linux/amd64/awstoe) | | AMD64 | macOS 10.14.x (Mojave), 10.15.x (Catalina), 11.x (Big Sur), 12.x (Monterey) | https://awstoe-region.s3.region.amazonaws.com/latest/darwin/amd64/awstoe | [https://awstoe-us-east-1.s3.us-east-1.amazonaws.com/latest/darwin/amd64/awstoe](https://awstoe-us-east-1.s3.us-east-1.amazonaws.com/latest/darwin/amd64/awstoe) | | AMD64 | Windows Server 2012 R2, 2016, 2019, and 2022 | `https://awstoe-.s3..amazonaws.com/latest/windows/amd64/awstoe.exe` | [https://awstoe-us-east-1.s3.us-east-1.amazonaws.com/latest/windows/amd64/awstoe.exe](https://awstoe-us-east-1.s3.us-east-1.amazonaws.com/latest/windows/amd64/awstoe.exe) | | ARM64 | AL 2 and 2023 RHEL 7, 8, and 9 Ubuntu 16.04, 18.04, 20.04, 22.04, and 24.04 CentOS 7 and 8 CentOS Stream 8 SUSE 12 and 15 | https://awstoe-.s3..amazonaws.com/latest/linux/arm64/awstoe | [https://awstoe-us-east-1.s3.us-east-1.amazonaws.com/latest/linux/arm64/awstoe](https://awstoe-us-east-1.s3.us-east-1.amazonaws.com/latest/linux/arm64/awstoe) | ## Supported Regions AWSTOE is supported as a standalone application in the following Regions. | AWS Region name | AWS Region | | --- | --- | | US East (Ohio) | us-east-2 | | US East (N. Virginia) | us-east-1 | | AWS GovCloud (US-East) | us-gov-east-1 | | AWS GovCloud (US-West) | us-gov-west-1 | | US West (N. California) | us-west-1 | | US West (Oregon) | us-west-2 | | Africa (Cape Town) | af-south-1 | | Asia Pacific (Hong Kong) | ap-east-1 | | Asia Pacific (Osaka) | ap-northeast-3 | | Asia Pacific (Seoul) | ap-northeast-2 | | Asia Pacific (Mumbai) | ap-south-1 | | Asia Pacific (Hyderabad) | ap-south-2 | | Asia Pacific (Singapore) | ap-southeast-1 | | Asia Pacific (Sydney) | ap-southeast-2 | | Asia Pacific (Jakarta) | ap-southeast-3 | | Asia Pacific (Tokyo) | ap-northeast-1 | | Canada (Central) | ca-central-1 | | Europe (Frankfurt) | eu-central-1 | | Europe (Zurich) | eu-central-2 | | Europe (Stockholm) | eu-north-1 | | Europe (Milan) | eu-south-1 | | Europe (Spain) | eu-south-2 | | Europe (Ireland) | eu-west-1 | | Europe (London) | eu-west-2 | | Europe (Paris) | eu-west-3 | | Israel (Tel Aviv) | il-central-1 | | Middle East (UAE) | me-central-1 | | Middle East (Bahrain) | me-south-1 | | South America (São Paulo) | sa-east-1 | | China (Beijing) | cn-north-1 | | China (Ningxia) | cn-northwest-1 | ## AWSTOE command reference AWSTOE is a command line component management application that runs on Amazon EC2 instances. When Image Builder launches an EC2 build or test instance, it installs AWSTOE on the instance. Then it runs AWSTOE commands in the AWS CLI to install or validate the components that are specified in the image or container recipe. **Note** Some AWSTOE action modules require elevated permissions to run on a Linux server. To use elevated permissions, prefix the command syntax with **sudo**, or run the **sudo su** command one time when you log in before running the commands linked below. For more information about AWSTOE action modules, see [Action modules supported by AWSTOE component manager](toe-action-modules.md). ***[run](#cmd-run)*** Use the **run** command to run the YAML document scripts for one or more component documents. ***[validate](#cmd-validate)*** Run the **validate** command to validate the YAML document syntax for one or more component documents. ### awstoe run command This command runs the YAML component document scripts in the order in which they are included in the configuration file specified by the `--config` parameter, or the list of component documents specified by the `--documents` parameter. **Note** You must specify exactly one of the following parameters, never both: --config --documents #### Syntax ``` awstoe run [--config ] [--cw-ignore-failures ] [--cw-log-group ] [--cw-log-region us-west-2] [--cw-log-stream ] [--document-s3-bucket-owner ] [--documents ] [--execution-id ] [--log-directory ] [--log-s3-bucket-name ] [--log-s3-bucket-owner ] [--log-s3-key-prefix ] [--parameters name1=value1,name2=value2...] [--phases ] [--state-directory ] [--version ] [--help] [--trace] ``` #### Parameters and options Parameters **--config *`./config-example.json`*** Short form: -c *`./config-example.json`* The configuration file *(conditional)*. This parameter contains the file location for the JSON file that contains configuration settings for the components this command is running. If you specify **run** command settings in a configuration file, you must not specify the `--documents` parameter. For more information about input configuration, see [Configure input for the AWSTOE run command](toe-run-config-input.md). Valid locations include: + A local file path (*`./config-example.json`*) + An S3 URI (`s3://bucket/key`) **--cw-ignore-failures** Short form: N/A Ignore logging failures from the CloudWatch Logs. **--cw-log-group** Short form: N/A The `LogGroup` name for the CloudWatch Logs. **--cw-log-region** Short form: N/A The AWS Region that applies to the CloudWatch Logs. **--cw-log-stream** Short form: N/A The `LogStream` name for the CloudWatch Logs, that directs AWSTOE where to stream the `console.log` file. **--document-s3-bucket-owner** Short form: N/A The account ID of the bucket owner for S3 URI-based documents. **--documents *`./doc-1.yaml`,`./doc-n.yaml`*** Short form: -d *`./doc-1.yaml`*,*`./doc-n`* The component documents *(conditional)*. This parameter contains a comma-separated list of file locations for the YAML component documents to run. If you specify YAML documents for the **run** command using the `--documents` parameter, you must not specify the `--config` parameter. Valid locations include: + local file paths (*./component-doc-example.yaml*). + S3 URIs (`s3://bucket/key`). + Image Builder component build version ARNs (arn:aws:imagebuilder:us-west-*2:123456789012*:component/*my-example-component*/2021.12.02/1). There are no spaces between items in the list, only commas. **--execution-id** Short form: -i This is the unique ID that applies to the execution of the current **run** command. This ID is included in output and log file names, to uniquely identify those files, and link them to the current command execution. If this setting is left out, AWSTOE generates a GUID. **--log-directory** Short form: -l The destination directory where AWSTOE stores all of the log files from this command execution. By default, this directory is located inside of the following parent directory: `TOE__`. If you do not specify the log directory, AWSTOE uses the current working directory (`.`). **--log-s3-bucket-name** Short form: -b If component logs are stored in Amazon S3 (recommended), AWSTOE uploads the component application logs to the S3 bucket named in this parameter. **--log-s3-bucket-owner** Short form: N/A If component logs are stored in Amazon S3 (recommended), this is the owner account ID for the bucket where AWSTOE writes the log files. **--log-s3-key-prefix** Short form: -k If component logs are stored in Amazon S3 (recommended), this is the S3 object key prefix for the log location in the bucket. **--parameters *name1*=*value1*,*name2*=*value2*...** Short form: N/A Parameters are mutable variables that are defined in the component document, with settings that the calling application can provide at runtime. **--phases** Short form: -p A comma-separated list that specifies which phases to run from the YAML component documents. If a component document includes additional phases, those will not run. **--state-directory** Short form: -s The file path where state tracking files are stored. **--version** Short form: -v Specifies the component application version.Options **--help** Short form: -h Displays a help manual for using the component management application options. **--trace** Short form: -t Enables verbose logging to the console. ### awstoe validate command When you run this command, it validates the YAML document syntax for each of the component documents specified by the `--documents` parameter. #### Syntax ``` awstoe validate [--document-s3-bucket-owner ] --documents [--help] [--trace] ``` #### Parameters and options Parameters **--document-s3-bucket-owner** Short form: N/A Source account ID of S3 URI-based documents provided. **--documents *`./doc-1.yaml`,`./doc-n.yaml`*** Short form: -d *`./doc-1.yaml`*,*`./doc-n`* The component documents *(required)*. This parameter contains a comma-separated list of file locations for the YAML component documents to run. Valid locations include: + local file paths (*./component-doc-example.yaml*) + S3 URIs (`s3://bucket/key`) + Image Builder component build version ARNs (arn:aws:imagebuilder:us-west-*2:123456789012*:component/*my-example-component*/2021.12.02/1) There are no spaces between items in the list, only commas.Options **--help** Short form: -h Displays a help manual for using the component management application options. **--trace** Short form: -t Enables verbose logging to the console. # Manual set up to develop custom components with AWSTOE The AWS Task Orchestrator and Executor (AWSTOE) application is a standalone application that creates, validates, and runs commands within a component definition framework. AWS services can use AWSTOE to orchestrate workflows, install software, modify system configurations, and test image builds. Follow these steps to manually install the AWSTOE application and use it as a stand-alone application to develop custom components. Image Builder takes care of these steps for you, if you use the Image Builder console or AWS CLI commands to create custom components. For more information, see [Create custom components with Image Builder](create-component.md). # Verify the signature of the AWSTOE installation download This section describes the recommended process for verifying the validity of the installation download for AWSTOE on Linux, macOS and Windows based operating systems. **Topics** + [Verify the signature of the AWSTOE installation download on Linux or macOS](#awstoe-verify-sig-linux) + [Verify the signature of the AWSTOE installation download on Windows](#awstoe-verify-sig-win) ## Verify the signature of the AWSTOE installation download on Linux or macOS This topic describes the recommended process for verifying the validity of the installation download for the AWSTOE on Linux-based or macOS operating systems. Whenever you download an application from the internet, we recommend that you authenticate the identity of the software publisher. Also, check that the application is not altered or corrupted since it was published. This protects you from installing a version of the application that contains a virus or other malicious code. If, after running the steps in this topic, you determine that the software for the AWSTOE application is altered or corrupted, do not run the installation file. Instead, contact Support For more information about your support options, see [Support](https://aws.amazon.com/premiumsupport/). AWSTOE files for Linux-based and macOS operating systems are signed using `GnuPG`, an open source implementation of the Pretty Good Privacy (OpenPGP) standard for secure digital signatures. `GnuPG` (also known as `GPG`) provides authentication and integrity checking through a digital signature. Amazon EC2 publishes a public key and signatures that you can use to verify the downloaded Amazon EC2 CLI tools. For more information about `PGP` and `GnuPG` (`GPG`), see [http://www.gnupg.org](https://www.gnupg.org/). The first step is to establish trust with the software publisher. Download the public key of the software publisher, check that the owner of the public key is who they claim to be, and then add the public key to your *keyring*. Your keyring is a collection of known public keys. After you establish the authenticity of the public key, you can use it to verify the signature of the application. **Topics** + [Installing the GPG tools](#awstoe-verify-signature-of-binary-download-install-tools) + [Authenticating and importing the public key](#awstoe-verify-signature-of-binary-download-authenticate-import-public-key) + [Verify the signature of the package](#awstoe-verify-signature-of-binary-package) ### Installing the GPG tools If your operating system is Linux, Unix, or macOS, the GPG tools are likely already installed. To test whether the tools are installed on your system, type **gpg** at a command prompt. If the GPG tools are installed, you see a GPG command prompt. If the GPG tools are not installed, you see an error message stating that the command cannot be found. You can install the GnuPG package from a repository. To install GPG tools, select the operating system that matches your instance. ------ #### [ Debian-based Linux ] From a terminal, run the following command: ``` apt-get install gnupg ``` ------ #### [ Red Hat–based Linux ] From a terminal, run the following command: ``` yum install gnupg ``` ------ #### [ macOS ] From a terminal, run the following command: ``` brew install gnupg ``` ------ ### Authenticating and importing the public key The next step in the process is to authenticate the AWSTOE public key and add it as a trusted key in your `GPG` keyring. **To authenticate and import the AWSTOE public key** 1. Obtain a copy of our public `GPG` build key by doing one of the following: + Download the key from https://awstoe-****.s3.****.amazonaws.com/assets/awstoe.gpg. For example, [https://awstoe-us-east-1.s3.us-east-1.amazonaws.com/latest/assets/awstoe.gpg](https://awstoe-us-east-1.s3.us-east-1.amazonaws.com/latest/assets/awstoe.gpg). + Copy the key from the following text and paste it into a file called `awstoe.gpg`. Make sure to include everything that follows: ``` -----BEGIN PGP PUBLIC KEY BLOCK----- Version: GnuPG v2 mQENBF8UqwsBCACdiRF2bkZYaFSDPFC+LIkWLwFvtUCRwAHtD8KIwTJ6LVn3fHAU GhuK0ZH9mRrqRT2bq/xJjGsnF9VqTj2AJqndGJdDjz75YCZYM+ocZ+r5HSJaeW9i S5dykHj7Txti2zHe0G5+W0v7v5bPi2sPHsN7XWQ7+G2AMEPTz8PjxY//I0DvMQns Sle3l9hz6wCClz1l9LbBzTyHfSm5ucTXvNe88XX5Gmt37OCDM7vfli0Ctv8WFoLN 6jbxuA/sV71yIkPm9IYp3+GvaKeT870+sn8/JOOKE/U4sJV1ppbqmuUzDfhrZUaw 8eW8IN9A1FTIuWiZED/5L83UZuQs1S7s2PjlABEBAAG0GkFXU1RPRSA8YXdzdG9l QGFtYXpvbi5jb20+iQE5BBMBCAAjBQJfFKsLAhsDBwsJCAcDAgEGFQgCCQoLBBYC AwECHgECF4AACgkQ3r3BVvWuvFJGiwf9EVmrBR77+Qe/DUeXZJYoaFr7If/fVDZl 6V3TC6p0J0Veme7uXleRUTFOjzbh+7e5sDX19HrnPquzCnzfMiqbp4lSoeUuNdOf FcpuTCQH+M+sIEIgPno4PLl0Uj2uE1o++mxmonBl/Krk+hly8hB2L/9n/vW3L7BN OMb1Ll9PmgGPbWipcT8KRdz4SUex9TXGYzjlWb3jU3uXetdaQY1M3kVKE1siRsRN YYDtpcjmwbhjpu4xm19aFqNoAHCDctEsXJA/mkU3erwIRocPyjAZE2dnlkL9ZkFZ z9DQkcIarbCnybDM5lemBbdhXJ6hezJE/b17VA0t1fY04MoEkn6oJg== =oyze -----END PGP PUBLIC KEY BLOCK----- ``` 1. At a command prompt in the directory where you saved **awstoe.gpg**, use the following command to import the AWSTOE public key into your keyring. ``` gpg --import awstoe.gpg ``` The command returns results that are similar to the following: ``` gpg: key F5AEBC52: public key "AWSTOE " imported gpg: Total number processed: 1 gpg: imported: 1 (RSA: 1) ``` Make a note of the key value; you need it in the next step. In the preceding example, the key value is `F5AEBC52`. 1. Verify the fingerprint by running the following command, replacing *key-value* with the value from the preceding step: ``` gpg --fingerprint key-value ``` This command returns results similar to the following: ``` pub 2048R/F5AEBC52 2020-07-19 Key fingerprint = F6DD E01C 869F D639 15E5 5742 DEBD C156 F5AE BC52 uid [ unknown] AWSTOE ``` Additionally, the fingerprint string should be identical to `F6DD E01C 869F D639 15E5 5742 DEBD C156 F5AE BC52`, as shown in the preceding example. Compare the key fingerprint that is returned to the one published on this page. They should match. If they don't match, do not install the AWSTOE installation script, and contact Support. ### Verify the signature of the package After you install the `GPG` tools, authenticate and import the AWSTOE public key, and verify that the public key is trusted, you are ready to verify the signature of the installation script. **To verify the installation script signature** 1. At a command prompt, run the following command to download the application binary: ``` curl -O https://awstoe-.s3..amazonaws.com/latest/linux//awstoe ``` For example: ``` curl -O https://awstoe-us-east-1.s3.us-east-1.amazonaws.com/latest/linux/amd64/awstoe ``` Supported values for **architecture** can be `amd64`, `386`, and `arm64`. 1. At a command prompt, run the following command to download the signature file for the corresponding application binary from the same S3 key prefix path: ``` curl -O https://awstoe-.s3..amazonaws.com/latest/linux//awstoe.sig ``` For example: ``` curl -O https://awstoe-us-east-1.s3.us-east-1.amazonaws.com/latest/linux/amd64/awstoe.sig ``` Supported values for **architecture** can be `amd64`, `386`, and `arm64`. 1. Verify the signature by running the following command at a command prompt in the directory where you saved `awstoe.sig` and the AWSTOE installation file. Both files must be present. ``` gpg --verify ./awstoe.sig ~/awstoe ``` The output should look something like the following: ``` gpg: Signature made Mon 20 Jul 2020 08:54:55 AM IST using RSA key ID F5AEBC52 gpg: Good signature from "AWSTOE awstoe@amazon.com" [unknown] gpg: WARNING: This key is not certified with a trusted signature! gpg: There is no indication that the signature belongs to the owner. Primary key fingerprint: F6DD E01C 869F D639 15E5 5742 DEBD C156 F5AE BC52 ``` If the output contains the phrase `Good signature from "AWSTOE "`, it means that the signature has successfully been verified, and you can proceed to run the AWSTOE installation script. If the output includes the phrase `BAD signature`, check whether you performed the procedure correctly. If you continue to get this response, do not run the installation file that you downloaded previously, and contact Support. The following are details about the warnings that you might see: + **WARNING: This key is not certified with a trusted signature\$1 There is no indication that the signature belongs to the owner.** Ideally, you would visit an AWS office and receive the key in person. However, you would most likely download it from a website. In this case, the website is an AWS website. + **gpg: no ultimately trusted keys found.** This means that the specific key is not "ultimately trusted" by you, or by other people that you trust. For more information, see [http://www.gnupg.org](http://www.gnupg.org). ## Verify the signature of the AWSTOE installation download on Windows This topic describes the recommended process for verifying the validity of the installation file for the AWS Task Orchestrator and Executor application on Windows-based operating systems. Whenever you download an application from the internet, we recommend that you authenticate the identity of the software publisher and check that the application is not altered or corrupted since it was published. This protects you from installing a version of the application that contains a virus or other malicious code. If, after running the steps in this topic, you determine that the software for the AWSTOE application is altered or corrupted, do not run the installation file. Instead, contact Support. To verify the validity of the downloaded awstoe binary on Windows-based operating systems, make sure that the thumbprint of its Amazon Services LLC signer certificate is equal to this value: **9D CA 72 17 DA FF B8 2F E4 C4 67 77 36 2F A4 AA C9 08 82 15** **Note** During the roll-out window for a new binary, your signer certificate might not match the new thumbprint. If your signer certificate doesn't match, verify that the thumbprint value is: **BA 81 25 EE AC 64 2E A9 F3 C5 93 CA 6D 3E B7 93 7D 68 75 74** To verify this value, perform the following procedure: 1. Right-click the downloaded `awstoe.exe`, and open the **Properties** window. 1. Choose the **Digital Signatures** tab. 1. From the **Signature List**, choose **Amazon Services LLC**, and then choose **Details**. 1. Choose the **General** tab, if not already selected, and then choose **View Certificate**. 1. Choose the **Details** tab, and then choose **All** in the **Show** dropdown list, if not already selected. 1. Scroll down until you see the **Thumbprint** field and then choose **Thumbprint**. This displays the entire thumbprint value in the lower window. + If the thumbprint value in the lower window is identical to the following value: **9D CA 72 17 DA FF B8 2F E4 C4 67 77 36 2F A4 AA C9 08 82 15** then your downloaded AWSTOE binary is authentic and can be safely installed. + If the thumbprint value in the lower details window is not identical to the previous value, do not run `awstoe.exe`. **Topics** + [Verify the signature of the AWSTOE installation download](awstoe-verify-sig.md) + [Step 1: Install AWSTOE](#toe-start-install) + [Step 2: Set AWS credentials](#toe-start-credentials) + [Step 3: Develop component documents locally](#toe-start-develop) + [Step 4: Validate AWSTOE components](#toe-start-validate) + [Step 5: Run AWSTOE components](#toe-start-run) ## Step 1: Install AWSTOE To develop components locally, download and install the AWSTOE application. 1. **Download the AWSTOE application** To install AWSTOE, choose the appropriate download link for your architecture and platform. For the full list of application download links, see [AWSTOE downloads](toe-component-manager.md#toe-downloads) **Important** AWS is phasing out support for TLS versions 1.0 and 1.1. To access the S3 bucket for AWSTOE downloads, your client software must use TLS version 1.2 or later. For more information, see this [AWS Security Blog post](https://aws.amazon.com/blogs/security/tls-1-2-required-for-aws-endpoints/). 1. **Verify the signature** The steps for verifying your download depend on the server platform where you run the AWSTOE application after you install it. To verify your download on a Linux server, see [Verify the signature on Linux or macOS](awstoe-verify-sig.md#awstoe-verify-sig-linux). To verify your download on a Windows server, see [Verify the signature on Windows](awstoe-verify-sig.md#awstoe-verify-sig-win). **Note** AWSTOE is invoked directly from its download location. There is no need for a separate install step. This also means that AWSTOE can make changes to the local environment. To ensure that you isolate changes during component development, we recommend that you use an EC2 instance to develop and test AWSTOE components. ## Step 2: Set AWS credentials AWSTOE requires AWS credentials to connect to other AWS services, such as Amazon S3 and Amazon CloudWatch, when running tasks, such as: + Downloading AWSTOE documents from a user-provided Amazon S3 path. + Running `S3Download` or `S3Upload` action modules. + Streaming logs to CloudWatch, when enabled. If you are running AWSTOE on an EC2 instance, then running AWSTOE uses the same permissions as the IAM role attached to the EC2 instance. For more information about IAM roles for EC2, see [IAM roles for Amazon EC2](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html). The following examples show how to set AWS credentials using the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables. To set these variables on Linux, macOS, or Unix, use `export`. ``` export AWS_ACCESS_KEY_ID=your_access_key_id ``` ``` export AWS_SECRET_ACCESS_KEY=your_secret_access_key ``` To set these variables on Windows using PowerShell, use `$env`. ``` $env:AWS_ACCESS_KEY_ID=your_access_key_id ``` ``` $env:AWS_SECRET_ACCESS_KEY=your_secret_access_key ``` To set these variables on Windows using the command prompt, use `set`. ``` set AWS_ACCESS_KEY_ID=your_access_key_id ``` ``` set AWS_SECRET_ACCESS_KEY=your_secret_access_key ``` ## Step 3: Develop component documents locally Components are authored with plaintext YAML documents. For more information about document syntax, see [Use the AWSTOE component document framework for custom components](toe-use-documents.md). The following are example *Hello World* component documents to help you get started. ------ #### [ Linux ] Some of the Linux component examples in this guide refer to a component document file named `hello-world-linux.yml`. You can use the following document to get started with those examples. ``` name: Hello World description: This is hello world testing document for Linux. schemaVersion: 1.0 phases: - name: build steps: - name: HelloWorldStep action: ExecuteBash inputs: commands: - echo 'Hello World from the build phase.' - name: validate steps: - name: HelloWorldStep action: ExecuteBash inputs: commands: - echo 'Hello World from the validate phase.' - name: test steps: - name: HelloWorldStep action: ExecuteBash inputs: commands: - echo 'Hello World from the test phase.' ``` ------ #### [ Windows ] Some of the Windows component examples in this guide refer to a component document file named `hello-world-windows.yml`. You can use the following document to get started with those examples. ``` name: Hello World description: This is Hello World testing document for Windows. schemaVersion: 1.0 phases: - name: build steps: - name: HelloWorldStep action: ExecutePowerShell inputs: commands: - Write-Host 'Hello World from the build phase.' - name: validate steps: - name: HelloWorldStep action: ExecutePowerShell inputs: commands: - Write-Host 'Hello World from the validate phase.' - name: test steps: - name: HelloWorldStep action: ExecutePowerShell inputs: commands: - Write-Host 'Hello World from the test phase.' ``` ------ #### [ macOS ] Some of the macOS component examples in this guide refer to a component document file named `hello-world-macos.yml`. You can use the following document to get started with those examples. ``` name: Hello World description: This is hello world testing document for macOS. schemaVersion: 1.0 phases: - name: build steps: - name: HelloWorldStep action: ExecuteBash inputs: commands: - echo 'Hello World from the build phase.' - name: validate steps: - name: HelloWorldStep action: ExecuteBash inputs: commands: - echo 'Hello World from the validate phase.' - name: test steps: - name: HelloWorldStep action: ExecuteBash inputs: commands: - echo 'Hello World from the test phase.' ``` ------ ## Step 4: Validate AWSTOE components You can validate the syntax of AWSTOE components locally with the AWSTOE application. The following examples show the AWSTOE application `validate` command to validate the syntax of a component without running it. **Note** The AWSTOE application can validate only the component syntax for the current operating system. For example, when running `awstoe.exe` on Windows, you cannot validate the syntax for a Linux document that uses the `ExecuteBash` action module. Linux or macOS ``` awstoe validate --documents /home/user/hello-world.yml ``` Windows ``` awstoe.exe validate --documents C:\Users\user\Documents\hello-world.yml ``` ## Step 5: Run AWSTOE components The AWSTOE application can run one or more phases of specified documents using the `--phases` command line argument. Supported values for `--phases` are `build`, `validate`, and `test`. Multiple phase values can be entered as comma separated values. When you provide a list of phases, the AWSTOE application sequentially runs the specified phases of each document. For example, AWSTOE runs the `build` and `validate` phases of `document1.yaml`, followed by the `build` and `validate` phases of `document2.yaml`. To ensure that your logs are stored securely and retained for troubleshooting, we recommend configuring log storage in Amazon S3. In Image Builder, the Amazon S3 location for publishing logs is specified in the infrastructure configuration. For more information about infrastructure configuration, see [Manage Image Builder infrastructure configuration](manage-infra-config.md) If a list of phases is not provided, the AWSTOE application runs all phases in the order listed in the YAML document. To run specific phases in single or multiple documents, use the following commands. Single phase ``` awstoe run --documents hello-world.yml --phases build ``` Multiple phases ``` awstoe run --documents hello-world.yml --phases build,test ``` **Document run** Run all phases in a single document ``` awstoe run --documents documentName.yaml ``` Run all phases in multiple documents ``` awstoe run --documents documentName1.yaml,documentName2.yaml ``` Enter Amazon S3 information to upload AWSTOE logs from a user-defined local path (recommended) ``` awstoe run --documents documentName.yaml --log-s3-bucket-name amzn-s3-demo-destination-bucket --log-s3-key-prefix S3KeyPrefix --log-s3-bucket-owner S3BucketOwner --log-directory local_path ``` Run all phases in a single document, and display all logs on the console ``` awstoe run --documents documentName.yaml --trace ``` Example command ``` awstoe run --documents s3://bucket/key/doc.yaml --phases build,validate ``` Run document with unique ID ``` awstoe run --documents documentName.yaml --execution-id user-provided-id --phases build,test ``` Get help with AWSTOE ``` awstoe --help ``` # Use the AWSTOE component document framework for custom components To build a component using the AWS Task Orchestrator and Executor (AWSTOE) component framework, you must provide a YAML-based document that represents the phases and steps that apply for the component you create. AWS services use your component when they create a new Amazon Machine Image (AMI) or container image. **Topics** + [Component document workflow](#component-doc-workflow) + [Component logging](#component-logging) + [Input and output chaining](#document-chaining) + [Document schema and definitions](#document-schema) + [Document examples](#document-example) + [Use variables in your custom component document](toe-user-defined-variables.md) + [Use conditional constructs in AWSTOE](toe-conditional-constructs.md) + [Use comparison operators in AWSTOE component documents](toe-comparison-operators.md) + [Use logical operators in AWSTOE component documents](toe-logical-operators.md) + [Use looping constructs in AWSTOE](toe-looping-constructs.md) ## Component document workflow The AWSTOE component document uses phases and steps to group related tasks, and organize those tasks into a logical workflow for the component. **Tip** The service that uses your component to build an image might implement rules about what phases to use for their build process, and when those phases are allowed to run. This is important to consider when you design your component. **Phases** Phases represent the progression of your workflow through the image build process. For example, the Image Builder service uses `build` and `validate` phases during its *build stage* for the images it produces. It uses the `test` and `container-host-test` phases during its *test stage* to ensure that the image snapshot or container image produces the expected results before creating the final AMI or distributing the container image. When the component runs, the associated commands for each phase are applied in the order that they appear in the component document. **Rules for phases** + Each phase name must be unique within a document. + You can define many phases in your document. + You must include at least one of the following phases in your document: + **build** – for Image Builder, this phase is generally used during the *build stage*. + **validate** – for Image Builder, this phase is generally used during the *build stage*. + **test** – for Image Builder, this phase is generally used during the *test stage*. + Phases always run in the order that they are defined in the document. The order in which they are specified for AWSTOE commands in the AWS CLI has no effect. **Steps** Steps are individual units of work that define the workflow within each phase. Steps run in sequential order. However, input or output for one step can also feed into a subsequent step as input. This is called "chaining". **Rules for steps** + The step name must be unique for the phase. + The step must use a supported action (action module) that returns an exit code. For a complete list of supported action modules, how they work, input/output values, and examples, see [Action modules supported by AWSTOE component manager](toe-action-modules.md). ## Component logging AWSTOE creates a new log folder on the EC2 instances that are used for building and testing a new image, each time your component runs. For container images, the log folder is stored in the container. To assist with troubleshooting if something goes wrong during the image creation process, the input document and all of the output files AWSTOE creates while running the component are stored in the log folder. The log folder name is comprised of the following parts: 1. **Log directory** – when a service runs a AWSTOE component, it passes in the log directory, along with other settings for the command. For the following examples, we show the log file format that Image Builder uses. + **Linux and macOS**: `/var/lib/amazon/toe/` + **Windows**: `$env:ProgramFiles\Amazon\TaskOrchestratorAndExecutor\` 1. **File prefix** – This is a standard prefix used for all components: "`TOE_`". 1. **Run time** – This is a timestamp in YYYY-MM-DD\$1HH-MM-SS\$1UTC-0 format. 1. **Execution ID** – This is the GUID that is assigned when AWSTOE runs one or more components. Example: `/var/lib/amazon/toe/TOE_2021-07-01_12-34-56_UTC-0_a1bcd2e3-45f6-789a-bcde-0fa1b2c3def4` AWSTOE stores the following core files in the log folder: **Input files** + **document.yaml** – The document that is used as input for the command. After the component runs, this file is stored as an artifact. **Output files** + **application.log** – The application log contains timestamped debug level information from AWSTOE about what's happening as the component is running. + **detailedoutput.json** – This JSON file has detailed information about run status, inputs, outputs, and failures for all documents, phases, and steps that apply for the component as it runs. + **console.log** – The console log contains all of the standard out (stdout) and standard error (stderr) information that AWSTOE writes to the console while the component is running. + **chaining.json** – This JSON file represents optimizations that AWSTOE applied to resolve chaining expressions. **Note** The log folder might also contain other temporary files that are not covered here. ## Input and output chaining The AWSTOE configuration management application provides a feature for chaining inputs and outputs by writing references in the following formats: `{{ phase_name.step_name.inputs/outputs.variable }}` or `{{ phase_name.step_name.inputs/outputs[index].variable }}` The chaining feature allows you to recycle code and improve the maintainability of the document. **Rules for chaining** + Chaining expressions can be used only in the inputs section of each step. + Statements with chaining expressions must be enclosed in quotes. For example: + **Invalid expression**: `echo {{ phase.step.inputs.variable }}` + **Valid expression**: `"echo {{ phase.step.inputs.variable }}"` + **Valid expression**: `'echo {{ phase.step.inputs.variable }}'` + Chaining expressions can reference variables from other steps and phases in the same document. However, the calling service might have rules that require chaining expressions to operate only within the context of a single stage. For example, Image Builder does not support chaining from the *build stage* to the *test stage*, as it runs each stage independently. + Indexes in chaining expressions follow zero-based indexing. The index starts with zero (0) to reference the first element. **Examples** To refer to the source variable in the second entry of the following example step, the chaining pattern is `{{ build.SampleS3Download.inputs[1].source }}`. ``` phases: - name: 'build' steps: - name: SampleS3Download action: S3Download timeoutSeconds: 60 onFailure: Abort maxAttempts: 3 inputs: - source: 's3://sample-bucket/sample1.ps1' destination: 'C:\sample1.ps1' - source: 's3://sample-bucket/sample2.ps1' destination: 'C:\sample2.ps1' ``` To refer to the output variable (equal to "Hello") of the following example step, the chaining pattern is `{{ build.SamplePowerShellStep.outputs.stdout }}`. ``` phases: - name: 'build' steps: - name: SamplePowerShellStep action: ExecutePowerShell timeoutSeconds: 120 onFailure: Abort maxAttempts: 3 inputs: commands: - 'Write-Host "Hello"' ``` ## Document schema and definitions The following is the YAML schema for a document. ``` name: (optional) description: (optional) schemaVersion: "string" phases: - name: "string" steps: - name: "string" action: "string" timeoutSeconds: integer onFailure: "Abort|Continue|Ignore" maxAttempts: integer inputs: ``` The schema definitions for a document are as follows. | Field | Description | Type | Required | | --- | --- | --- | --- | | name | Name of the document. | String | No | | description | Description of the document. | String | No | | schemaVersion | Schema version of the document, currently 1.0. | String | Yes | | phases | A list of phases with their steps. | List | Yes | The schema definitions for a phase are as follows. | Field | Description | Type | Required | | --- | --- | --- | --- | | name | Name of the phase. | String | Yes | | steps | List of the steps in the phase. | List | Yes | The schema definitions for a step are as follows. | Field | Description | Type | Required | Default value | | --- | --- | --- | --- | --- | | name | User-defined name for the step. | String | | | | action | Keyword pertaining to the module that runs the step. | String | | | | timeoutSeconds | Number of seconds that the step runs before failing or retrying. Also, supports -1 value, which indicates infinite timeout. 0 and other negative values are not allowed. | Integer | No | 7,200 sec (120 mins) | | onFailure | Specifies what the step should do in case of failure. Valid values are as follows: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/imagebuilder/latest/userguide/toe-use-documents.html) | String | No | Abort | | maxAttempts | Maximum number of attempts allowed before failing the step. | Integer | No | 1 | | inputs | Contains parameters required by the action module to run the step. | Dict | Yes | | ## Document examples The following examples show AWSTOE component documents that perform tasks for the target operating system. ------ #### [ Linux ] **Example 1: Run a custom binary file** The following is an example document that downloads and runs a custom binary file on a Linux instance. ``` name: LinuxBin description: Download and run a custom Linux binary file. schemaVersion: 1.0 phases: - name: build steps: - name: Download action: S3Download inputs: - source: s3://amzn-s3-demo-source-bucket/myapplication destination: /tmp/myapplication - name: Enable action: ExecuteBash onFailure: Continue inputs: commands: - 'chmod u+x {{ build.Download.inputs[0].destination }}' - name: Install action: ExecuteBinary onFailure: Continue inputs: path: '{{ build.Download.inputs[0].destination }}' arguments: - '--install' - name: Delete action: DeleteFile inputs: - path: '{{ build.Download.inputs[0].destination }}' ``` ------ #### [ Windows ] **Example 1: Install Windows updates** The following is an example document that installs all available Windows updates, runs a configuration script, validates the changes before the AMI is created, and tests the changes after the AMI is created. ``` name: RunConfig_UpdateWindows description: 'This document will install all available Windows updates and run a config script. It will then validate the changes before an AMI is created. Then after AMI creation, it will test all the changes.' schemaVersion: 1.0 phases: - name: build steps: - name: DownloadConfigScript action: S3Download timeoutSeconds: 60 onFailure: Abort maxAttempts: 3 inputs: - source: 's3://customer-bucket/config.ps1' destination: 'C:\config.ps1' - name: RunConfigScript action: ExecutePowerShell timeoutSeconds: 120 onFailure: Abort maxAttempts: 3 inputs: file: '{{build.DownloadConfigScript.inputs[0].destination}}' - name: Cleanup action: DeleteFile onFailure: Abort maxAttempts: 3 inputs: - path: '{{build.DownloadConfigScript.inputs[0].destination}}' - name: RebootAfterConfigApplied action: Reboot inputs: delaySeconds: 60 - name: InstallWindowsUpdates action: UpdateOS - name: validate steps: - name: DownloadTestConfigScript action: S3Download timeoutSeconds: 60 onFailure: Abort maxAttempts: 3 inputs: - source: 's3://customer-bucket/testConfig.ps1' destination: 'C:\testConfig.ps1' - name: ValidateConfigScript action: ExecutePowerShell timeoutSeconds: 120 onFailure: Abort maxAttempts: 3 inputs: file: '{{validate.DownloadTestConfigScript.inputs[0].destination}}' - name: Cleanup action: DeleteFile onFailure: Abort maxAttempts: 3 inputs: - path: '{{validate.DownloadTestConfigScript.inputs[0].destination}}' - name: test steps: - name: DownloadTestConfigScript action: S3Download timeoutSeconds: 60 onFailure: Abort maxAttempts: 3 inputs: - source: 's3://customer-bucket/testConfig.ps1' destination: 'C:\testConfig.ps1' - name: ValidateConfigScript action: ExecutePowerShell timeoutSeconds: 120 onFailure: Abort maxAttempts: 3 inputs: file: '{{test.DownloadTestConfigScript.inputs[0].destination}}' ``` **Example 2: Install the AWS CLI on a Windows instance** The following is an example document that installs the AWS CLI on a Windows instance, using the setup file. ``` name: InstallCLISetUp description: Install &CLI; using the setup file schemaVersion: 1.0 phases: - name: build steps: - name: Download action: S3Download inputs: - source: s3://aws-cli/AWSCLISetup.exe destination: C:\Windows\temp\AWSCLISetup.exe - name: Install action: ExecuteBinary onFailure: Continue inputs: path: '{{ build.Download.inputs[0].destination }}' arguments: - '/install' - '/quiet' - '/norestart' - name: Delete action: DeleteFile inputs: - path: '{{ build.Download.inputs[0].destination }}' ``` **Example 3: Install the AWS CLI with the MSI installer** The following is an example document that installs the AWS CLI with the MSI installer. ``` name: InstallCLIMSI description: Install &CLI; using the MSI installer schemaVersion: 1.0 phases: - name: build steps: - name: Download action: S3Download inputs: - source: s3://aws-cli/AWSCLI64PY3.msi destination: C:\Windows\temp\AWSCLI64PY3.msi - name: Install action: ExecuteBinary onFailure: Continue inputs: path: 'C:\Windows\System32\msiexec.exe' arguments: - '/i' - '{{ build.Download.inputs[0].destination }}' - '/quiet' - '/norestart' - name: Delete action: DeleteFile inputs: - path: '{{ build.Download.inputs[0].destination }}' ``` ------ #### [ macOS ] **Example 1: Run a custom macOS binary file** The following is an example document that downloads and runs a custom binary file on a macOS instance. ``` name: macOSBin description: Download and run a binary file on macOS. schemaVersion: 1.0 phases: - name: build steps: - name: Download action: S3Download inputs: - source: s3://amzn-s3-demo-source-bucket/myapplication destination: /tmp/myapplication - name: Enable action: ExecuteBash onFailure: Continue inputs: commands: - 'chmod u+x {{ build.Download.inputs[0].destination }}' - name: Install action: ExecuteBinary onFailure: Continue inputs: path: '{{ build.Download.inputs[0].destination }}' arguments: - '--install' - name: Delete action: DeleteFile inputs: - path: '{{ build.Download.inputs[0].destination }}' ``` ------ # Use variables in your custom component document Variables provide a way to label data with meaningful names that can be used throughout an application. You can define custom variables with simple and readable formats for complex workflows, and reference them in the YAML application component document for an AWSTOE component. This section provides information to help you define variables for your AWSTOE component in the YAML application component document, including syntax, name constraints, and examples. ## Constants Constants are immutable variables that cannot be modified or overridden once defined. Constants can be defined using values in the `constants` section of an AWSTOE document. **Rules for constant names** + The name must be between 3 and 128 characters in length. + The name can only contain alphanumeric characters (a-z, A-Z, 0-9), dashes (-), or underscores (\$1). + The name must be unique within the document. + The name must be specified as a YAML string. **Syntax** ``` constants: - : type: value: ``` | Key name | Required | Description | | --- | --- | --- | | `name` | Yes | Name of the constant. Must be unique for the document (it must not be the same as any other parameter names or constants). | | `value` | Yes | Value of the constant. | | `type` | Yes | Type of the constant. Supported type is string. | **Reference constant values in a document** You can reference constants in step or loop inputs inside of your YAML document, as follows: + Constant references are case-sensitive, and the name must match exactly. + The name must be enclosed within double curly braces `{{` *MyConstant* `}}`. + Spaces are allowed within the curly braces, and are automatically trimmed. For example, all of the following references are valid: `{{ MyConstant }}`, `{{ MyConstant}}`, `{{MyConstant }}`, `{{MyConstant}}` + The reference in the YAML document must be specified as a string (enclosed in single or double quotes). For example: `- {{ MyConstant }}` is not valid, as it is not identified as a string. However, the following references are both valid: `- '{{ MyConstant }}'` and `- "{{ MyConstant }}"`. **Examples** Constant referenced in step inputs ``` name: Download AWS CLI version 2 schemaVersion: 1.0 constants: - Source: type: string value: https://awscli.amazonaws.com/AWSCLIV2.msi phases: - name: build steps: - name: Download action: WebDownload inputs: - source: '{{ Source }}' destination: 'C:\Windows\Temp\AWSCLIV2.msi' ``` Constant referenced in loop inputs ``` name: PingHosts schemaVersion: 1.0 constants: - Hosts: type: string value: 127.0.0.1,amazon.com phases: - name: build steps: - name: Ping action: ExecuteBash loop: forEach: list: '{{ Hosts }}' delimiter: ',' inputs: commands: - ping -c 4 {{ loop.value }} ``` ## Parameters Parameters are mutable variables, with settings that the calling application can provide at runtime. You can define parameters in the `Parameters` section of the YAML document. **Rules for parameter names** + The name must be between 3 and 128 characters in length. + The name can only contain alphanumeric characters (a-z, A-Z, 0-9), dashes (-), or underscores (\$1). + The name must be unique within the document. + The name must be specified as a YAML string. ### Syntax ``` parameters: - : type: default: description: ``` | Key name | Required | Description | | --- | --- | --- | | `name` | Yes | The name of the parameter. Must be unique for the document (it must not be the same as any other parameter names or constants). | | `type` | Yes | The data type of the parameter. Supported types include: `string`. | | `default` | No | The default value for the parameter. | | `description` | No | Describes the parameter. | ### Reference parameter values in a document You can reference parameters in step or loop inputs inside of your YAML document, as follows: + Parameter references are case-sensitive, and the name must match exactly. + The name must be enclosed within double curly braces `{{` *MyParameter* `}}`. + Spaces are allowed within the curly braces, and are automatically trimmed. For example, all of the following references are valid: `{{ MyParameter }}`, `{{ MyParameter}}`, `{{MyParameter }}`, `{{MyParameter}}` + The reference in the YAML document must be specified as a string (enclosed in single or double quotes). For example: `- {{ MyParameter }}` is not valid, as it is not identified as a string. However, the following references are both valid: `- '{{ MyParameter }}'` and `- "{{ MyParameter }}"`. **Examples** The following examples show how to use parameters in your YAML document: + Refer to a parameter in step inputs: ``` name: Download AWS CLI version 2 schemaVersion: 1.0 parameters: - Source: type: string default: 'https://awscli.amazonaws.com/AWSCLIV2.msi' description: The AWS CLI installer source URL. phases: - name: build steps: - name: Download action: WebDownload inputs: - source: '{{ Source }}' destination: 'C:\Windows\Temp\AWSCLIV2.msi' ``` + Refer to a parameter in loop inputs: ``` name: PingHosts schemaVersion: 1.0 parameters: - Hosts: type: string default: 127.0.0.1,amazon.com description: A comma separated list of hosts to ping. phases: - name: build steps: - name: Ping action: ExecuteBash loop: forEach: list: '{{ Hosts }}' delimiter: ',' inputs: commands: - ping -c 4 {{ loop.value }} ``` ### Override parameters at runtime You can use the `--parameters` option from the AWS CLI with a key-value pair to set a parameter value at runtime. + Specify the parameter key-value pair as the name and value, separated by an equals sign (=). + Multiple parameters must be separated by a comma. + Parameter names that are not found in the YAML component document are ignored. + The parameter name and value are both required. **Important** Component parameters are plain text values, and are logged in AWS CloudTrail. We recommend that you use AWS Secrets Manager or the AWS Systems Manager Parameter Store to store your secrets. For more information about Secrets Manager, see [What is Secrets Manager?](https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html) in the *AWS Secrets Manager User Guide*. For more information about AWS Systems Manager Parameter Store, see [AWS Systems Manager Parameter Store](https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-parameter-store.html) in the *AWS Systems Manager User Guide*. #### Syntax ``` --parameters name1=value1,name2=value2... ``` | CLI option | Required | Description | | --- | --- | --- | | --parameters *name*=*value*,... | No | This option takes list of key-value pairs, with the parameter name as the key. | **Examples** The following examples show how to use parameters in your YAML document: + The parameter key-value pair specified in this `--parameter` option is not valid: ``` --parameters ntp-server= ``` + Set one parameter key-value pair with the `--parameter` option in the AWS CLI: ``` --parameters ntp-server=ntp-server-windows-qe.us-east1.amazon.com ``` + Set multiple parameter key-value pairs with the `--parameter` option in the AWS CLI: ``` --parameters ntp-server=ntp-server.amazon.com,http-url=https://internal-us-east1.amazon.com ``` ## Use Systems Manager Parameter Store parameters You can reference AWS Systems Manager Parameter Store parameters (SSM parameters) in your component documents by prefixing variables with `aws:ssm`. For example, `{{ aws:ssm:/my/param }}` resolves to the value of the SSM parameter `/my/param`. This feature supports the following SSM parameter types: + String – Maps to the AWSTOE string type. + StringList – Maps to the AWSTOE `stringList` type. + SecureString – Maps to the AWSTOE string type. For more information about the Parameter Store see [AWS Systems Manager Parameter Store](https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-parameter-store.html) in the *AWS Systems Manager User Guide*. You can also reference AWS Secrets Manager secrets using an SSM parameter `SecureString`. For example: `{{ aws:ssm:/aws/reference/secretsmanager/test/test-secret }}`. For more information, see [Referencing AWS Secrets Manager secrets from Parameter Store parameters](https://docs.aws.amazon.com/systems-manager/latest/userguide/integration-ps-secretsmanager.html). **Important** Image Builder excludes `SecureString` parameter resolution from its logs. However, you are also responsible for ensuring that sensitive information is not logged through commands issued in the component document. For example, if you use the `echo` command with a secure string, the command writes a plaintext value to the log. ### Required IAM permissions To use Systems Manager parameters in your components, your instance role must have the `ssm:GetParameter` permission for the parameter resource ARN. For example: ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "ssm:GetParameter", "Resource": "arn:aws:ssm:*:111122223333:parameter/ImageBuilder-*" } ] } ``` ------ To access encrypted values, you'll also need the following permissions: + Add `kms:Decrypt` for `SecureString` parameters or AWS Secrets Manager values that are encrypted with a customer managed AWS KMS key. + Add `secretsmanager:GetSecretValue` if you reference a Secrets Manager secret. ### Reference an SSM parameter in a component document The following example shows how to reference an Systems Manager Parameter Store parameter of Systems Manager parameters in a component: ``` name: UseSSMParameterVariable description: This is a sample component document that prints out the value of an SSM Parameter. Never do this for a SecureString parameter. schemaVersion: 1.0 phases: - name: verify steps: - name: EchoParameterValue action: ExecuteBash inputs: commands: - echo "Log SSM parameter name: /my/test/param, value {{ aws:ssm:/my/test/param }}." ``` ### Dynamic runtime variable resolution for SSM parameters AWSTOE provides the following built-in function that you can use within variable references to manipulate or transform values at runtime. #### resolve function The `resolve` function resolves a variable reference inside of another variable reference, allowing for dynamic variable name referencing. This is useful when working with SSM parameters where part of the parameter path may be variable and passed in as a document parameter. The `resolve` function only supports dynamic resolution of the name portion of an SSM parameter. ##### Syntax The `dynamic_variable` in the following example represents the name of an SSM parameter, and must be one of the following: + An SSM parameter reference (for example, `aws:ssm:/my/param`) + A component document parameter reference (for example, `parameter-name`) ``` {{ aws:ssm:resolve(dynamic_variable) }} ``` ##### Example: Resolve an SSM parameter at runtime The following example shows how to use the `resolve` function in a YAML component document: ``` name: SsmParameterTest description: This component verifies an SSM parameter variable reference with the echo command. schemaVersion: 1.0 parameters: - parameter-name: type: string description: "test" phases: - name: validate steps: - name: PrintDynamicVariable action: ExecuteBash inputs: commands: - echo "{{ aws:ssm:resolve(parameter-name) }}" ``` # Use conditional constructs in AWSTOE Conditional constructs perform different actions in your component document based on whether the specified conditional expression evaluates to `true` or `false`. You can use the `if` construct to control the flow of execution in your component document. ## if Construct You can use the `if` construct to evaluate whether a step should run or not. By default, when the `if` conditional expression evaluates to `true`, AWSTOE runs the step, and when the condition evaluates to `false`, AWSTOE skips the step. If a step is skipped, it's treated as a successful step when AWSTOE evaluates whether the phase and document ran successfully. **Note** An `if` statement is only evaluated one time, even if the step triggers a restart. If a step restarts, it recognizes that the `if` statement has already been evaluated, and continues where it left off. ### Syntax ``` if: - : [then: ] [else: ] ``` | Key name | Required | Description | | --- | --- | --- | | conditional expression | Yes | The conditional expression can contain exactly one of the following types of operators at the top level. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/imagebuilder/latest/userguide/toe-conditional-constructs.html) If your expression must satisfy multiple conditions, use a logical operator to specify your conditions. | | then | No | Defines the action to take if the conditional expression evaluates to `true`. | | else | No | Defines the action to take if the conditional expression evaluates to `false`. | | step action | Conditional | When you use `then` or `else`, you must specify one of the following step actions: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/imagebuilder/latest/userguide/toe-conditional-constructs.html) | **Example 1: Install package** The following example steps from an AWSTOE component document use logical operators to test a parameter value and run the appropriate package manager commands to install an application if the package is unzipped. ``` - name: InstallUnzipAptGet action: ExecuteBash if: and: - binaryExists: 'apt-get' - not: binaryExists: 'unzip' inputs: commands: - sudo apt-get update - sudo apt-get install -y unzip - name: InstallUnzipYum action: ExecuteBash if: and: - binaryExists: 'yum' - not: binaryExists: 'unzip' inputs: commands: - sudo yum install -y unzip - name: InstallUnzipZypper action: ExecuteBash if: and: - binaryExists: 'zypper' - not: binaryExists: 'unzip' inputs: commands: - sudo zypper refresh - sudo zypper install -y unzip ``` **Example 2: Skip a step** The following example shows two ways to skip a step. One uses a logical operator, and one uses a comparison operator with the `Skip` step action. ``` # Creates a file if it does not exist using not - name: CreateMyConfigFile-1 action: ExecuteBash if: not: fileExists: '/etc/my_config' inputs: commands: - echo "Hello world" > '/etc/my_config' # Creates a file if it does not exist using then and else - name: CreateMyConfigFile-2 action: ExecuteBash if: fileExists: '/etc/my_config' then: Skip else: Execute inputs: commands: - echo "Hello world" > '/etc/my_config' ``` # Use comparison operators in AWSTOE component documents You can use the following comparison operators with the **[Assert](toe-action-modules.md#action-modules-assertion)** action module and with conditional expressions that use the [if ConstructSyntax](toe-conditional-constructs.md#toe-conditional-if). A comparison operator can operate on a single value, for example `stringIsEmpty`, or it can compare a baseline value to a second value (variable value) to determine whether the conditional expression evaluates to `true` or `false`. If the comparison operates on two values, the second value can be a chaining variable. When comparing values of a different type, the following value conversions can occur prior to the comparison: + For numeric comparisons, if the variable value is a string, AWSTOE converts the string to a number prior to the evaluation. If the conversion is not possible, the comparison returns `false`. For example, if the variable value is `"1.0"`, the conversion works, but if the variable value is `"a10"` the conversion fails. + For string comparisons, if the variable value is a number, AWSTOE converts it to a string prior to the evaluation. ## Compare strings The following comparison operators work with strings to compare values, to test for spaces or an empty string, or to compare an input value to a regex pattern. String comparisons are not case sensitive, and they don't trim spaces from the beginning or the end of the string inputs. **String comparison operators** + [stringIsEmpty](#stringIsEmpty) + [stringIsWhitespace](#stringIsWhitespace) + [stringEquals](#stringEquals) + [stringLessThan](#stringLessThan) + [stringLessThanEquals](#stringLessThanEquals) + [stringGreaterThan](#stringGreaterThan) + [stringGreaterThanEquals](#stringGreaterThanEquals) + [patternMatches](#patternMatches) **stringIsEmpty** The `stringIsEmpty` operator returns `true` if the specified string doesn't contain any characters. For example: ``` # Evaluates to true stringIsEmpty: "" # Evaluates to false stringIsEmpty: " " # Evaluates to false stringIsEmpty: "Hello." ``` **stringIsWhitespace** Tests if the string specified for `stringIsWhitespace` contains only spaces. For example: ``` # Evaluates to true stringIsWhitespace: " " # Evaluates to false stringIsWhitespace: "" # Evaluates to false stringIsWhitespace: " Hello?" ``` **stringEquals** Tests if the string specified for `stringEquals` is an exact match for the string specified in the `value` parameter. For example: ``` # Evaluates to true stringEquals: 'Testing, testing...' value: 'Testing, testing...' # Evaluates to false stringEquals: 'Testing, testing...' value: 'Hello again.' # Evaluates to false stringEquals: 'Testing, testing...' value: 'TESTING, TESTING....' # Evaluates to false stringEquals: 'Testing, testing...' value: ' Testing, testing...' # Evaluates to false stringEquals: 'Testing, testing...' value: 'Testing, testing... ' ``` **stringLessThan** Tests if the string specified for `stringLessThan` is less than the string specified in the `value` parameter. For example: ``` # Evaluates to true # This comparison operator isn't case sensitive stringlessThan: 'A' value: 'a' # Evaluates to true - 'a' is less than 'b' stringlessThan: 'b' value: 'a' # Evaluates to true # Numeric strings compare as less than alphabetic strings stringlessThan: 'a' value: '0' # Evaluates to false stringlessThan: '0' value: 'a' ``` **stringLessThanEquals** Tests if the string specified for `stringLessThanEquals` is less than or equal to the string specified in the `value` parameter. For example: ``` # Evaluates to true - 'a' is equal to 'a' stringLessThanEquals: 'a' value: 'a' # Evaluates to true - since the comparison isn't case sensitive, 'a' is equal to 'A' stringLessThanEquals: 'A' value: 'a' # Evaluates to true - 'a' is less than 'b' stringLessThanEquals: 'b' value: 'a' # Evaluates to true - '0' is less than 'a' stringLessThanEquals: 'a' value: '0' # Evaluates to false - 'a' is greater than '0' stringLessThanEquals: '0' value: 'a' ``` **stringGreaterThan** Tests if the string specified for `stringGreaterThan` is greater than the string specified in the `value` parameter. For example: ``` # Evaluates to false - since the comparison isn't case sensitive, 'A' is equal to 'a' stringGreaterThan: 'a' value: 'A' # Evaluates to true - 'b' is greater than 'a' stringGreaterThan: 'a' value: 'b' # Evaluates to true - 'a' is greater than '0' stringGreaterThan: '0' value: 'a' # Evaluates to false - '0' is less than 'a' stringGreaterThan: 'a' value: '0' ``` **stringGreaterThanEquals** Tests if the string specified for `stringGreaterThanEquals` is greater than or equal to the string specified in the `value` parameter. For example: ``` # Evaluates to true - 'a' is equal to 'A' stringGreaterThanEquals: 'A' value: 'a' # Evaluates to true - 'b' is greater than 'a' stringGreaterThanEquals: 'a' value: 'b' # Evaluates to true - 'a' is greater than '0' stringGreaterThanEquals: '0' value: 'a' # Evaluates to false - '0' is less than 'a' stringGreaterThanEquals: 'a' value: '0' ``` **patternMatches** Tests if the string specified in the `value` parameter matches the regex pattern specified for `patternMatches`. The comparison uses to the [Golang regexp package](https://pkg.go.dev/regexp), which conforms to the RE2 syntax. For more information about RE2 rules, see the [google / re2](https://github.com/google/re2/wiki/Syntax) repository in *GitHub*. The following example shows a pattern match that returns `true`: ``` patternMatches: '^[a-z]+$' value: 'ThisIsValue' ``` ## Compare numbers The following comparison operators work with numbers. The values provided for these operators must be one of the following types, according to the YAML specification. Support for numeric comparisons uses the golang big package comparison operator, for example: [func (\$1Float) Cmp](https://pkg.go.dev/math/big#Float.Cmp). + Integer + Float (based on float64, which supports numbers from -1.7e\$1308 to \$11.7e\$1308) + A string that matches the following regex pattern: `^[-+]?([0-9]+[.])?[0-9]+$` **Number comparison operators** + [numberEquals](#numberEquals) + [numberLessThan](#numberLessThan) + [numberLessThanEquals](#numberLessThanEquals) + [numberGreaterThan](#numberGreaterThan) + [numberGreaterThanEquals](#numberGreaterThanEquals) **numberEquals** Tests if the number specified for `numberEquals` is equal to the number specified in the `value` parameter. All of the following example comparisons return `true`: ``` # Values provided as a positive number numberEquals: 1 value: 1 # Comparison value provided as a string numberEquals: '1' value: 1 # Value provided as a string numberEquals: 1 value: '1' # Values provided as floats numberEquals: 5.0 value: 5.0 # Values provided as a negative number numberEquals: -1 value: -1 ``` **numberLessThan** Tests if the number specified for `numberLessThan` is less than the number specified in the `value` parameter. For example: ``` # Evaluates to true numberLessThan: 2 value: 1 # Evaluates to true numberLessThan: 2 value: 1.9 # Evaluates to false numberLessThan: 2 value: '2' ``` **numberLessThanEquals** Tests if the number specified for `numberLessThanEquals` is less than or equal to the number specified in the `value` parameter. For example: ``` # Evaluates to true numberLessThanEquals: 2 value: 1 # Evaluates to true numberLessThanEquals: 2 value: 1.9 # Evaluates to true numberLessThanEquals: 2 value: '2' # Evaluates to false numberLessThanEquals: 2 value: 2.1 ``` **numberGreaterThan** Tests if the number specified for `numberGreaterThan` is greater than the number specified in the `value` parameter. For example: ``` # Evaluates to true numberGreaterThan: 1 value: 2 # Evaluates to true numberGreaterThan: 1 value: 1.1 # Evaluates to false numberGreaterThan: 1 value: '1' ``` **numberGreaterThanEquals** Tests if the number specified for `numberGreaterThanEquals` is greater than or equal to the number specified in the `value` parameter. For example: ``` # Evaluates to true numberGreaterThanEquals: 1 value: 2 # Evaluates to true numberGreaterThanEquals: 1 value: 1.1 # Evaluates to true numberGreaterThanEquals: 1 value: '1' # Evaluates to false numberGreaterThanEquals: 1 value: 0.8 ``` ## Check files The following comparison operators check the file hash or check if a file or folder exists. **File and folder operators** + [binaryExists](#binaryExists) + [fileExists](#fileExists) + [folderExists](#folderExists) + [fileMD5Equals](#fileMD5Equals) + [fileSHA1Equals](#fileSHA1Equals) + [fileSHA256Equals](#fileSHA256Equals) + [fileSHA512Equals](#fileSHA512Equals) **binaryExists** Tests whether an application is available in the current path. For example: ``` binaryExists: 'foo' ``` On Linux and macOS systems, for an application named *foo*, this works the same as the following bash command: **type *foo* >/dev/null 2>&1**, where **\$1? == 0** indicates a successful comparison. On Windows systems, for an application named *foo*, this works the same as the PowerShell command **& C:\$1Windows\$1System32\$1where.exe /Q *foo*** where **\$1LASTEXITCODE = 0** indicates a successful comparison. **fileExists** Tests whether a file exists at the specified path. You can provide an absolute or relative path. If the location you specify exists and is a file, the comparison evaluates to `true`. For example: ``` fileExists: '/path/to/file' ``` On Linux and macOS systems, this works the same as the following bash command: **-d */path/to/file***, where **\$1? == 0** indicates a successful comparison. On Windows systems, this works the same as the PowerShell command **Test-Path -Path '*C:\$1path\$1to\$1file*' -PathType 'Leaf'**. **folderExists** Tests whether a folder exists at the specified path. You can provide an absolute or relative path. If the location you specify exists and is a folder, the comparison evaluates to `true`. For example: ``` folderExists: '/path/to/folder' ``` On Linux and macOS systems, this works the same as the following bash command: **-d */path/to/folder***, where **\$1? == 0** indicates a successful comparison. On Windows systems, this works the same as the PowerShell command **Test-Path -Path '*C:\$1path\$1to\$1folder*' -PathType 'Container'**. **fileMD5Equals** Tests whether a file’s MD5 hash equals a specified value. For example: ``` fileMD5Equals: '' path: '/path/to/file' ``` **fileSHA1Equals** Tests whether a file’s SHA1 hash equals a specified value. For example: ``` fileSHA1Equals: '' path: '/path/to/file' ``` **fileSHA256Equals** Tests whether a file’s SHA256 hash equals a specified value. For example: ``` fileSHA256Equals: '' path: '/path/to/file' ``` **fileSHA512Equals** Tests whether a file’s SHA512 hash equals a specified value. For example: ``` fileSHA512Equals: '' path: '/path/to/file' ``` # Use logical operators in AWSTOE component documents You can use the following logical operators to add or modify conditional expressions in your component document. AWSTOE evaluates conditional expressions in the order that the conditions are specified. For more information about comparison operators for component documents, see [Use comparison operators in AWSTOE component documents](toe-comparison-operators.md). **and** With the `and` operator, you can evaluate two or more comparisons as a single expression. The expression evaluates to `true` when all of the conditions in the list are true. Otherwise, the expression evaluates to `false`. **Examples:** The following example performs two comparisons – a string and a number. Both comparisons are true, so the expression evaluates to true. ``` and: - stringEquals: 'test_string' value: 'test_string' - numberEquals: 1 value: 1 ``` The following example also performs two comparisons. The first comparison is false, at which point evaluation stops and the second comparison is skipped. The expression evaluates to `false`. ``` and: - stringEquals: 'test_string' value: 'Hello world!' - numberEquals: 1 value: 1 ``` **or** With the `or` operator, you can evaluate two or more comparisons as a single expression. The expression evaluates to `true` when one of the specified comparisons is true. If none of the specified comparisons evaluate to `true`, the expression evaluates to `false`. **Examples:** The following example performs two comparisons – a string and a number. The first comparison is true, so the expression evaluates to `true` and the second comparison is skipped. ``` or: - stringEquals: 'test_string' value: 'test_string' - numberEquals: 1 value: 3 ``` The following example also performs two comparisons. The first comparison is false, and evaluation continues. The second comparison is true, so the expression evaluates to `true`. ``` or: - stringEquals: 'test_string' value: 'Hello world!' - numberEquals: 1 value: 1 ``` In the final example, both comparisons are false, so the expression evaluates to `false`. ``` or: - stringEquals: 'test_string' value: 'Hello world!' - numberEquals: 1 value: 3 ``` **not** With the `not` operator, you can negate a single comparison. The expression evaluates to `true` if the comparison is false. If the comparison is true, then the expression evaluates to `false`. **Examples:** The following example performs a string comparison. The comparison is false, so the expression evaluates to `true`. ``` not: - stringEquals: 'test_string' value: 'Hello world!' ``` The following example also performs a string comparison. The comparison is true, so the expression evaluates to `false`. ``` not: - stringEquals: 'test_string' value: 'test_string' ``` # Use looping constructs in AWSTOE This section provides information to help you create looping constructs in the AWSTOE. Looping constructs define a repeated sequence of instructions. You can use the following types of looping constructs in AWSTOE: + `for` constructs – Iterate over a bounded sequence of integers. + `forEach` constructs + `forEach` loop with input list – Iterates over a finite collection of strings. + `forEach` loop with delimited list – Iterates over a finite collection of strings joined by a delimiter. **Note** Looping constructs support only string data types. **Topics** + [Reference iteration variables](#toe-loop-iteration-variables) + [Types of looping constructs](#toe-loop-types) + [Step fields](#toe-loop-step-fields) + [Step and iteration outputs](#toe-loop-step-output) ## Reference iteration variables To refer to the index and value of the current iteration variable, the reference expression `{{ loop.* }}` must be used within the input body of a step that contains a looping construct. This expression cannot be used to refer to the iteration variables of the looping construct of another step. The reference expression consists of the following members: + `{{ loop.index }}` – The ordinal position of the current iteration, which is indexed at `0`. + `{{ loop.value }}` – The value associated with the current iteration variable. ### Loop names All looping constructs have an optional name field for identification. If a loop name is provided, it can be used to refer to iteration variables in the input body of the step. To refer to the iteration indices and values of a named loop, use `{{ .* }}` with `{{ loop.* }}` in the input body of the step. This expression cannot be used to refer to the named looping construct of another step. The reference expression consists of the following members: + `{{ .index }}` – The ordinal position of the current iteration of the named loop, which is indexed at `0`. + `{{ .value }}` – The value associated with the current iteration variable of the named loop. ### Resolve reference expressions The AWSTOE resolves reference expressions as follows: + `{{ .* }}` – AWSTOE resolves this expression using the following logic: + If the loop of the currently running step matches the `` value, then the reference expression resolves to the looping construct of the currently running step. + `` resolves to the named looping construct if it appears within the currently running step. + `{{ loop.* }}` – AWSTOE resolves the expression using the looping construct defined in the currently running step. If reference expressions are used within a step that does not contain a loop, then AWSTOE does not resolve the expressions and they appear in the step with no replacement. **Note** Reference expressions must be enclosed in double quotes to be correctly interpreted by the YAML compiler. ## Types of looping constructs This section provides information and examples about looping construct types that can be used in the AWSTOE. **Topics** + [`for` loop](#toe-loop-types-for) + [`forEach` loop with input list](#toe-loop-types-foreach) + [`forEach` loop with delimited list](#toe-loop-types-foreach-delimited) ### `for` loop The `for` loop iterates on a range of integers specified within a boundary outlined by the start and end of the variables. The iterating values are in the set `[start, end]` and includes boundary values. AWSTOE verifies the `start`, `end`, and `updateBy` values to ensure that the combination does not result in an infinite loop. `for` loop schema ``` - name: "StepName" action: "ActionModule" loop: name: "string" for: start: int end: int updateBy: int inputs: ... ``` **`for` loop input** | Field | Description | Type | Required | Default | | --- | --- | --- | --- | --- | | `name` | Unique name of the loop. It must be unique compared to other loop names in the same phase. | String | No | "" | | `start` | Starting value of iteration. Does not accept chaining expressions. | Integer | Yes | n/a | | `end` | Ending value of iteration. Does not accept chaining expressions. | Integer | Yes | n/a | | `updateBy` | Difference by which an iterating value is updated through addition. It must be a negative or positive non-zero value. Does not accept chaining expressions. | Integer | Yes | n/a | `for` loop input example ``` - name: "CalculateFileUploadLatencies" action: "ExecutePowerShell" loop: for: start: 100000 end: 1000000 updateBy: 100000 inputs: commands: - | $f = new-object System.IO.FileStream c:\temp\test{{ loop.index }}.txt, Create, ReadWrite $f.SetLength({{ loop.value }}MB) $f.Close() - c:\users\administrator\downloads\latencyTest.exe --file c:\temp\test{{ loop.index }}.txt - AWS s3 cp c:\users\administrator\downloads\latencyMetrics.json s3://bucket/latencyMetrics.json - | Remove-Item -Path c:\temp\test{{ loop.index }}.txt Remove-Item -Path c:\users\administrator\downloads\latencyMetrics.json ``` ### `forEach` loop with input list The `forEach` loop iterates on an explicit list of values, which can be strings and chained expressions. `forEach` loop with input list schema ``` - name: "StepName" action: "ActionModule" loop: name: "string" forEach: - "string" inputs: ... ``` **`forEach` loop with input list input** | Field | Description | Type | Required | Default | | --- | --- | --- | --- | --- | | `name` | Unique name of the loop. It must be unique compared to other loop names in the same phase. | String | No | "" | | List of strings of `forEach` loop | List of strings for iteration. Accepts chained expressions as strings in the list. Chained expressions must be enclosed by double quotes for the YAML compiler to correctly interpret them. | List of strings | Yes | n/a | `forEach` loop with input list example 1 ``` - name: "ExecuteCustomScripts" action: "ExecuteBash" loop: name: BatchExecLoop forEach: - /tmp/script1.sh - /tmp/script2.sh - /tmp/script3.sh inputs: commands: - echo "Count {{ BatchExecLoop.index }}" - sh "{{ loop.value }}" - | retVal=$? if [ $retVal -ne 0 ]; then echo "Failed" else echo "Passed" fi ``` `forEach` loop with input list example 2 ``` - name: "RunMSIWithDifferentArgs" action: "ExecuteBinary" loop: name: MultiArgLoop forEach: - "ARG1=C:\Users ARG2=1" - "ARG1=C:\Users" - "ARG1=C:\Users ARG3=C:\Users\Administrator\Documents\f1.txt" inputs: commands: path: "c:\users\administrator\downloads\runner.exe" args: - "{{ MultiArgLoop.value }}" ``` `forEach` loop with input list example 3 ``` - name: "DownloadAllBinaries" action: "S3Download" loop: name: MultiArgLoop forEach: - "bin1.exe" - "bin10.exe" - "bin5.exe" inputs: - source: "s3://bucket/{{ loop.value }}" destination: "c:\temp\{{ loop.value }}" ``` ### `forEach` loop with delimited list The loop iterates over a string containing values separated by a delimiter. To iterate over the string’s constituents, AWSTOE uses the delimiter to split the string into an array suitable for iteration. `forEach` loop with delimited list schema ``` - name: "StepName" action: "ActionModule" loop: name: "string" forEach: list: "string" delimiter: ".,;:\n\t -_" inputs: ... ``` **`forEach` loop with delimited list input** | Field | Description | Type | Required | Default | | --- | --- | --- | --- | --- | | `name` | Unique name given to the loop. It should be unique when compared to other loop names in the same phase. | String | No | "" | | `list` | A string that is composed of constituent strings joined by a common delimiter character. Also accepts chained expressions. In case of chained expressions, ensure that those are enclosed by double quotes for correct interpretation by the YAML compiler. | String | Yes | n/a | | `delimiter` | Character used to separate out strings within a block. Default is the comma character. Only one delimiter character is allowed from the given list: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/imagebuilder/latest/userguide/toe-looping-constructs.html) Chaining expressions cannot be used. | String | No | Comma: "," | **Note** The value of `list` is treated as an immutable string. If the source of `list` is changed during runtime, it will not be reflected during the run. `forEach` loop with delimited list example 1 This example uses the following chaining expression pattern to refer to another step's output: `..[inputs | outputs].`. ``` - name: "RunMSIs" action: "ExecuteBinary" loop: forEach: list: "{{ build.GetAllMSIPathsForInstallation.outputs.stdout }}" delimiter: "\n" inputs: commands: path: "{{ loop.value }}" ``` `forEach` loop with delimited list example 2 ``` - name: "UploadMetricFiles" action: "S3Upload" loop: forEach: list: "/tmp/m1.txt,/tmp/m2.txt,/tmp/m3.txt,..." inputs: commands: - source: "{{ loop.value }}" destination: "s3://bucket/key/{{ loop.value }}" ``` ## Step fields Loops are part of a step. Any field related to the running of a step is not applied to individual iterations. Step fields apply only at the step level, as follows: + *timeoutSeconds* – All iterations of the loop must be run within the time period specified by this field. If the loop run times out, then AWSTOE runs the retry policy of the step and resets the timeout parameter for each new attempt. If the loop run exceeds the timeout value after reaching the maximum number of retries, the failure message of the step states that the loop run had timed out. + *onFailure* – Failure handling is applied to the step as follows: + If *onFailure* is set to `Abort`, AWSTOE exits the loop and retries the step according to the retry policy. After the maximum number of retry attempts, AWSTOE marks the current step as failed, and stops running the process. AWSTOE sets the status code for the parent phase and document to `Failed`. **Note** No further steps run after the failed step. + If *onFailure* is set to `Continue`, AWSTOE exits the loop and retries the step according to the retry policy. After the maximum number of retry attempts, AWSTOE marks the current step as failed, and continues on to run the next step. AWSTOE sets the status code for the parent phase and document to `Failed`. + If *onFailure* is set to `Ignore`, AWSTOE exits the loop and retries the step according to the retry policy. After the maximum number of retry attempts, AWSTOE marks the current step as `IgnoredFailure`, and continues on to run the next step. AWSTOE sets the status code for the parent phase and document to `SuccessWithIgnoredFailure`. **Note** This is still considered a successful run, but includes information to let you know that one or more steps failed and were ignored. + *maxAttempts * – For every retry, the entire step and all iterations are run from the beginning. + *status* – The overall status of the running of a step.`status` does not represent the status of individual iterations. The status of a step with loops is determined as follows: + If a single iteration fails to run, the status of a step points to a failure. + If all iterations succeed, the status of a step points to a success. + *startTime * – The overall start time of the running of a step. Does not represent the start time of individual iterations. + *endTime * – The overall end time of the running of a step. Does not represent the end time of individual iterations. + *failureMessage * – Includes the iteration indices that failed in case of non-timeout errors. In case of timeout errors, the message states that the loop run has failed. Individual error messages for each iteration are not provided to minimize the size of failure messages. ## Step and iteration outputs Every iteration contains an output. At the end of a loop run, AWSTOE consolidates all successful iteration outputs in `detailedOutput.json`. The consolidated outputs are a collation of values that belong to the corresponding output keys as defined in the output schema of the action module. The following example shows how the outputs are consolidated: **Output of `ExecuteBash` for Iteration 1** ``` { "stdout":"Hello" } ``` **Output of `ExecuteBash` for Iteration 2** ``` { "stdout":"World" } ``` **Output of `ExecuteBash` for Step** ``` { "stdout":"Hello\nWorld" } ``` For example, `ExecuteBash`, `ExecutePowerShell`, and `ExecuteBinary` are action modules which return `STDOUT` as the action module output. `STDOUT` messages are joined with the new line character to produce the overall output of the step in `detailedOutput.json`. AWSTOE will not consolidate the outputs of unsuccessful iterations. # Action modules supported by AWSTOE component manager Image building services, such as EC2 Image Builder, use AWSTOE action modules to help configure the EC2 instances that are used for building and testing customized machine images. This section describes the features of commonly used AWSTOE action modules, and how to configure them, including examples. Components are authored with plaintext YAML documents. For more information about document syntax, see [Use the AWSTOE component document framework for custom components](toe-use-documents.md). **Note** All action modules use the same account as the Systems Manager agent when they run, which is `root` on Linux, and `NT Authority\SYSTEM` on Windows. The following cross-reference categorizes action modules by the type of actions that they perform.   **General execution** + [Assert (Linux, Windows, macOS)](#action-modules-assertion) + [ExecuteBash (Linux, macOS)](#action-modules-executebash) + [ExecuteBinary (Linux, Windows, macOS)](#action-modules-executebinary) + [ExecuteDocument (Linux, Windows, macOS)](#action-modules-executedocument) + [ExecutePowerShell (Windows)](#action-modules-executepowershell)   **File download and upload** + [S3Download (Linux, Windows, macOS)](#action-modules-s3download) + [S3Upload (Linux, Windows, macOS)](#action-modules-s3upload) + [WebDownload (Linux, Windows, macOS)](#action-modules-webdownload)   **File system operations** + [AppendFile (Linux, Windows, macOS)](#action-modules-appendfile) + [CopyFile (Linux, Windows, macOS)](#action-modules-copyfile) + [CopyFolder (Linux, Windows, macOS)](#action-modules-copyfolder) + [CreateFile (Linux, Windows, macOS)](#action-modules-createfile) + [CreateFolder (Linux, Windows, macOS)](#action-modules-createfolder) + [CreateSymlink (Linux, Windows, macOS)](#action-modules-createsymlink) + [DeleteFile (Linux, Windows, macOS)](#action-modules-deletefile) + [DeleteFolder (Linux, Windows, macOS)](#action-modules-deletefolder) + [ListFiles (Linux, Windows, macOS)](#action-modules-listfiles) + [MoveFile (Linux, Windows, macOS)](#action-modules-movefile) + [MoveFolder (Linux, Windows, macOS)](#action-modules-movefolder) + [ReadFile (Linux, Windows, macOS)](#action-modules-readfile) + [SetFileEncoding (Linux, Windows, macOS)](#action-modules-setfileencoding) + [SetFileOwner (Linux, Windows, macOS)](#action-modules-setfileowner) + [SetFolderOwner (Linux, Windows, macOS)](#action-modules-setfolderowner) + [SetFilePermissions (Linux, Windows, macOS)](#action-modules-setfilepermissions) + [SetFolderPermissions (Linux, Windows, macOS)](#action-modules-setfolderpermissions)   **Software installation actions** + [InstallMSI (Windows)](#action-modules-install-msi) + [UninstallMSI (Windows)](#action-modules-uninstall-msi)   **System actions** + [Reboot (Linux, Windows)](#action-modules-reboot) + [SetRegistry (Windows)](#action-modules-setregistry) + [UpdateOS (Linux, Windows)](#action-modules-updateos) ## General execution modules The following section contains details for action modules that run commands and control execution workflow. **Topics** + [Assert (Linux, Windows, macOS)](#action-modules-assertion) + [ExecuteBash (Linux, macOS)](#action-modules-executebash) + [ExecuteBinary (Linux, Windows, macOS)](#action-modules-executebinary) + [ExecuteDocument (Linux, Windows, macOS)](#action-modules-executedocument) + [ExecutePowerShell (Windows)](#action-modules-executepowershell) ### Assert (Linux, Windows, macOS) The **Assert** action module performs value comparisons using [Comparison operators](toe-comparison-operators.md) or [Logical operators](toe-logical-operators.md) as input. The result of the operator expression (true or false) indicates the overall success or failure status for the step. If the comparison or logical operator expression evaluates to `true`, the step is marked as `Success`. Otherwise, the step is marked as `Failed`. If the step fails, the `onFailure` parameter decides the outcome of the step. **Input** | Key name | Description | Type | Required | | --- | --- | --- | --- | | input | Contains a single comparison or logical operator. Note, logical operators can contain more than one comparison operator. | This is variable, depending on the operator | Yes | **Input example: A simple comparison using the `stringEquals` comparison operator** This example evaluates to `true`. ``` - name: StringComparison action: Assert inputs: stringEquals: '2.1.1' value: '{{ validate.ApplicationVersion.outputs.stdout }}' ``` **Input example: Regex comparisons using the `patternMatches` comparison operator** These examples all evaluate to `true`. ``` - name: Letters only action: Assert inputs: patternMatches: '^[a-zA-Z]+$' value: 'ThisIsOnlyLetters' - name: Letters and spaces only action: Assert inputs: patternMatches: '^[a-zA-Z\s]+$' value: 'This text contains spaces' - name: Numbers only action: Assert inputs: patternMatches: '^[0-9]+$' value: '1234567890' ``` **Input example: Nested comparisons with logical operators and chained variables** The following example demonstrates nested comparisons with logical operators that use comparisons with chained variables. The `Assert` evaluates to `true` if either of the following are true: + The `ApplicationVersion` is greater than `2.0` and the `CPUArchitecture` equals `arm64`. + The `CPUArchitecture` equals `x86_64`. ``` - name: NestedComparisons action: Assert inputs: or: # <- first level deep - and: # <- second level deep - numberGreaterThan: 2.0 # <- third level deep value: '{{ validate.ApplicationVersion.outputs.stdout }}' - stringEquals: 'arm64' value: '{{ validate.CPUArchitecture.outputs.stdout }}' - stringEquals: 'x86_64' value: '{{ validate.CPUArchitecture.outputs.stdout }}' ``` **Output:** The output of an `Assert` is success or failure of the step. ### ExecuteBash (Linux, macOS) The **ExecuteBash** action module allows you to run bash scripts with inline shell code/commands. This module supports Linux. All of the commands and instructions that you specify in the commands block are converted into a file (for example, `input.sh`) and run with the bash shell. The result of running the shell file is the exit code of the step. The **ExecuteBash** module handles system restarts if the script exits with an exit code of `194`. When initiated, the application performs one of the following actions: + The application hands the exit code to the caller if it is run by the Systems Manager Agent. The Systems Manager Agent handles the system reboot and runs the same step that initiated the restart, as described in [Rebooting Managed Instance from Scripts](https://docs.aws.amazon.com/systems-manager/latest/userguide/send-commands-reboot.html). + The application saves the current `executionstate`, configures a restart trigger to rerun the application, and restarts the system. After system restart, the application runs the same step that initiated the restart. If you require this functionality, you must write idempotent scripts that can handle multiple invocations of the same shell command. **Input** | Key name | Description | Type | Required | | --- | --- | --- | --- | | commands | Contains a list of instructions or commands to run as per bash syntax. Multi-line YAML is allowed. | List | Yes | **Input example: Before and after a reboot** ``` name: ExitCode194Example description: This shows how the exit code can be used to restart a system with ExecuteBash schemaVersion: 1.0 phases: - name: build steps: - name: RestartTrigger action: ExecuteBash inputs: commands: - | REBOOT_INDICATOR=/var/tmp/reboot-indicator if [ -f "${REBOOT_INDICATOR}" ]; then echo 'The reboot file exists. Deleting it and exiting with success.' rm "${REBOOT_INDICATOR}" exit 0 fi echo 'The reboot file does not exist. Creating it and triggering a restart.' touch "${REBOOT_INDICATOR}" exit 194 ``` **Output** | Field | Description | Type | | --- | --- | --- | | stdout | Standard output of command execution. | string | If you start a reboot and return exit code `194` as part of the action module, the build will resume at the same action module step that initiated the reboot. If you start a reboot without the exit code, the build process may fail. **Output example: Before reboot (first time through document)** ``` { “stdout”: “The reboot file does not exist. Creating it and triggering a restart." } ``` **Output example: After reboot, (second time through document)** ``` { “stdout”: “The reboot file exists. Deleting it and exiting with success." } ``` ### ExecuteBinary (Linux, Windows, macOS) The **ExecuteBinary** action module allows you to run binary files with a list of command-line arguments. The **ExecuteBinary** module handles system restarts if the binary file exits with an exit code of `194` (Linux) or `3010` (Windows). When this happens, the application performs one of the following actions: + The application hands the exit code to the caller if it is run by the Systems Manager Agent. The Systems Manager Agent handles restarting the system and runs the same step that initiated the restart, as described in [Rebooting Managed Instance from Scripts](https://docs.aws.amazon.com/systems-manager/latest/userguide/send-commands-reboot.html). + The application saves the current `executionstate`, configures a restart trigger to rerun the application, and restarts the system. After the system restarts, the application runs the same step that initiated the restart. If you require this functionality, you must write idempotent scripts that can handle multiple invocations of the same shell command. **Input** | Key name | Description | Type | Required | | --- | --- | --- | --- | | path | The path to the binary file for execution. | String | Yes | | arguments | Contains a list of command-line arguments to use when running the binary. | String List | No | **Input example: install .NET** ``` - name: "InstallDotnet" action: ExecuteBinary inputs: path: C:\PathTo\dotnet_installer.exe arguments: - /qb - /norestart ``` **Output** | Field | Description | Type | | --- | --- | --- | | stdout | Standard output of command execution. | string | **Output example** ``` { "stdout": "success" } ``` ### ExecuteDocument (Linux, Windows, macOS) The **ExecuteDocument** action module adds support for nested component documents, running multiple component documents from one document. AWSTOE validates the document that is passed in the input parameter at run time. **Restrictions** + This action module runs one time, with no retries allowed, and no option to set timeout limits. **ExecuteDocument** sets the following default values, and returns an error if you try to change them. + `timeoutSeconds`: -1 + `maxAttempts`: 1 **Note** You can leave these values blank, and AWSTOE uses the default values. + Document nesting is allowed, up to three levels deep, but no more than that. Three levels of nesting translates to four document levels, as the top level isn't nested. In this scenario, the lowest level document must not call any other documents. + Cyclic execution of component documents is not allowed. Any document that calls itself outside of a looping construct, or that calls another document higher up in the current chain of execution, initiates a cycle that can result in an endless loop. When AWSTOE detects a cyclic execution, it stops the execution and records the failure. ![\[Nesting level restrictions for the ExecuteDocument action module.\]](http://docs.aws.amazon.com/imagebuilder/latest/userguide/images/toe-component-document-nesting.png) If a component document tries to run itself, or to run any of the component documents that are higher up in the current chain of execution, the execution fails. **Input** | Key name | Description | Type | Required | | --- | --- | --- | --- | | document | Path of component document. Valid options include: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/imagebuilder/latest/userguide/toe-action-modules.html) | String | Yes | | document-s3-bucket-owner | The account ID of the S3 bucket owner for the S3 bucket where component documents are stored. *(Recommended if you are using S3 URIs in your component document.)* | String | No | | phases | Phases to run in the component document, expressed as a comma-separated list. If no phases are specified, then all phases run. | String | No | | parameters | Input parameters that are passed in to the component document at runtime as key value pairs. | Parameter Map List | No | **Parameter map input** | Key name | Description | Type | Required | | --- | --- | --- | --- | | name | The name of the input parameter to pass to the component document that the **ExecuteDocument** action module is running. | String | Yes | | value | The value of the input parameter. | String | Yes | **Input examples** The following examples show variations of the inputs for your component document, depending on your installation path. **Input example: Local document path** ``` # main.yaml schemaVersion: 1.0 phases: - name: build steps: - name: ExecuteNestedDocument action: ExecuteDocument inputs: document: Sample-1.yaml phases: build parameters: - name: parameter-1 value: value-1 - name: parameter-2 value: value-2 ``` **Input example: S3 URI as a document path** ``` # main.yaml schemaVersion: 1.0 phases: - name: build steps: - name: ExecuteNestedDocument action: ExecuteDocument inputs: document: s3://my-bucket/Sample-1.yaml document-s3-bucket-owner: 123456789012 phases: build,validate parameters: - name: parameter-1 value: value-1 - name: parameter-2 value: value-2 ``` **Input example: EC2 Image Builder component ARN as a document path** ``` # main.yaml schemaVersion: 1.0 phases: - name: build steps: - name: ExecuteNestedDocument action: ExecuteDocument inputs: document: arn:aws:imagebuilder:us-west-2:aws:component/Sample-Test/1.0.0 phases: test parameters: - name: parameter-1 value: value-1 - name: parameter-2 value: value-2 ``` **Using a ForEach loop to run documents** ``` # main.yaml schemaVersion: 1.0 phases: - name: build steps: - name: ExecuteNestedDocument action: ExecuteDocument loop: name: 'myForEachLoop' forEach: - Sample-1.yaml - Sample-2.yaml inputs: document: "{{myForEachLoop.value}}" phases: test parameters: - name: parameter-1 value: value-1 - name: parameter-2 value: value-2 ``` **Using a For loop to run documents** ``` # main.yaml schemaVersion: 1.0 phases: - name: build steps: - name: ExecuteNestedDocument action: ExecuteDocument loop: name: 'myForLoop' for: start: 1 end: 2 updateBy: 1 inputs: document: "Sample-{{myForLoop.value}}.yaml" phases: test parameters: - name: parameter-1 value: value-1 - name: parameter-2 value: value-2 ``` **Output** AWSTOE creates an output file called `detailedoutput.json` every time it runs. The file contains details about every phase and step of every component document that is invoked while it's running. For the **ExecuteDocument** action module, you can find a brief runtime summary in the `outputs` field, and details about the phases, steps, and documents that it runs in the `detailedOutput`. ``` { \"executedStepCount\":1,\"executionId\":\"97054e22-06cc-11ec-9b14-acde48001122\",\"failedStepCount\":0,\"failureMessage\":\"\",\"ignoredFailedStepCount\":0,\"logUrl\":\"\",\"status\":\"success\" }", ``` Each component document's output summary object contains the following details, as shown here, with sample values: + executedStepCount":1 + "executionId":"12345a67-89bc-01de-2f34-abcd56789012" + "failedStepCount":0 + "failureMessage":"" + "ignoredFailedStepCount":0 + "logUrl":"" + "status":"success" **Output example** The following example shows output from the **ExecuteDocument** action module when a nested execution occurs. In this example, the `main.yaml` component document successfully runs the `Sample-1.yaml` component document. ``` { "executionId": "12345a67-89bc-01de-2f34-abcd56789012", "status": "success", "startTime": "2021-08-26T17:20:31-07:00", "endTime": "2021-08-26T17:20:31-07:00", "failureMessage": "", "documents": [ { "name": "", "filePath": "main.yaml", "status": "success", "description": "", "startTime": "2021-08-26T17:20:31-07:00", "endTime": "2021-08-26T17:20:31-07:00", "failureMessage": "", "phases": [ { "name": "build", "status": "success", "startTime": "2021-08-26T17:20:31-07:00", "endTime": "2021-08-26T17:20:31-07:00", "failureMessage": "", "steps": [ { "name": "ExecuteNestedDocument", "status": "success", "failureMessage": "", "timeoutSeconds": -1, "onFailure": "Abort", "maxAttempts": 1, "action": "ExecuteDocument", "startTime": "2021-08-26T17:20:31-07:00", "endTime": "2021-08-26T17:20:31-07:00", "inputs": "[{\"document\":\"Sample-1.yaml\",\"document-s3-bucket-owner\":\"\",\"phases\":\"\",\"parameters\":null}]", "outputs": "[{\"executedStepCount\":1,\"executionId\":\"98765f43-21ed-09cb-8a76-fedc54321098\",\"failedStepCount\":0,\"failureMessage\":\"\",\"ignoredFailedStepCount\":0,\"logUrl\":\"\",\"status\":\"success\"}]", "loop": null, "detailedOutput": [ { "executionId": "98765f43-21ed-09cb-8a76-fedc54321098", "status": "success", "startTime": "2021-08-26T17:20:31-07:00", "endTime": "2021-08-26T17:20:31-07:00", "failureMessage": "", "documents": [ { "name": "", "filePath": "Sample-1.yaml", "status": "success", "description": "", "startTime": "2021-08-26T17:20:31-07:00", "endTime": "2021-08-26T17:20:31-07:00", "failureMessage": "", "phases": [ { "name": "build", "status": "success", "startTime": "2021-08-26T17:20:31-07:00", "endTime": "2021-08-26T17:20:31-07:00", "failureMessage": "", "steps": [ { "name": "ExecuteBashStep", "status": "success", "failureMessage": "", "timeoutSeconds": 7200, "onFailure": "Abort", "maxAttempts": 1, "action": "ExecuteBash", "startTime": "2021-08-26T17:20:31-07:00", "endTime": "2021-08-26T17:20:31-07:00", "inputs": "[{\"commands\":[\"echo \\\"Hello World!\\\"\"]}]", "outputs": "[{\"stdout\":\"Hello World!\"}]", "loop": null, "detailedOutput": null }] }] }] }] }] }] }] } ``` ### ExecutePowerShell (Windows) The **ExecutePowerShell** action module allows you to run PowerShell scripts with inline shell code/commands. This module supports the Windows platform and Windows PowerShell. All of the commands/instructions specified in the commands block are converted into a script file (for example, `input.ps1`) and run using Windows PowerShell. The result of running the shell file is the exit code. The **ExecutePowerShell** module handles system restarts if the shell command exits with an exit code of `3010`. When initiated, the application performs one of the following actions: + Hands the exit code to the caller if run by the Systems Manager Agent. The Systems Manager Agent handles the system reboot and runs the same step that initiated the restart, as described in [Rebooting Managed Instance from Scripts](https://docs.aws.amazon.com/systems-manager/latest/userguide/send-commands-reboot.html). + Saves the current `executionstate`, configures a restart trigger to rerun the application, and reboots the system. After system restart, the application runs the same step that initiated the restart. If you require this functionality, you must write idempotent scripts that can handle multiple invocations of the same shell command. **Input** | Key name | Description | Type | Required | | --- | --- | --- | --- | | commands | Contains a list of instructions or commands to run as per PowerShell syntax. Multi-line YAML is allowed. | String List | Yes. Must specify `commands` or `file`, not both. | | file | Contains the path to a PowerShell script file. PowerShell will run against this file using the -file command line argument. The path must point to a .ps1 file. | String | Yes. Must specify `commands` or `file`, not both. | **Input example: Before and after a reboot** ``` name: ExitCode3010Example description: This shows how the exit code can be used to restart a system with ExecutePowerShell schemaVersion: 1.0 phases: - name: build steps: - name: RestartTrigger action: ExecutePowerShell inputs: commands: - | $rebootIndicator = Join-Path -Path $env:SystemDrive -ChildPath 'reboot-indicator' if (Test-Path -Path $rebootIndicator) { Write-Host 'The reboot file exists. Deleting it and exiting with success.' Remove-Item -Path $rebootIndicator -Force | Out-Null [System.Environment]::Exit(0) } Write-Host 'The reboot file does not exist. Creating it and triggering a restart.' New-Item -Path $rebootIndicator -ItemType File | Out-Null [System.Environment]::Exit(3010) ``` **Output** | Field | Description | Type | | --- | --- | --- | | stdout | Standard output of command execution. | string | If you run a reboot and return exit code `3010` as part of the action module, the build will resume at the same action module step that initiated the reboot. If you run a reboot without the exit code, the build process may fail. **Output example: Before reboot (first time through document)** ``` { “stdout”: “The reboot file does not exist. Creating it and triggering a restart." } ``` **Output example: After reboot, (second time through document)** ``` { “stdout”: “The reboot file exists. Deleting it and exiting with success." } ``` ## File download and upload modules The following section contains details for action modules that upload or download files. **Topics** + [S3Download (Linux, Windows, macOS)](#action-modules-s3download) + [S3Upload (Linux, Windows, macOS)](#action-modules-s3upload) + [WebDownload (Linux, Windows, macOS)](#action-modules-webdownload) ### S3Download (Linux, Windows, macOS) With the `S3Download` action module, you can download an Amazon S3 object, or a set of objects, to a local file or folder that you specify with the `destination` path. If any file already exists in the specified location, and the `overwrite` flag is set to true, `S3Download` overwrites the file. Your `source` location can point to a specific object in Amazon S3, or you can use a key prefix with an asterisk wildcard (`*`) to download a set of objects that match the key prefix path. When you specify a key prefix in your `source` location, the `S3Download` action module downloads everything that matches the prefix (files and folders included). Make sure that the key prefix ends with a forward-slash, followed by an asterisk (`/*`), so that you download everything that matches the prefix. For example: `s3://my-bucket/my-folder/*`. If the `S3Download` action for a specified key prefix fails during a download, the folder content is not rolled back to its state prior to the failure. The destination folder remains as it was at the time of the failure. **Supported use cases** The `S3Download` action module supports the following use cases: + The Amazon S3 object is downloaded to a local folder, as specified in the download path. + Amazon S3 objects (with a key prefix in the Amazon S3 file path) are downloaded to the specified local folder, which recursively copies all Amazon S3 objects that match the key prefix to the local folder. **IAM requirements** The IAM role that you associate with your instance profile must have permissions to run the `S3Download` action module. The following IAM policies must be attached to the IAM role that is associated with the instance profile: + **Single file**: `s3:GetObject` against the bucket/object (for example, `arn:aws:s3:::BucketName/*`). + **Multiple files**: `s3:ListBucket` against the bucket/object (for example, `arn:aws:s3:::BucketName`) and `s3:GetObject` against the bucket/object (for example, `arn:aws:s3:::BucketName/*`). **Input** | Key | Description | Type | Required | Default | | --- | --- | --- | --- | --- | | `source` | The Amazon S3 bucket that is the source for your download. You can specify a path to a specific object, or use a key prefix, that ends with a forward-slash, followed by an asterisk wildcard (`/*`), to download a set of objects that match the key prefix. | String | Yes | N/A | | `destination` | The local path where the Amazon S3 objects are downloaded. To download a single file, you must specify the file name as part of the path. For example, `/myfolder/package.zip`. | String | Yes | N/A | | `expectedBucketOwner` | Expected owner account ID of the bucket provided in the `source` path. We recommend that you verify the ownership of the Amazon S3 bucket specified in the source. | String | No | N/A | | `overwrite` | When set to true, if a file of the same name already exists in the destination folder for the specified local path, the download file overwrites the local file. When set to false, the existing file on the local system is protected from being overwritten, and the action module fails with a download error. For example, `Error: S3Download: File already exists and "overwrite" property for "destination" file is set to false. Cannot download.` | Boolean | No | true | **Note** For the following examples, the Windows folder path can be replaced with a Linux path. For example, `C:\myfolder\package.zip` can be replaced with `/myfolder/package.zip`. **Input example: copy an Amazon S3 object to a local file** The following example shows how to copy an Amazon S3 object to a local file. ``` - name: DownloadMyFile action: S3Download inputs: - source: s3://amzn-s3-demo-source-bucket/path/to/package.zip destination: C:\myfolder\package.zip expectedBucketOwner: 123456789022 overwrite: false - source: s3://amzn-s3-demo-source-bucket/path/to/package.zip destination: C:\myfolder\package.zip expectedBucketOwner: 123456789022 overwrite: true - source: s3://amzn-s3-demo-source-bucket/path/to/package.zip destination: C:\myfolder\package.zip expectedBucketOwner: 123456789022 ``` **Input example: copy all Amazon S3 objects in an Amazon S3 bucket with key prefix to a local folder** The following example shows how to copy all Amazon S3 objects in an Amazon S3 bucket with the key prefix to a local folder. Amazon S3 has no concept of a folder, therefore all objects that match the key prefix are copied. The maximum number of objects that can be downloaded is 1000. ``` - name: MyS3DownloadKeyprefix action: S3Download maxAttempts: 3 inputs: - source: s3://amzn-s3-demo-source-bucket/path/to/* destination: C:\myfolder\ expectedBucketOwner: 123456789022 overwrite: false - source: s3://amzn-s3-demo-source-bucket/path/to/* destination: C:\myfolder\ expectedBucketOwner: 123456789022 overwrite: true - source: s3://amzn-s3-demo-source-bucket/path/to/* destination: C:\myfolder\ expectedBucketOwner: 123456789022 ``` **Output** None. ### S3Upload (Linux, Windows, macOS) With the **S3Upload** action module, you can upload a file from a source file or folder to an Amazon S3 location. You can use a wildcard (`*`) in the path specified for your source location to upload all of the files whose path matches the wildcard pattern. If the recursive **S3Upload** action fails, any files that have already been uploaded will remain in the destination Amazon S3 bucket. **Supported use cases** + Local file to Amazon S3 object. + Local files in folder (with wildcard) to Amazon S3 key prefix. + Copy local folder (must have `recurse` set to `true`) to Amazon S3 key prefix. **IAM requirements** The IAM role that you associate with your instance profile must have permissions to run the `S3Upload` action module. The following IAM policy must be attached to the IAM role that is associated with the instance profile. The policy must grant `s3:PutObject` permissions to the target Amazon S3 bucket. For example, `arn:aws:s3:::BucketName/*`). **Input** | Key | Description | Type | Required | Default | | --- | --- | --- | --- | --- | | `source` | The local path where source files/folders originate. The `source` supports an asterisk wildcard (`*`). | String | Yes | N/A | | `destination` | The path for the destination Amazon S3 bucket where source files/folders are uploaded. | String | Yes | N/A | | `recurse` | When set to `true`, performs **S3Upload** recursively. | String | No | `false` | | `expectedBucketOwner` | The expected owner account ID for the Amazon S3 bucket specified in the destination path. We recommend that you verify the ownership of the Amazon S3 bucket specified in the destination. | String | No | N/A | **Input example: copy a local file to an Amazon S3 object** The following example shows how to copy a local file to an Amazon S3 object. ``` - name: MyS3UploadFile action: S3Upload onFailure: Abort maxAttempts: 3 inputs: - source: C:\myfolder\package.zip destination: s3://amzn-s3-demo-destination-bucket/path/to/package.zip expectedBucketOwner: 123456789022 ``` **Input example: copy all files in a local folder to an Amazon S3 bucket with key prefix** The following example shows how to copy all files in the local folder to an Amazon S3 bucket with key prefix. This example does not copy sub-folders or their contents because `recurse` is not specified, and it defaults to `false`. ``` - name: MyS3UploadMultipleFiles action: S3Upload onFailure: Abort maxAttempts: 3 inputs: - source: C:\myfolder\* destination: s3://amzn-s3-demo-destination-bucket/path/to/ expectedBucketOwner: 123456789022 ``` **Input example: copy all files and folders recursively from a local folder to an Amazon S3 bucket** The following example shows how to copy all files and folders recursively from a local folder to an Amazon S3 bucket with key prefix. ``` - name: MyS3UploadFolder action: S3Upload onFailure: Abort maxAttempts: 3 inputs: - source: C:\myfolder\* destination: s3://amzn-s3-demo-destination-bucket/path/to/ recurse: true expectedBucketOwner: 123456789022 ``` **Output** None. ### WebDownload (Linux, Windows, macOS) The **WebDownload** action module allows you to download files and resources from a remote location over the HTTP/HTTPS protocol (*HTTPS is recommended*). There are no limits on the number or size of downloads. This module handles retry and exponential backoff logic. Each download operation is allocated a maximum of 5 attempts to succeed according to user inputs. These attempts differ from those specified in the `maxAttempts` field of document `steps`, which are related to action module failures. This action module implicitly handles redirects. All HTTP status codes, except for `200`, result in an error. **Input** | Key name | Description | Type | Required | Default | | --- | --- | --- | --- | --- | | source | The valid HTTP/HTTPS URL (HTTPS is recommended), which follows the RFC 3986 standard. Chaining expressions are permitted. | String | Yes | N/A | | destination | An absolute or relative file or folder path on the local system. Folder paths must end with /. If they do not end with /, they will be treated as file paths. The module creates any required file or folder for successful downloads. Chaining expressions are permitted. | String | Yes | N/A | | overwrite | When enabled, overwrites any existing files on the local system with the downloaded file or resource. When not enabled, any existing files on the local system are not overwritten, and the action module fails with an error. When overwrite is enabled and checksum and algorithm are specified, then the action module downloads the file only if the checksum and the hash of any pre-existing files do not match. | Boolean | No | true | | checksum | When you specify the checksum, it is checked against the hash of the downloaded file that is generated with the supplied algorithm. For file verification to be enabled, both the checksum and the algorithm must be provided. Chaining expressions are permitted. | String | No | N/A | | algorithm | The algorithm used to calculate the checksum. The options are MD5, SHA1, SHA256, and SHA512. For file verification to be enabled, both the checksum and the algorithm must be provided. Chaining expressions are permitted. | String | No | N/A | | ignoreCertificateErrors | SSL certificate validation is ignored when enabled. | Boolean | No | false | **Output** | Key name | Description | Type | | --- | --- | --- | | destination | Newline character-delimited string that specifies the destination path where the downloaded files or resources are stored. | String | **Input example: download remote file to local destination** ``` - name: DownloadRemoteFile action: WebDownload maxAttempts: 3 inputs: - source: https://testdomain/path/to/java14.zip destination: C:\testfolder\package.zip ``` **Output:** ``` { "destination": "C:\\testfolder\\package.zip" } ``` **Input example: download more than one remote file to more than one local destination ** ``` - name: DownloadRemoteFiles action: WebDownload maxAttempts: 3 inputs: - source: https://testdomain/path/to/java14.zip destination: /tmp/java14_renamed.zip - source: https://testdomain/path/to/java14.zip destination: /tmp/create_new_folder_and_add_java14_as_zip/ ``` **Output:** ``` { "destination": "/tmp/create_new_folder/java14_renamed.zip\n/tmp/create_new_folder_and_add_java14_as_zip/java14.zip" } ``` **Input example: download one remote file without overwriting local destination, and download another remote file with file verification** ``` - name: DownloadRemoteMultipleProperties action: WebDownload maxAttempts: 3 inputs: - source: https://testdomain/path/to/java14.zip destination: C:\create_new_folder\java14_renamed.zip overwrite: false - source: https://testdomain/path/to/java14.zip destination: C:\create_new_folder_and_add_java14_as_zip\ checksum: ac68bbf921d953d1cfab916cb6120864 algorithm: MD5 overwrite: true ``` **Output:** ``` { "destination": "C:\\create_new_folder\\java14_renamed.zip\nC:\\create_new_folder_and_add_java14_as_zip\\java14.zip" } ``` **Input example: download remote file and ignore SSL certification validation** ``` - name: DownloadRemoteIgnoreValidation action: WebDownload maxAttempts: 3 inputs: - source: https://www.bad-ssl.com/resource destination: /tmp/downloads/ ignoreCertificateErrors: true ``` **Output:** ``` { "destination": "/tmp/downloads/resource" } ``` ## File system operations modules The following section contains details for action modules that perform file system operations. **Topics** + [AppendFile (Linux, Windows, macOS)](#action-modules-appendfile) + [CopyFile (Linux, Windows, macOS)](#action-modules-copyfile) + [CopyFolder (Linux, Windows, macOS)](#action-modules-copyfolder) + [CreateFile (Linux, Windows, macOS)](#action-modules-createfile) + [CreateFolder (Linux, Windows, macOS)](#action-modules-createfolder) + [CreateSymlink (Linux, Windows, macOS)](#action-modules-createsymlink) + [DeleteFile (Linux, Windows, macOS)](#action-modules-deletefile) + [DeleteFolder (Linux, Windows, macOS)](#action-modules-deletefolder) + [ListFiles (Linux, Windows, macOS)](#action-modules-listfiles) + [MoveFile (Linux, Windows, macOS)](#action-modules-movefile) + [MoveFolder (Linux, Windows, macOS)](#action-modules-movefolder) + [ReadFile (Linux, Windows, macOS)](#action-modules-readfile) + [SetFileEncoding (Linux, Windows, macOS)](#action-modules-setfileencoding) + [SetFileOwner (Linux, Windows, macOS)](#action-modules-setfileowner) + [SetFolderOwner (Linux, Windows, macOS)](#action-modules-setfolderowner) + [SetFilePermissions (Linux, Windows, macOS)](#action-modules-setfilepermissions) + [SetFolderPermissions (Linux, Windows, macOS)](#action-modules-setfolderpermissions) ### AppendFile (Linux, Windows, macOS) The **AppendFile** action module adds specified content to the preexisting content of a file. If the file encoding value is different from the default encoding (`utf-8`) value, then you can specify the file encoding value by using the `encoding` option. By default, `utf-16` and `utf-32` are assumed to use little-endian encoding. The action module returns an error when the following occurs: + The specified file does not exist at runtime. + You don't have write permissions to modify the file content. + The module encounters an error during the file operation. **Input** | Key name | Description | Type | Required | Default value | Acceptable values | Supported on all platforms | | --- | --- | --- | --- | --- | --- | --- | | path | The file path. | String | Yes | N/A | N/A | Yes | | content | The content to be appended to the file. | String | No | Empty string | N/A | Yes | | encoding | The encoding standard. | String | No | utf8 | utf8, utf-8, utf16,utf-16, utf16-LE, utf-16-LE utf16-BE, utf-16-BE, utf32, utf-32, utf32-LE,utf-32-LE, utf32-BE, and utf-32-BE. The value of the encoding option is case insensitive. | Yes | **Input example: append file without encoding (Linux)** ``` - name: AppendingFileWithOutEncodingLinux action: AppendFile inputs: - path: ./Sample.txt content: "The string to be appended to the file" ``` **Input example: append file without encoding (Windows)** ``` - name: AppendingFileWithOutEncodingWindows action: AppendFile inputs: - path: C:\MyFolder\MyFile.txt content: "The string to be appended to the file" ``` **Input example: append file with encoding (Linux)** ``` - name: AppendingFileWithEncodingLinux action: AppendFile inputs: - path: /FolderName/SampleFile.txt content: "The string to be appended to the file" encoding: UTF-32 ``` **Input example: append file with encoding (Windows)** ``` - name: AppendingFileWithEncodingWindows action: AppendFile inputs: - path: C:\MyFolderName\SampleFile.txt content: "The string to be appended to the file" encoding: UTF-32 ``` **Input example: append file with empty string (Linux)** ``` - name: AppendingEmptyStringLinux action: AppendFile inputs: - path: /FolderName/SampleFile.txt ``` **Input example: append file with empty string (Windows)** ``` - name: AppendingEmptyStringWindows action: AppendFile inputs: - path: C:\MyFolderName\SampleFile.txt ``` **Output** None. ### CopyFile (Linux, Windows, macOS) The **CopyFile** action module copies files from the specified source to the specified destination. By default, the module recursively creates the destination folder if it does not exist at runtime. If a file with the specified name already exists in the specified folder, the action module, by default, overwrites the existing file. You can override this default behavior by setting the overwrite option to `false`. When the overwrite option is set to `false`, and there is already a file in the specified location with the specified name, the action module will return an error. This option works the same as the `cp` command in Linux, which overwrites by default. The source file name can include a wildcard (`*`). Wildcard characters are accepted only after the last file path separator (`/` or `\`). If wildcard characters are included in the source file name, all of the files that match the wildcard are copied to the destination folder. If you want to move more than one file by using a wildcard character, the input to the `destination` option must end with a file path separator (`/` or `\`), which indicates that the destination input is a folder. If the destination file name is different from the source file name, you can specify the destination file name using the `destination` option. If you do not specify a destination file name, the name of the source file is used to create the destination file. Any text that follows the last file path separator (`/` or `\`) is treated as the file name. If you want to use the same file name as the source file, then the input of the `destination` option must end with a file path separator (`/` or `\`). The action module returns an error when the following occurs: + You do not have permission to create a file in the specified folder. + The source files do not exist at runtime. + There is already a folder with the specified file name and the `overwrite` option is set to `false`. + The action module encounters an error while performing the operation. **Input** | Key name | Description | Type | Required | Default value | Acceptable values | Supported on all platforms | | --- | --- | --- | --- | --- | --- | --- | | source | The source file path. | String | Yes | N/A | N/A | Yes | | destination | The destination file path. | String | Yes | N/A | N/A | Yes | | overwrite | When set to false, the destination files will not be replaced when there is already a file in the specified location with the specified name. | Boolean | No | true | N/A | Yes | **Input example: copy a file (Linux)** ``` - name: CopyingAFileLinux action: CopyFile inputs: - source: /Sample/MyFolder/Sample.txt destination: /MyFolder/destinationFile.txt ``` **Input example: copy a file (Windows)** ``` - name: CopyingAFileWindows action: CopyFile inputs: - source: C:\MyFolder\Sample.txt destination: C:\MyFolder\destinationFile.txt ``` **Input example: copy a file using the source file name (Linux)** ``` - name: CopyingFileWithSourceFileNameLinux action: CopyFile inputs: - source: /Sample/MyFolder/Sample.txt destination: /MyFolder/ ``` **Input example: copy a file using the source file name (Windows)** ``` - name: CopyingFileWithSourceFileNameWindows action: CopyFile inputs: - source: C:\Sample\MyFolder\Sample.txt destination: C:\MyFolder\ ``` **Input example: copy a file using the wildcard character (Linux)** ``` - name: CopyingFilesWithWildCardLinux action: CopyFile inputs: - source: /Sample/MyFolder/Sample* destination: /MyFolder/ ``` **Input example: copy a file using the wildcard character (Windows)** ``` - name: CopyingFilesWithWildCardWindows action: CopyFile inputs: - source: C:\Sample\MyFolder\Sample* destination: C:\MyFolder\ ``` **Input example: copy a file without overwriting (Linux)** ``` - name: CopyingFilesWithoutOverwriteLinux action: CopyFile inputs: - source: /Sample/MyFolder/Sample.txt destination: /MyFolder/destinationFile.txt overwrite: false ``` **Input example: copy a file without overwriting (Windows)** ``` - name: CopyingFilesWithoutOverwriteWindows action: CopyFile inputs: - source: C:\Sample\MyFolder\Sample.txt destination: C:\MyFolder\destinationFile.txt overwrite: false ``` **Output** None. ### CopyFolder (Linux, Windows, macOS) The **CopyFolder** action module copies a folder from the specified source to the specified destination. The input for the `source` option is the folder to copy, and the input for the `destination` option is the folder where the contents of the source folder are copied. By default, the module recursively creates the destination folder if it does not exist at runtime. If a folder with the specified name already exists in the specified folder, the action module, by default, overwrites the existing folder. You can override this default behavior by setting the overwrite option to `false`. When the overwrite option is set to `false`, and there is already a folder in the specified location with the specified name, the action module will return an error. The source folder name can include a wildcard (`*`). Wildcard characters are accepted only after the last file path separator (`/` or `\`). If wildcard characters are included in the source folder name, all of the folders that match the wildcard are copied to the destination folder. If you want to copy more than one folder by using a wildcard character, the input to the `destination` option must end with a file path separator (`/` or `\`), which indicates that the destination input is a folder. If the destination folder name is different from the source folder name, you can specify the destination folder name using the `destination` option. If you do not specify a destination folder name, the name of the source folder is used to create the destination folder. Any text that follows the last file path separator (`/` or `\`) is treated as the folder name. If you want to use the same folder name as the source folder, then the input of the `destination` option must end with a file path separator (`/` or `\`). The action module returns an error when the following occurs: + You do not have permission to create a folder in the specified folder. + The source folders do not exist at runtime. + There is already a folder with the specified folder name and the `overwrite` option is set to `false`. + The action module encounters an error while performing the operation. **Input** | Key name | Description | Type | Required | Default value | Acceptable values | Supported on all platforms | | --- | --- | --- | --- | --- | --- | --- | | source | The source folder path. | String | Yes | N/A | N/A | Yes | | destination | The destination folder path. | String | Yes | N/A | N/A | Yes | | overwrite | When set to false, the destination folders will not be replaced when there is already a folder in the specified location with the specified name. | Boolean | No | true | N/A | Yes | **Input example: copy a folder (Linux)** ``` - name: CopyingAFolderLinux action: CopyFolder inputs: - source: /Sample/MyFolder/SampleFolder destination: /MyFolder/destinationFolder ``` **Input example: copy a folder (Windows)** ``` - name: CopyingAFolderWindows action: CopyFolder inputs: - source: C:\Sample\MyFolder\SampleFolder destination: C:\MyFolder\destinationFolder ``` **Input example: copy a folder using the source folder name (Linux)** ``` - name: CopyingFolderSourceFolderNameLinux action: CopyFolder inputs: - source: /Sample/MyFolder/SourceFolder destination: /MyFolder/ ``` **Input example: copy a folder using the source folder name (Windows)** ``` - name: CopyingFolderSourceFolderNameWindows action: CopyFolder inputs: - source: C:\Sample\MyFolder\SampleFolder destination: C:\MyFolder\ ``` **Input example: copy a folder using the wildcard character (Linux)** ``` - name: CopyingFoldersWithWildCardLinux action: CopyFolder inputs: - source: /Sample/MyFolder/Sample* destination: /MyFolder/ ``` **Input example: copy a folder using the wildcard character (Windows)** ``` - name: CopyingFoldersWithWildCardWindows action: CopyFolder inputs: - source: C:\Sample\MyFolder\Sample* destination: C:\MyFolder\ ``` **Input example: copy a folder without overwriting (Linux)** ``` - name: CopyingFoldersWithoutOverwriteLinux action: CopyFolder inputs: - source: /Sample/MyFolder/SourceFolder destination: /MyFolder/destinationFolder overwrite: false ``` **Input example: copy a folder without overwriting (Windows)** ``` - name: CopyingFoldersWithoutOverwrite action: CopyFolder inputs: - source: C:\Sample\MyFolder\SourceFolder destination: C:\MyFolder\destinationFolder overwrite: false ``` **Output** None. ### CreateFile (Linux, Windows, macOS) The **CreateFile** action module creates a file in a specified location. By default, if required, the module also recursively creates the parent folders. If the file already exists in the specified folder, the action module, by default, truncates or overwrites the existing file. You can override this default behavior by setting the overwrite option to `false`. When the overwrite option is set to `false`, and there is already a file in the specified location with the specified name, the action module will return an error. If the file encoding value is different from the default encoding (`utf-8`) value, then you can specify the file encoding value by using the `encoding` option. By default, `utf-16` and `utf-32` are assumed to use little-endian encoding. `owner`, `group`, and `permissions` are optional inputs. The input for `permissions` must be a string value. Files are created with default values when not provided. These options are not supported on Windows platforms. This action module validates and returns an error if the `owner`, `group`, and `permissions` options are used on Windows platforms. This action module can create a file the with permissions defined by the default `umask` value of the operating system. You must set the `umask` value if you want to override the default value. The action module returns an error when the following occurs: + You do not have permission to create a file or a folder in the specified parent folder. + The action module encounters an error while performing the operation. **Input** | Key name | Description | Type | Required | Default value | Acceptable values | Supported on all platforms | | --- | --- | --- | --- | --- | --- | --- | | path | The file path. | String | Yes | N/A | N/A | Yes | | content | The text content of the file. | String | No | N/A | N/A | Yes | | encoding | The encoding standard. | String | No | utf8 | utf8, utf-8, utf16,utf-16, utf16-LE, utf-16-LE utf16-BE, utf-16-BE, utf32, utf-32, utf32-LE,utf-32-LE, utf32-BE, and utf-32-BE. The value of the encoding option is case insensitive. | Yes | | owner | The user name or ID. | String | No | N/A | N/A | Not supported on Windows. | | group | The group name or ID. | String | No | The current user. | N/A | Not supported on Windows. | | permissions | The file permissions. | String | No | 0666 | N/A | Not supported on Windows. | | overwrite | If the name of the specified file already exists, setting this value to false prevents the file from being truncated or overwritten by default. | Boolean | No | true | N/A | Yes | **Input example: create a file without overwriting (Linux)** ``` - name: CreatingFileWithoutOverwriteLinux action: CreateFile inputs: - path: /home/UserName/Sample.txt content: The text content of the sample file. overwrite: false ``` **Input example: create a file without overwriting (Windows)** ``` - name: CreatingFileWithoutOverwriteWindows action: CreateFile inputs: - path: C:\Temp\Sample.txt content: The text content of the sample file. overwrite: false ``` **Input example: create a file with file properties** ``` - name: CreatingFileWithFileProperties action: CreateFile inputs: - path: SampleFolder/Sample.txt content: The text content of the sample file. encoding: UTF-16 owner: Ubuntu group: UbuntuGroup permissions: 0777 - path: SampleFolder/SampleFile.txt permissions: 755 - path: SampleFolder/TextFile.txt encoding: UTF-16 owner: root group: rootUserGroup ``` **Input example: create a file without file properties** ``` - name: CreatingFileWithoutFileProperties action: CreateFile inputs: - path: ./Sample.txt - path: Sample1.txt ``` **Input example: create an empty file to skip a section in the Linux clean up script** ``` - name: CreateSkipCleanupfile action: CreateFile inputs: - path: ``` For more information, see [Override the Linux clean up script](security-best-practices.md#override-linux-cleanup-script) **Output** None. ### CreateFolder (Linux, Windows, macOS) The **CreateFolder** action module creates a folder in a specified location. By default, if required, the module also recursively creates the parent folders. If the folder already exists in the specified folder, the action module, by default, truncates or overwrites the existing folder. You can override this default behavior by setting the overwrite option to `false`. When the overwrite option is set to `false`, and there is already a folder in the specified location with the specified name, the action module will return an error. `owner`, `group`, and `permissions` are optional inputs. The input for `permissions` must be a string value. These options are not supported on Windows platforms. This action module validates and returns an error if the `owner`, `group`, and `permissions` options are used on Windows platforms. This action module can create a folder the with permissions defined by the default `umask` value of the operating system. You must set the `umask` value if you want to override the default value. The action module returns an error when the following occurs: + You do not have permission to create a folder in the specified location. + The action module encounters an error while performing the operation. **Input** | Key name | Description | Type | Required | Default value | Acceptable values | Supported on all platforms | | --- | --- | --- | --- | --- | --- | --- | | path | The folder path. | String | Yes | N/A | N/A | Yes | | owner | The user name or ID. | String | No | The current user. | N/A | Not supported on Windows. | | group | The group name or ID. | String | No | The group of the current user. | N/A | Not supported on Windows. | | permissions | The folder permissions. | String | No | 0777 | N/A | Not supported on Windows. | | overwrite | If the name of the specified file already exists, setting this value to false prevents the file from being truncated or overwritten by default. | Boolean | No | true | N/A | Yes | **Input example: create a folder (Linux)** ``` - name: CreatingFolderLinux action: CreateFolder inputs: - path: /Sample/MyFolder/ ``` **Input example: create a folder (Windows)** ``` - name: CreatingFolderWindows action: CreateFolder inputs: - path: C:\MyFolder ``` **Input example: create a folder specifying folder properties** ``` - name: CreatingFolderWithFolderProperties action: CreateFolder inputs: - path: /Sample/MyFolder/Sample/ owner: SampleOwnerName group: SampleGroupName permissions: 0777 - path: /Sample/MyFolder/SampleFoler/ permissions: 777 ``` **Input example: create a folder that overwrites the existing folder, if there is one.** ``` - name: CreatingFolderWithOverwrite action: CreateFolder inputs: - path: /Sample/MyFolder/Sample/ overwrite: true ``` **Output** None. ### CreateSymlink (Linux, Windows, macOS) The **CreateSymlink** action module creates symbolic links, or files that contain a reference to another file. This module is not supported on Windows platforms. The input for the `path` and `target` options can be either an absolute or relative path. If the input for the `path` option is a relative path, it is replaced with the absolute path when the link is created. By default, when a link with the specified name already exists in the specified folder, the action module returns an error. You can override this default behavior by setting the `force` option to `true`. When the `force` option is set to `true`, the module will overwrite the existing link. If a parent folder does not exist, the action module creates the folder recursively, by default. The action module returns an error when the following occurs: + The target file does not exist at runtime. + A nonsymbolic link file with the specified name already exists. + The action module encounters an error while performing the operation. **Input** | Key name | Description | Type | Required | Default value | Acceptable values | Supported on all platforms | | --- | --- | --- | --- | --- | --- | --- | | path | The file path. | String | Yes | N/A | N/A | Not supported on Windows. | | target | The target file path to which the symbolic link points. | String | Yes | N/A | N/A | Not supported on Windows. | | force | Forces the creation of a link when a link with the same name already exists. | Boolean | No | false | N/A | Not supported on Windows. | **Input example: create symbolic link that forces the creation of a link** ``` - name: CreatingSymbolicLinkWithForce action: CreateSymlink inputs: - path: /Folder2/Symboliclink.txt target: /Folder/Sample.txt force: true ``` **Input example: create a symbolic link that does not force the creation of a link** ``` - name: CreatingSymbolicLinkWithOutForce action: CreateSymlink inputs: - path: Symboliclink.txt target: /Folder/Sample.txt ``` **Output** None. ### DeleteFile (Linux, Windows, macOS) The **DeleteFile** action module deletes a file or files in a specified location. The input of `path` should be a valid file path or a file path with a wild card character (`*`) in the file name. When wildcard characters are specified in the file name, all of the files within the same folder that match the wildcard will be deleted. The action module returns an error when the following occurs: + You do not have permission to perform delete operations. + The action module encounters an error while performing the operation. **Input** | Key name | Description | Type | Required | Default value | Acceptable values | Supported on all platforms | | --- | --- | --- | --- | --- | --- | --- | | path | The file path. | String | Yes | N/A | N/A | Yes | **Input example: delete a single file (Linux)** ``` - name: DeletingSingleFileLinux action: DeleteFile inputs: - path: /SampleFolder/MyFolder/Sample.txt ``` **Input example: delete a single file (Windows)** ``` - name: DeletingSingleFileWindows action: DeleteFile inputs: - path: C:\SampleFolder\MyFolder\Sample.txt ``` **Input example: delete a file that ends with "log" (Linux)** ``` - name: DeletingFileEndingWithLogLinux action: DeleteFile inputs: - path: /SampleFolder/MyFolder/*log ``` **Input example: delete a file that ends with "log" (Windows)** ``` - name: DeletingFileEndingWithLogWindows action: DeleteFile inputs: - path: C:\SampleFolder\MyFolder\*log ``` **Input example: delete all files in a specified folder (Linux)** ``` - name: DeletingAllFilesInAFolderLinux action: DeleteFile inputs: - path: /SampleFolder/MyFolder/* ``` **Input example: delete all files in a specified folder (Windows)** ``` - name: DeletingAllFilesInAFolderWindows action: DeleteFile inputs: - path: C:\SampleFolder\MyFolder\* ``` **Output** None. ### DeleteFolder (Linux, Windows, macOS) The **DeleteFolder** action module deletes folders. If the folder is not empty, you must set the `force` option to `true` to remove the folder and its contents. If you do not set the `force` option to `true`, and the folder you are trying to delete is not empty, the action module returns an error. The default value of the `force` option is `false`. The action module returns an error when the following occurs: + You do not have permission to perform delete operations. + The action module encounters an error while performing the operation. **Input** | Key name | Description | Type | Required | Default value | Acceptable values | Supported on all platforms | | --- | --- | --- | --- | --- | --- | --- | | path | The folder path. | String | Yes | N/A | N/A | Yes | | force | Removes the folder whether or not the folder is empty. | Boolean | No | false | N/A | Yes | **Input example: delete a folder that is not empty using the `force` option (Linux)** ``` - name: DeletingFolderWithForceOptionLinux action: DeleteFolder inputs: - path: /Sample/MyFolder/Sample/ force: true ``` **Input example: delete a folder that is not empty using the `force` option (Windows)** ``` - name: DeletingFolderWithForceOptionWindows action: DeleteFolder inputs: - path: C:\Sample\MyFolder\Sample\ force: true ``` **Input example: delete a folder (Linux)** ``` - name: DeletingFolderWithOutForceLinux action: DeleteFolder inputs: - path: /Sample/MyFolder/Sample/ ``` **Input example: delete a folder (Windows)** ``` - name: DeletingFolderWithOutForce action: DeleteFolder inputs: - path: C:\Sample\MyFolder\Sample\ ``` **Output** None. ### ListFiles (Linux, Windows, macOS) The **ListFiles** action module lists the files in a specified folder. When the recursive option is set to `true`, it lists the files in subfolders. This module does not list files in subfolders by default. To list all of the files with names that match a specified pattern, use the `fileNamePattern` option to provide the pattern. The `fileNamePattern` option accepts the wildcard (`*`) value. When the `fileNamePattern` is provided, all of the files that match the specified file name format are returned. The action module returns an error when the following occurs: + The specified folder does not exist at runtime. + You do not have permission to create a file or a folder in the specified parent folder. + The action module encounters an error while performing the operation. **Input** | Key name | Description | Type | Required | Default value | Acceptable values | Supported on all platforms | | --- | --- | --- | --- | --- | --- | --- | | path | The folder path. | String | Yes | N/A | N/A | Yes | | fileNamePattern | The pattern to match to list all files with names that match the pattern. | String | No | N/A | N/A | Yes | | recursive | Lists files in the folder recursively. | Boolean | No | false | N/A | Yes | **Input example: list files in specified folder (Linux)** ``` - name: ListingFilesInSampleFolderLinux action: ListFiles inputs: - path: /Sample/MyFolder/Sample ``` **Input example: list files in specified folder (Windows)** ``` - name: ListingFilesInSampleFolderWindows action: ListFiles inputs: - path: C:\Sample\MyFolder\Sample ``` **Input example: list files that end with "log" (Linux)** ``` - name: ListingFilesWithEndingWithLogLinux action: ListFiles inputs: - path: /Sample/MyFolder/ fileNamePattern: *log ``` **Input example: list files that end with "log" (Windows)** ``` - name: ListingFilesWithEndingWithLogWindows action: ListFiles inputs: - path: C:\Sample\MyFolder\ fileNamePattern: *log ``` **Input example: list files recursively** ``` - name: ListingFilesRecursively action: ListFiles inputs: - path: /Sample/MyFolder/ recursive: true ``` **Output** | Key name | Description | Type | | --- | --- | --- | | files | The list of files. | String | **Output example** ``` { "files": "/sample1.txt,/sample2.txt,/sample3.txt" } ``` ### MoveFile (Linux, Windows, macOS) The **MoveFile** action module moves files from the specified source to the specified destination. If the file already exists in the specified folder, the action module, by default, overwrites the existing file. You can override this default behavior by setting the overwrite option to `false`. When the overwrite option is set to `false`, and there is already a file in the specified location with the specified name, the action module will return an error. This option works the same as the `mv` command in Linux, which overwrites by default. The source file name can include a wildcard (`*`). Wildcard characters are accepted only after the last file path separator (`/` or `\`). If wildcard characters are included in the source file name, all of the files that match the wildcard are copied to the destination folder. If you want to move more than one file by using a wildcard character, the input to the `destination` option must end with a file path separator (`/` or `\`), which indicates that the destination input is a folder. If the destination file name is different from the source file name, you can specify the destination file name using the `destination` option. If you do not specify a destination file name, the name of the source file is used to create the destination file. Any text that follows the last file path separator (`/` or `\`) is treated as the file name. If you want to use the same file name as the source file, then the input of the `destination` option must end with a file path separator (`/` or `\`). The action module returns an error when the following occurs: + You do not have permission to create a file in the specified folder. + The source files do not exist at runtime. + There is already a folder with the specified file name and the `overwrite` option is set to `false`. + The action module encounters an error while performing the operation. **Input** | Key name | Description | Type | Required | Default value | Acceptable values | Supported on all platforms | | --- | --- | --- | --- | --- | --- | --- | | source | The source file path. | String | Yes | N/A | N/A | Yes | | destination | The destination file path. | String | Yes | N/A | N/A | Yes | | overwrite | When set to false, the destination files will not be replaced when there is already a file in the specified location with the specified name. | Boolean | No | true | N/A | Yes | **Input example: move a file (Linux)** ``` - name: MovingAFileLinux action: MoveFile inputs: - source: /Sample/MyFolder/Sample.txt destination: /MyFolder/destinationFile.txt ``` **Input example: move a file (Windows)** ``` - name: MovingAFileWindows action: MoveFile inputs: - source: C:\Sample\MyFolder\Sample.txt destination: C:\MyFolder\destinationFile.txt ``` **Input example: move a file using the source file name (Linux)** ``` - name: MovingFileWithSourceFileNameLinux action: MoveFile inputs: - source: /Sample/MyFolder/Sample.txt destination: /MyFolder/ ``` **Input example: move a file using the source file name (Windows)** ``` - name: MovingFileWithSourceFileNameWindows action: MoveFile inputs: - source: C:\Sample\MyFolder\Sample.txt destination: C:\MyFolder ``` **Input example: move a file using a wildcard character (Linux)** ``` - name: MovingFilesWithWildCardLinux action: MoveFile inputs: - source: /Sample/MyFolder/Sample* destination: /MyFolder/ ``` **Input example: move a file using a wildcard character (Windows)** ``` - name: MovingFilesWithWildCardWindows action: MoveFile inputs: - source: C:\Sample\MyFolder\Sample* destination: C:\MyFolder ``` **Input example: move a file without overwriting (Linux)** ``` - name: MovingFilesWithoutOverwriteLinux action: MoveFile inputs: - source: /Sample/MyFolder/Sample.txt destination: /MyFolder/destinationFile.txt overwrite: false ``` **Input example: move a file without overwriting (Windows)** ``` - name: MovingFilesWithoutOverwrite action: MoveFile inputs: - source: C:\Sample\MyFolder\Sample.txt destination: C:\MyFolder\destinationFile.txt overwrite: false ``` **Output** None. ### MoveFolder (Linux, Windows, macOS) The **MoveFolder** action module moves folders from the specified source to the specified destination. The input for the `source` option is the folder to move, and the input to the `destination` option is the folder where the contents of the source folders are moved. If the destination parent folder or the input to the `destination` option does not exist at runtime, the default behavior of the module is to recursively create the folder at the specified destination. If a folder with the same as the source folder already exists in the destination folder, the action module, by default, overwrites the existing folder. You can override this default behavior by setting the overwrite option to `false`. When the overwrite option is set to `false`, and there is already a folder in the specified location with the specified name, the action module will return an error. The source folder name can include a wildcard (`*`). Wildcard characters are accepted only after the last file path separator (`/` or `\`). If wildcard characters are included in the source folder name, all of the folders that match the wildcard are copied to the destination folder. If you want to move more than one folder by using a wildcard character, the input to the `destination` option must end with a file path separator (`/` or `\`), which indicates that the destination input is a folder. If the destination folder name is different from the source folder name, you can specify the destination folder name using the `destination` option. If you do not specify a destination folder name, the name of the source folder is used to create the destination folder. Any text that follows the last file path separator (`/` or `\`) is treated as the folder name. If you want to use the same folder name as the source folder, then the input of the `destination` option must end with a file path separator (`/` or `\`). The action module returns an error when the following occurs: + You do not have permission to create a folder in the destination folder. + The source folders do not exist at runtime. + There is already a folder with the specified name and the `overwrite` option is set to `false`. + The action module encounters an error while performing the operation. **Input** | Key name | Description | Type | Required | Default value | Acceptable values | Supported on all platforms | | --- | --- | --- | --- | --- | --- | --- | | source | The source folder path. | String | Yes | N/A | N/A | Yes | | destination | The destination folder path. | String | Yes | N/A | N/A | Yes | | overwrite | When set to false, the destination folders will not be replaced when there is already a folder in the specified location with the specified name. | Boolean | No | true | N/A | Yes | **Input example: move a folder (Linux)** ``` - name: MovingAFolderLinux action: MoveFolder inputs: - source: /Sample/MyFolder/SourceFolder destination: /MyFolder/destinationFolder ``` **Input example: move a folder (Windows)** ``` - name: MovingAFolderWindows action: MoveFolder inputs: - source: C:\Sample\MyFolder\SourceFolder destination: C:\MyFolder\destinationFolder ``` **Input example: move a folder using the source folder name (Linux)** ``` - name: MovingFolderWithSourceFolderNameLinux action: MoveFolder inputs: - source: /Sample/MyFolder/SampleFolder destination: /MyFolder/ ``` **Input example: move a folder using the source folder name (Windows)** ``` - name: MovingFolderWithSourceFolderNameWindows action: MoveFolder inputs: - source: C:\Sample\MyFolder\SampleFolder destination: C:\MyFolder\ ``` **Input example: move a folder using a wildcard character (Linux)** ``` - name: MovingFoldersWithWildCardLinux action: MoveFolder inputs: - source: /Sample/MyFolder/Sample* destination: /MyFolder/ ``` **Input example: move a folder using a wildcard character (Windows)** ``` - name: MovingFoldersWithWildCardWindows action: MoveFolder inputs: - source: C:\Sample\MyFolder\Sample* destination: C:\MyFolder\ ``` **Input example: move a folder without overwriting (Linux)** ``` - name: MovingFoldersWithoutOverwriteLinux action: MoveFolder inputs: - source: /Sample/MyFolder/SampleFolder destination: /MyFolder/destinationFolder overwrite: false ``` **Input example: move a folder without overwriting (Windows)** ``` - name: MovingFoldersWithoutOverwriteWindows action: MoveFolder inputs: - source: C:\Sample\MyFolder\SampleFolder destination: C:\MyFolder\destinationFolder overwrite: false ``` **Output** None. ### ReadFile (Linux, Windows, macOS) The **ReadFile** action module reads the content of a text file of type string. This module can be used to read the content of a file for use in subsequent steps through chaining or for reading data to the `console.log` file. If the specified path is a symbolic link, this module returns the content of the target file. This module only supports text files. If the file encoding value is different from the default encoding (`utf-8`) value, then you can specify the file encoding value by using the `encoding` option. By default, `utf-16` and `utf-32` are assumed to use little-endian encoding. By default, this module cannot print the file content to the `console.log` file. You can override this setting by setting the `printFileContent` property to `true`. This module can return only the content of a file. It cannot parse files, such as Excel or JSON files. The action module returns an error when the following occurs: + The file does not exist at runtime. + The action module encounters an error while performing the operation. **Input** | Key name | Description | Type | Required | Default value | Acceptable values | Supported on all platforms | | --- | --- | --- | --- | --- | --- | --- | | path | The file path. | String | Yes | N/A | N/A | Yes | | encoding | The encoding standard. | String | No | utf8 | utf8, utf-8, utf16,utf-16, utf16-LE, utf-16-LE utf16-BE, utf-16-BE, utf32, utf-32, utf32-LE,utf-32-LE, utf32-BE, and utf-32-BE. The value of the encoding option is case insensitive. | Yes | | printFileContent | Prints the file content to the console.log file. | Boolean | No | false | N/A | Yes. | **Input example: read a file (Linux)** ``` - name: ReadingFileLinux action: ReadFile inputs: - path: /home/UserName/SampleFile.txt ``` **Input example: read a file (Windows)** ``` - name: ReadingFileWindows action: ReadFile inputs: - path: C:\Windows\WindowsUpdate.log ``` **Input example: read a file and specify encoding standard** ``` - name: ReadingFileWithFileEncoding action: ReadFile inputs: - path: /FolderName/SampleFile.txt encoding: UTF-32 ``` **Input example: read a file and print to the `console.log` file** ``` - name: ReadingFileToConsole action: ReadFile inputs: - path: /home/UserName/SampleFile.txt printFileContent: true ``` **Output** | Field | Description | Type | | --- | --- | --- | | content | The file content. | string | **Output example** ``` { "content" : "The file content" } ``` ### SetFileEncoding (Linux, Windows, macOS) The **SetFileEncoding** action module modifies the encoding property of an existing file. This module can convert file encoding from `utf-8` to a specified encoding standard. By default, `utf-16` and `utf-32` are assumed to be little-endian encoding. The action module returns an error when the following occurs: + You do not have permission to perform the specified modification. + The file does not exist at runtime. + The action module encounters an error while performing the operation. **Input** | Key name | Description | Type | Required | Default value | Acceptable values | Supported on all platforms | | --- | --- | --- | --- | --- | --- | --- | | path | The file path. | String | Yes | N/A | N/A | Yes | | encoding | The encoding standard. | String | No | utf8 | utf8, utf-8, utf16,utf-16, utf16-LE, utf-16-LE utf16-BE, utf-16-BE, utf32, utf-32, utf32-LE,utf-32-LE, utf32-BE, and utf-32-BE. The value of the encoding option is case insensitive. | Yes | **Input example: set file encoding property** ``` - name: SettingFileEncodingProperty action: SetFileEncoding inputs: - path: /home/UserName/SampleFile.txt encoding: UTF-16 ``` **Output** None. ### SetFileOwner (Linux, Windows, macOS) The **SetFileOwner** action module modifies the `owner` and`group` owner properties of an existing file. If the specified file is a symbolic link, the module modifies the `owner` property of the source file. This module is not supported on Windows platforms. This module accepts user and group names as inputs. If the group name is not provided, the module assigns the group owner of the file to the group that the user belongs to. The action module returns an error when the following occurs: + You do not have permission to perform the specified modification. + The specified user or group name does not exist at runtime. + The file does not exist at runtime. + The action module encounters an error while performing the operation. **Input** | Key name | Description | Type | Required | Default value | Acceptable values | Supported on all platforms | | --- | --- | --- | --- | --- | --- | --- | | path | The file path. | String | Yes | N/A | N/A | Not supported on Windows. | | owner | The user name. | string | Yes | N/A | N/A | Not supported on Windows. | | group | The name of the user group. | String | No | The name of the group that the user belongs to. | N/A | Not supported on Windows. | **Input example: set file owner property without specifying the name of the user group** ``` - name: SettingFileOwnerPropertyNoGroup action: SetFileOwner inputs: - path: /home/UserName/SampleText.txt owner: LinuxUser ``` **Input example: set file owner property by specifying the owner and the user group** ``` - name: SettingFileOwnerProperty action: SetFileOwner inputs: - path: /home/UserName/SampleText.txt owner: LinuxUser group: LinuxUserGroup ``` **Output** None. ### SetFolderOwner (Linux, Windows, macOS) The **SetFolderOwner** action module recursively modifies the `owner` and`group` owner properties of an existing folder. By default, the module can modify ownership for all of the contents in a folder. You can set the `recursive` option to `false` to override this behavior. This module is not supported on Windows platforms. This module accepts user and group names as inputs. If the group name is not provided, the module assigns the group owner of the file to the group that the user belongs to. The action module returns an error when the following occurs: + You do not have permission to perform the specified modification. + The specified user or group name does not exist at runtime. + The folder does not exist at runtime. + The action module encounters an error while performing the operation. **Input** | Key name | Description | Type | Required | Default value | Acceptable values | Supported on all platforms | | --- | --- | --- | --- | --- | --- | --- | | path | The folder path. | String | Yes | N/A | N/A | Not supported on Windows. | | owner | The user name. | string | Yes | N/A | N/A | Not supported on Windows. | | group | The name of the user group. | String | No | The name of the group that the user belongs to. | N/A | Not supported on Windows. | | recursive | Overrides the default behavior of modifying ownership for all of the contents of a folder when set to false. | Boolean | No | true | N/A | Not supported on Windows. | **Input example: set folder owner property without specifying the name of the user group** ``` - name: SettingFolderPropertyWithOutGroup action: SetFolderOwner inputs: - path: /SampleFolder/ owner: LinuxUser ``` **Input example: set folder owner property without overriding the ownership of all of the contents in a folder ** ``` - name: SettingFolderPropertyWithOutRecursively action: SetFolderOwner inputs: - path: /SampleFolder/ owner: LinuxUser recursive: false ``` **Input example: set file ownership property by specifying the name of the user group** ``` - name: SettingFolderPropertyWithGroup action: SetFolderOwner inputs: - path: /SampleFolder/ owner: LinuxUser group: LinuxUserGroup ``` **Output** None. ### SetFilePermissions (Linux, Windows, macOS) The **SetFilePermissions** action module modifies the `permissions` of an existing file. This module is not supported on Windows platforms. The input for `permissions` must be a string value. This action module can create a file the with permissions defined by the default umask value of the operating system. You must set the `umask` value if you want to override the default value. The action module returns an error when the following occurs: + You do not have permission to perform the specified modification. + The file does not exist at runtime. + The action module encounters an error while performing the operation. **Input** | Key name | Description | Type | Required | Default value | Acceptable values | Supported on all platforms | | --- | --- | --- | --- | --- | --- | --- | | path | The file path. | String | Yes | N/A | N/A | Not supported on Windows. | | permissions | The file permissions. | String | Yes | N/A | N/A | Not supported on Windows. | **Input example: modify file permissions** ``` - name: ModifyingFilePermissions action: SetFilePermissions inputs: - path: /home/UserName/SampleFile.txt permissions: 766 ``` **Output** None. ### SetFolderPermissions (Linux, Windows, macOS) The **SetFolderPermissions** action module recursively modifies the `permissions` of an existing folder and all of its subfiles and subfolders. By default, this module can modify permissions for all of the contents of the specified folder. You can set the `recursive` option to `false` to override this behavior. This module is not supported on Windows platforms. The input for `permissions` must be a string value. This action module can modify permissions according to the default umask value of the operating system. You must set the `umask` value if you want to override the default value. The action module returns an error when the following occurs: + You do not have permission to perform the specified modification. + The folder does not exist at runtime. + The action module encounters an error while performing the operation. **Input** | Key name | Description | Type | Required | Default value | Acceptable values | Supported on all platforms | | --- | --- | --- | --- | --- | --- | --- | | path | The folder path. | String | Yes | N/A | N/A | Not supported on Windows. | | permissions | The folder permissions. | String | Yes | N/A | N/A | Not supported on Windows. | | recursive | Overrides the default behavior of modifying permissions for all of the contents of a folder when set to false. | Boolean | No | true | N/A | Not supported on Windows. | **Input example: set folder permissions** ``` - name: SettingFolderPermissions action: SetFolderPermissions inputs: - path: SampleFolder/ permissions: 0777 ``` **Input example: set folder permissions without modifying permissions for all of the contents of a folder** ``` - name: SettingFolderPermissionsNoRecursive action: SetFolderPermissions inputs: - path: /home/UserName/SampleFolder/ permissions: 777 recursive: false ``` **Output** None. ## Software installation actions The following section describes action modules that install or uninstall software. **IAM requirements** If your installation download path is an S3 URI, then the IAM role that you associate with your instance profile must have permission to run the `S3Download` action module. To grant the required permission, attach the `S3:GetObject` IAM policy to the IAM role that is associated with your instance profile, and specify the path for your bucket. For example, `arn:aws:s3:::BucketName/*`). **Complex MSI Inputs** If your input strings contain double quote characters (`"`), you must use one of the following methods to ensure that they are interpreted correctly: + You can use single quotes (') on the outside of your string, to contain it, and double quotes (") inside of your string, as shown in the following example. ``` properties: COMPANYNAME: '"Acme ""Widgets"" and ""Gizmos."""' ``` In this case, if you need to use an apostrophe inside of your string, you must escape it. This means using another single quote (') before the apostrophe. + You can use double quotes (") on the outside of your string, to contain it. And you can escape any double quotes inside of your string, using the backslash character (`\`), as shown in the following example. ``` properties: COMPANYNAME: "\"Acme \"\"Widgets\"\" and \"\"Gizmos.\"\"\"" ``` Both of these methods pass the value `COMPANYNAME="Acme ""Widgets"" and ""Gizmos."""` to the **msiexec** command. **Topics** + [InstallMSI (Windows)](#action-modules-install-msi) + [UninstallMSI (Windows)](#action-modules-uninstall-msi) ### InstallMSI (Windows) The `InstallMSI` action module installs a Windows application using an MSI file. You can specify the MSI file using a local path, an S3 object URI, or a web URL. The reboot option configures the reboot behavior of the system. AWSTOE generates the **msiexec** command based on the input parameters for the action module. Values for the `path` (MSI file location) and `logFile` (log file location) input parameters must be enclosed in quotation marks ("). The following MSI exit codes are considered successful: + 0 (Success) + 1614 (ERROR\$1PRODUCT\$1UNINSTALLED) + 1641 (Reboot Initiated) + 3010 (Reboot Required) **Input** | Key name | Description | Type | Required | Default value | Acceptable values | | --- | --- | --- | --- | --- | --- | | path | Specify the MSI file location using one of the following: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/imagebuilder/latest/userguide/toe-action-modules.html) Chaining expressions are allowed. | String | Yes | N/A | N/A | | reboot | Configure the system reboot behavior that follows a successful run of the action module. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/imagebuilder/latest/userguide/toe-action-modules.html) | String | No | Allow | Allow, Force, Skip | | logOptions | Specify the options to use for MSI installation logging. Specified flags are passed to the MSI installer, along with the `/L` command line parameter to enable logging. If no flags are specified, AWSTOE uses the default value. For more information about log options for MSI, see [Command Line Options](https://learn.microsoft.com/en-us/windows/win32/msi/command-line-options) in the Microsoft *Windows Installer* product documentation. | String | No | \$1VX | i,w,e,a,r,u,c,m,o,p,v,x,\$1,\$1,\$1 | | logFile | An absolute or relative path to the log file location. If the log file path does not exist, it is created. If the log file path is not provided, AWSTOE does not store the MSI installation log. | String | No | N/A | N/A | | properties | MSI logging property key-value pairs , for example: `TARGETDIR: "C:\target\location"`   Note: Modification of the following properties is not allowed: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/imagebuilder/latest/userguide/toe-action-modules.html) | Map[String]String | No | N/A | N/A | | ignoreAuthenticodeSignatureErrors | Flag to ignore authenticode signature validation errors for the installer specified in path. The **Get-AuthenticodeSignature** command is used to validate installers. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/imagebuilder/latest/userguide/toe-action-modules.html) | Boolean | No | false | true, false | | allowUnsignedInstaller | Flag to allow running the unsigned installer specified in the path. The **Get-AuthenticodeSignature** command is used to validate installers. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/imagebuilder/latest/userguide/toe-action-modules.html) | Boolean | No | false | true, false | **Examples** The following examples show variations of the input section for your component document, depending on your installation path. **Input example: local document path installation** ``` - name: local-path-install steps: - name: LocalPathInstaller action: InstallMSI inputs: path: C:\sample.msi logFile: C:\msilogs\local-path-install.log logOptions: '*VX' reboot: Allow properties: COMPANYNAME: '"Amazon Web Services"' ignoreAuthenticodeSignatureErrors: true allowUnsignedInstaller: true ``` **Input example: Amazon S3 path installation** ``` - name: s3-path-install steps: - name: S3PathInstaller action: InstallMSI inputs: path: s3:///sample.msi logFile: s3-path-install.log reboot: Force ignoreAuthenticodeSignatureErrors: false allowUnsignedInstaller: true ``` **Input example: web path installation** ``` - name: web-path-install steps: - name: WebPathInstaller action: InstallMSI inputs: path: https:///sample.msi logFile: web-path-install.log reboot: Skip ignoreAuthenticodeSignatureErrors: true allowUnsignedInstaller: false ``` **Output** The following is an example of the output from the `InstallMSI` action module. ``` { "logFile": "web-path-install.log", "msiExitCode": 0, "stdout": "" } ``` ### UninstallMSI (Windows) The `UninstallMSI` action module allows you to remove a Windows application using an MSI file. You can specify the MSI file location using a local file path, an S3 object URI, or a web URL. The reboot option configures the reboot behavior of the system. AWSTOE generates the **msiexec** command based on the input parameters for the action module. The MSI file location (`path`) and log file location (`logFile`) are explicitly enclosed in double quotes (") while generating the **msiexec** command. The following MSI exit codes are considered successful: + 0 (Success) + 1605 (ERROR\$1UNKNOWN\$1PRODUCT) + 1614 (ERROR\$1PRODUCT\$1UNINSTALLED) + 1641 (Reboot Initiated) + 3010 (Reboot Required) **Input** | Key name | Description | Type | Required | Default value | Acceptable values | | --- | --- | --- | --- | --- | --- | | path | Specify the MSI file location using one of the following: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/imagebuilder/latest/userguide/toe-action-modules.html) Chaining expressions are allowed. | String | Yes | N/A | N/A | | reboot | Configures the system reboot behavior that follows a successful run of the action module. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/imagebuilder/latest/userguide/toe-action-modules.html) | String | No | Allow | Allow, Force, Skip | | logOptions | Specify the options to use for MSI installation logging. Specified flags are passed to the MSI installer, along with the `/L` command line parameter to enable logging. If no flags are specified, AWSTOE uses the default value. For more information about log options for MSI, see [Command Line Options](https://docs.microsoft.com/en-us/windows/win32/msi/command-line-options) in the Microsoft *Windows Installer* product documentation. | String | No | \$1VX | i,w,e,a,r,u,c,m,o,p,v,x,\$1,\$1,\$1 | | logFile | An absolute or relative path to the log file location. If the log file path does not exist, it is created. If the log file path is not provided, AWSTOE does not store the MSI installation log. | String | No | N/A | N/A | | properties | MSI logging property key-value pairs , for example: `TARGETDIR: "C:\target\location"`   Note: Modification of the following properties is not allowed: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/imagebuilder/latest/userguide/toe-action-modules.html) | Map[String]String | No | N/A | N/A | | ignoreAuthenticodeSignatureErrors | Flag to ignore authenticode signature validation errors for the installer specified in path. The **Get-AuthenticodeSignature** command is used to validate installers. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/imagebuilder/latest/userguide/toe-action-modules.html) | Boolean | No | false | true, false | | allowUnsignedInstaller | Flag to allow running the unsigned installer specified in the path. The **Get-AuthenticodeSignature** command is used to validate installers. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/imagebuilder/latest/userguide/toe-action-modules.html) | Boolean | No | false | true, false | **Examples** The following examples show variations of the input section for your component document, depending on your installation path. **Input example: remove local document path installation** ``` - name: local-path-uninstall steps: - name: LocalPathUninstaller action: UninstallMSI inputs: path: C:\sample.msi logFile: C:\msilogs\local-path-uninstall.log logOptions: '*VX' reboot: Allow properties: COMPANYNAME: '"Amazon Web Services"' ignoreAuthenticodeSignatureErrors: true allowUnsignedInstaller: true ``` **Input example: remove Amazon S3 path installation** ``` - name: s3-path-uninstall steps: - name: S3PathUninstaller action: UninstallMSI inputs: path: s3:///sample.msi logFile: s3-path-uninstall.log reboot: Force ignoreAuthenticodeSignatureErrors: false allowUnsignedInstaller: true ``` **Input example: remove web path installation** ``` - name: web-path-uninstall steps: - name: WebPathUninstaller action: UninstallMSI inputs: path: https:///sample.msi logFile: web-path-uninstall.log reboot: Skip ignoreAuthenticodeSignatureErrors: true allowUnsignedInstaller: false ``` **Output** The following is an example of the output from the `UninstallMSI` action module. ``` { "logFile": "web-path-uninstall.log", "msiExitCode": 0, "stdout": "" } ``` ## System action modules The following section describes action modules that perform system actions or update system settings. **Topics** + [Reboot (Linux, Windows)](#action-modules-reboot) + [SetRegistry (Windows)](#action-modules-setregistry) + [UpdateOS (Linux, Windows)](#action-modules-updateos) ### Reboot (Linux, Windows) The **Reboot** action module reboots the instance. It has a configurable option to delay the start of the reboot. By default, `delaySeconds` is set to `0`, which means that there is no delay. Step timeout is not supported for the Reboot action module, as it does not apply when the instance is rebooted. If the application is invoked by the Systems Manager Agent, it hands the exit code (`3010` for Windows, `194` for Linux) to the Systems Manager Agent. The Systems Manager Agent handles the system reboot as described in [Rebooting Managed Instance from Scripts](https://docs.aws.amazon.com/systems-manager/latest/userguide/send-commands-reboot.html). If the application is invoked on the host as a standalone process, it saves the current execution state, configures a post-reboot auto-run trigger to rerun the application after the reboot, and then reboots the system. **Post-reboot auto-run trigger:** + **Windows**. AWSTOE creates a Windows Task Scheduler entry with a trigger that runs automatically at `SystemStartup` + **Linux**. AWSTOE adds a job in crontab that runs automatically after the system reboots. ``` @reboot /download/path/awstoe run --document s3://bucket/key/doc.yaml ``` This trigger is cleaned up when the application starts. **Retries** By default, the maximum number of retries is set to the Systems Manager `CommandRetryLimit`. If the number of reboots exceeds the retry limit, the automation fails. You can change the limit by editing the Systems Manager agent config file (`Mds.CommandRetryLimit`). See [Runtime Configuration](https://github.com/aws/amazon-ssm-agent/blob/mainline/README.md#runtime-configuration) in the Systems Manager agent open source. To use the **Reboot** action module, for steps that contain reboot `exitcode` (for example, `3010`), you must run the application binary as `sudo user`. **Input** | Key name | Description | Type | Required | Default | | --- | --- | --- | --- | --- | | delaySeconds | Delays a specific amount of time before initiating a reboot. | Integer | No | `0` | **Input example: reboot step** ``` - name: RebootStep action: Reboot onFailure: Abort maxAttempts: 2 inputs: delaySeconds: 60 ``` **Output** None. When the **Reboot** module completes, Image Builder continues to the next step in the build. ### SetRegistry (Windows) The **SetRegistry** action module accepts a list of inputs and allows you to set the value for the specified registry key. If a registry key does not exist, it is created in the defined path. This feature applies only to Windows. **Input** | Key name | Description | Type | Required | | --- | --- | --- | --- | | path | Path of registry key. | String | Yes | | name | Name of registry key. | String | Yes | | value | Value of registry key. | String/Number/Array | Yes | | type | Value type of registry key. | String | Yes | **Supported path prefixes** + `HKEY_CLASSES_ROOT / HKCR:` + `HKEY_USERS / HKU:` + `HKEY_LOCAL_MACHINE / HKLM:` + `HKEY_CURRENT_CONFIG / HKCC:` + `HKEY_CURRENT_USER / HKCU:` **Supported types** + `BINARY` + `DWORD` + `QWORD` + `SZ` + `EXPAND_SZ` + `MULTI_SZ` **Input example: set registry key values** ``` - name: SetRegistryKeyValues action: SetRegistry maxAttempts: 3 inputs: - path: HKLM:\SOFTWARE\MySoftWare name: MyName value: FirstVersionSoftware type: SZ - path: HKEY_CURRENT_USER\Software\Test name: Version value: 1.1 type: DWORD ``` **Output** None. ### UpdateOS (Linux, Windows) The **UpdateOS** action module adds support for installing Windows and Linux updates. It installs all available updates by default. Alternatively, you can configure a list of one or more specific updates for the action module to install. You can also specify updates to exclude from the installation. If both "include" and "exclude" lists are provided, the resulting list of updates can include only those listed in the "include" list that are not listed in the "exclude" list. **Note** **UpdateOS** doesn't support Amazon Linux 2023 (AL2023). We recommend that you update your base AMI to the new version that comes with every release. For other alternatives, see [Control the updates received from major and minor releases](https://docs.aws.amazon.com/linux/al2023/ug/deterministic-upgrades.html#controlling-release-updates) in the *Amazon Linux 2023 User Guide*. + **Windows**. Updates are installed from the update source configured on the target machine. + **Linux**. The application checks for the supported package manager in the Linux platform and uses either `yum` or `apt-get` package manager. If neither are supported, an error is returned. You should have `sudo` permissions to run the **UpdateOS** action module. If you do not have `sudo` permissions an `error.Input` is returned. **Input** | Key name | Description | Type | Required | | --- | --- | --- | --- | | include | For Windows, you can specify the following: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/imagebuilder/latest/userguide/toe-action-modules.html) For Linux, you can specify one or more packages to be included in the list of updates for installation. | String List | No | | exclude | For Windows, you can specify the following: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/imagebuilder/latest/userguide/toe-action-modules.html) For Linux, you can specify one or more packages to be excluded from the list of updates for installation. | String List | No | **Input example: add support for installing Linux updates** ``` - name: UpdateMyLinux action: UpdateOS onFailure: Abort maxAttempts: 3 inputs: exclude: - ec2-hibinit-agent ``` **Input example: add support for installing Windows updates** ``` - name: UpdateWindowsOperatingSystem action: UpdateOS onFailure: Abort maxAttempts: 3 inputs: include: - KB1234567 - '*Security*' ``` **Output** None. # Configure input for the AWSTOE run command To streamline command line input for your AWSTOE **run** command, you can include settings for command parameters and options in a JSON format input configuration file with a `.json` file extension. AWSTOE can read your file from one of the following locations: + A local file path (*./config.json*). + An S3 bucket (*s3:////config.json*). When you enter the **run** command, you can specify the input configuration file using the **--config** parameter. For example: ``` awstoe run --config /config.json ``` **Input configuration file** The input configuration JSON file includes key-value pairs for all of the settings that you can provide directly through **run** command parameters and options. If you specify a setting in both the input configuration file and the **run** command, as a parameter or option, the following rules of precedence apply: **Rules of precedence** 1. A setting that is supplied directly to the **run** command in the AWS CLI, via a parameter or option, overrides any value that is defined in the input configuration file for the same setting. 1. A setting in the input configuration file overrides a component default value. 1. If no other settings are passed into the component document, it can apply a default value, if one exists. There are two exceptions to this rule – documents and parameters. These settings work differently in the input configuration and as command parameters. If you use the input configuration file, you must not specify these parameters directly to the **run** command. Doing so will generate an error. **Component settings** The input configuration file contains the following settings. To streamline your file, you can leave out any optional settings that aren't needed. All settings are optional unless otherwise noted. + **cwIgnoreFailures** (Boolean) – Ignore logging failures from the CloudWatch Logs. + **cwLogGroup** (String) – The `LogGroup` name for the CloudWatch Logs. + **cwLogRegion** (String) – The AWS Region that applies to the CloudWatch Logs. + **cwLogStream** (String) – The `LogStream` name for the CloudWatch Logs, that directs AWSTOE where to stream the `console.log` file. + **documentS3BucketOwner** (String) – The account ID of the bucket owner for S3 URI-based documents. + **documents** (array of objects, required) – An array of JSON objects representing the YAML component documents that the AWSTOE **run** command is running. At least one component document must be specified. Each object consists of the following fields: + **path** (String, required) – The file location of the YAML component document. This must be one of the following: + A local file path (*./component-doc-example.yaml*). + An S3 URI (`s3://bucket/key`). + An Image Builder component build version ARN (arn:aws:imagebuilder:us-west-*2:123456789012*:component/*my-example-component*/2021.12.02/1). + **parameters** (array of objects) – An array of key-value pair objects, each representing a component-specific parameter that the **run** command passes in when it runs the component document. Parameters are optional for components. The component document may or may not have parameters defined. Each object consists of the following fields: + **name** (String, required) – The name of the component parameter. + **value** (String, required) – The value to pass in to the component document for the named parameter. To learn more about component parameters, see the **Parameters** section in the [Use variables in your custom component document](toe-user-defined-variables.md) page. + **executonId** (String) – This is the unique ID that applies to the execution of the current **run** command. This ID is included in output and log file names, to uniquely identify those files, and link them to the current command execution. If this setting is left out, AWSTOE generates a GUID. + **logDirectory** (String) – The destination directory where AWSTOE stores all of the log files from this command execution. By default, this directory is located inside of the following parent directory: `TOE__`. If you do not specify the log directory, AWSTOE uses the current working directory (`.`). + **logS3BucketName** (String) – If component logs are stored in Amazon S3 (recommended), AWSTOE uploads the component application logs to the S3 bucket named in this parameter. + **logS3BucketOwner** (String) – If component logs are stored in Amazon S3 (recommended), this is the owner account ID for the bucket where AWSTOE writes the log files. + **logS3KeyPrefix** (String) – If component logs are stored in Amazon S3 (recommended), this is the S3 object key prefix for the log location in the bucket. + **parameters** (array of objects) – An array of key-value pair objects that represent parameters that apply globally to all of the components that are included in the current **run** command execution. + **name** (String, required) – The name of the global parameter. + **value** (String, required) – The value to pass in to all of the component documents for the named parameter. + **phases** (String) – A comma-separated list that specifies which phases to run from the YAML component documents. If a component document includes additional phases, those will not run. + **stateDirectory** (String) – The file path where state tracking files are stored. + **trace** (Boolean) – Enables verbose logging to the console. **Examples** The following example shows an input configuration file that runs the `build` and `test` phases for two component documents: `sampledoc.yaml`, and `conversation-intro.yaml`. Each component document has a parameter that applies only to itself, and both use one shared parameter. The `project` parameter applies to both component documents. ``` { "documents": [ { "path": "/awstoe/sampledoc.yaml>", "parameters": [ { "name": "dayofweek", "value": "Monday" } ] }, { "path": "/awstoe/conversation-intro.yaml>", "parameters": [ { "name": "greeting", "value": "Hello, HAL." } ] } ], "phases": "build,test", "parameters": [ { "name": "project", "value": "examples" } ], "cwLogGroup": "", "cwLogStream": "", "documentS3BucketOwner": "", "executionId": "", "logDirectory": "", "logS3BucketName": "", "logS3KeyPrefix": "", "logS3BucketOwner": "" } ```