# Manual set up to develop custom components with AWSTOE The AWS Task Orchestrator and Executor (AWSTOE) application is a standalone application that creates, validates, and runs commands within a component definition framework. AWS services can use AWSTOE to orchestrate workflows, install software, modify system configurations, and test image builds. Follow these steps to manually install the AWSTOE application and use it as a stand-alone application to develop custom components. Image Builder takes care of these steps for you, if you use the Image Builder console or AWS CLI commands to create custom components. For more information, see [Create custom components with Image Builder](create-component.md). # Verify the signature of the AWSTOE installation download This section describes the recommended process for verifying the validity of the installation download for AWSTOE on Linux, macOS and Windows based operating systems. **Topics** + [Verify the signature of the AWSTOE installation download on Linux or macOS](#awstoe-verify-sig-linux) + [Verify the signature of the AWSTOE installation download on Windows](#awstoe-verify-sig-win) ## Verify the signature of the AWSTOE installation download on Linux or macOS This topic describes the recommended process for verifying the validity of the installation download for the AWSTOE on Linux-based or macOS operating systems. Whenever you download an application from the internet, we recommend that you authenticate the identity of the software publisher. Also, check that the application is not altered or corrupted since it was published. This protects you from installing a version of the application that contains a virus or other malicious code. If, after running the steps in this topic, you determine that the software for the AWSTOE application is altered or corrupted, do not run the installation file. Instead, contact Support For more information about your support options, see [Support](https://aws.amazon.com/premiumsupport/). AWSTOE files for Linux-based and macOS operating systems are signed using `GnuPG`, an open source implementation of the Pretty Good Privacy (OpenPGP) standard for secure digital signatures. `GnuPG` (also known as `GPG`) provides authentication and integrity checking through a digital signature. Amazon EC2 publishes a public key and signatures that you can use to verify the downloaded Amazon EC2 CLI tools. For more information about `PGP` and `GnuPG` (`GPG`), see [http://www.gnupg.org](https://www.gnupg.org/). The first step is to establish trust with the software publisher. Download the public key of the software publisher, check that the owner of the public key is who they claim to be, and then add the public key to your *keyring*. Your keyring is a collection of known public keys. After you establish the authenticity of the public key, you can use it to verify the signature of the application. **Topics** + [Installing the GPG tools](#awstoe-verify-signature-of-binary-download-install-tools) + [Authenticating and importing the public key](#awstoe-verify-signature-of-binary-download-authenticate-import-public-key) + [Verify the signature of the package](#awstoe-verify-signature-of-binary-package) ### Installing the GPG tools If your operating system is Linux, Unix, or macOS, the GPG tools are likely already installed. To test whether the tools are installed on your system, type **gpg** at a command prompt. If the GPG tools are installed, you see a GPG command prompt. If the GPG tools are not installed, you see an error message stating that the command cannot be found. You can install the GnuPG package from a repository. To install GPG tools, select the operating system that matches your instance. ------ #### [ Debian-based Linux ] From a terminal, run the following command: ``` apt-get install gnupg ``` ------ #### [ Red Hat–based Linux ] From a terminal, run the following command: ``` yum install gnupg ``` ------ #### [ macOS ] From a terminal, run the following command: ``` brew install gnupg ``` ------ ### Authenticating and importing the public key The next step in the process is to authenticate the AWSTOE public key and add it as a trusted key in your `GPG` keyring. **To authenticate and import the AWSTOE public key** 1. Obtain a copy of our public `GPG` build key by doing one of the following: + Download the key from https://awstoe-****.s3.****.amazonaws.com/assets/awstoe.gpg. For example, [https://awstoe-us-east-1.s3.us-east-1.amazonaws.com/latest/assets/awstoe.gpg](https://awstoe-us-east-1.s3.us-east-1.amazonaws.com/latest/assets/awstoe.gpg). + Copy the key from the following text and paste it into a file called `awstoe.gpg`. Make sure to include everything that follows: ``` -----BEGIN PGP PUBLIC KEY BLOCK----- Version: GnuPG v2 mQENBF8UqwsBCACdiRF2bkZYaFSDPFC+LIkWLwFvtUCRwAHtD8KIwTJ6LVn3fHAU GhuK0ZH9mRrqRT2bq/xJjGsnF9VqTj2AJqndGJdDjz75YCZYM+ocZ+r5HSJaeW9i S5dykHj7Txti2zHe0G5+W0v7v5bPi2sPHsN7XWQ7+G2AMEPTz8PjxY//I0DvMQns Sle3l9hz6wCClz1l9LbBzTyHfSm5ucTXvNe88XX5Gmt37OCDM7vfli0Ctv8WFoLN 6jbxuA/sV71yIkPm9IYp3+GvaKeT870+sn8/JOOKE/U4sJV1ppbqmuUzDfhrZUaw 8eW8IN9A1FTIuWiZED/5L83UZuQs1S7s2PjlABEBAAG0GkFXU1RPRSA8YXdzdG9l QGFtYXpvbi5jb20+iQE5BBMBCAAjBQJfFKsLAhsDBwsJCAcDAgEGFQgCCQoLBBYC AwECHgECF4AACgkQ3r3BVvWuvFJGiwf9EVmrBR77+Qe/DUeXZJYoaFr7If/fVDZl 6V3TC6p0J0Veme7uXleRUTFOjzbh+7e5sDX19HrnPquzCnzfMiqbp4lSoeUuNdOf FcpuTCQH+M+sIEIgPno4PLl0Uj2uE1o++mxmonBl/Krk+hly8hB2L/9n/vW3L7BN OMb1Ll9PmgGPbWipcT8KRdz4SUex9TXGYzjlWb3jU3uXetdaQY1M3kVKE1siRsRN YYDtpcjmwbhjpu4xm19aFqNoAHCDctEsXJA/mkU3erwIRocPyjAZE2dnlkL9ZkFZ z9DQkcIarbCnybDM5lemBbdhXJ6hezJE/b17VA0t1fY04MoEkn6oJg== =oyze -----END PGP PUBLIC KEY BLOCK----- ``` 1. At a command prompt in the directory where you saved **awstoe.gpg**, use the following command to import the AWSTOE public key into your keyring. ``` gpg --import awstoe.gpg ``` The command returns results that are similar to the following: ``` gpg: key F5AEBC52: public key "AWSTOE " imported gpg: Total number processed: 1 gpg: imported: 1 (RSA: 1) ``` Make a note of the key value; you need it in the next step. In the preceding example, the key value is `F5AEBC52`. 1. Verify the fingerprint by running the following command, replacing *key-value* with the value from the preceding step: ``` gpg --fingerprint key-value ``` This command returns results similar to the following: ``` pub 2048R/F5AEBC52 2020-07-19 Key fingerprint = F6DD E01C 869F D639 15E5 5742 DEBD C156 F5AE BC52 uid [ unknown] AWSTOE ``` Additionally, the fingerprint string should be identical to `F6DD E01C 869F D639 15E5 5742 DEBD C156 F5AE BC52`, as shown in the preceding example. Compare the key fingerprint that is returned to the one published on this page. They should match. If they don't match, do not install the AWSTOE installation script, and contact Support. ### Verify the signature of the package After you install the `GPG` tools, authenticate and import the AWSTOE public key, and verify that the public key is trusted, you are ready to verify the signature of the installation script. **To verify the installation script signature** 1. At a command prompt, run the following command to download the application binary: ``` curl -O https://awstoe-.s3..amazonaws.com/latest/linux//awstoe ``` For example: ``` curl -O https://awstoe-us-east-1.s3.us-east-1.amazonaws.com/latest/linux/amd64/awstoe ``` Supported values for **architecture** can be `amd64`, `386`, and `arm64`. 1. At a command prompt, run the following command to download the signature file for the corresponding application binary from the same S3 key prefix path: ``` curl -O https://awstoe-.s3..amazonaws.com/latest/linux//awstoe.sig ``` For example: ``` curl -O https://awstoe-us-east-1.s3.us-east-1.amazonaws.com/latest/linux/amd64/awstoe.sig ``` Supported values for **architecture** can be `amd64`, `386`, and `arm64`. 1. Verify the signature by running the following command at a command prompt in the directory where you saved `awstoe.sig` and the AWSTOE installation file. Both files must be present. ``` gpg --verify ./awstoe.sig ~/awstoe ``` The output should look something like the following: ``` gpg: Signature made Mon 20 Jul 2020 08:54:55 AM IST using RSA key ID F5AEBC52 gpg: Good signature from "AWSTOE awstoe@amazon.com" [unknown] gpg: WARNING: This key is not certified with a trusted signature! gpg: There is no indication that the signature belongs to the owner. Primary key fingerprint: F6DD E01C 869F D639 15E5 5742 DEBD C156 F5AE BC52 ``` If the output contains the phrase `Good signature from "AWSTOE "`, it means that the signature has successfully been verified, and you can proceed to run the AWSTOE installation script. If the output includes the phrase `BAD signature`, check whether you performed the procedure correctly. If you continue to get this response, do not run the installation file that you downloaded previously, and contact Support. The following are details about the warnings that you might see: + **WARNING: This key is not certified with a trusted signature\$1 There is no indication that the signature belongs to the owner.** Ideally, you would visit an AWS office and receive the key in person. However, you would most likely download it from a website. In this case, the website is an AWS website. + **gpg: no ultimately trusted keys found.** This means that the specific key is not "ultimately trusted" by you, or by other people that you trust. For more information, see [http://www.gnupg.org](http://www.gnupg.org). ## Verify the signature of the AWSTOE installation download on Windows This topic describes the recommended process for verifying the validity of the installation file for the AWS Task Orchestrator and Executor application on Windows-based operating systems. Whenever you download an application from the internet, we recommend that you authenticate the identity of the software publisher and check that the application is not altered or corrupted since it was published. This protects you from installing a version of the application that contains a virus or other malicious code. If, after running the steps in this topic, you determine that the software for the AWSTOE application is altered or corrupted, do not run the installation file. Instead, contact Support. To verify the validity of the downloaded awstoe binary on Windows-based operating systems, make sure that the thumbprint of its Amazon Services LLC signer certificate is equal to this value: **9D CA 72 17 DA FF B8 2F E4 C4 67 77 36 2F A4 AA C9 08 82 15** **Note** During the roll-out window for a new binary, your signer certificate might not match the new thumbprint. If your signer certificate doesn't match, verify that the thumbprint value is: **BA 81 25 EE AC 64 2E A9 F3 C5 93 CA 6D 3E B7 93 7D 68 75 74** To verify this value, perform the following procedure: 1. Right-click the downloaded `awstoe.exe`, and open the **Properties** window. 1. Choose the **Digital Signatures** tab. 1. From the **Signature List**, choose **Amazon Services LLC**, and then choose **Details**. 1. Choose the **General** tab, if not already selected, and then choose **View Certificate**. 1. Choose the **Details** tab, and then choose **All** in the **Show** dropdown list, if not already selected. 1. Scroll down until you see the **Thumbprint** field and then choose **Thumbprint**. This displays the entire thumbprint value in the lower window. + If the thumbprint value in the lower window is identical to the following value: **9D CA 72 17 DA FF B8 2F E4 C4 67 77 36 2F A4 AA C9 08 82 15** then your downloaded AWSTOE binary is authentic and can be safely installed. + If the thumbprint value in the lower details window is not identical to the previous value, do not run `awstoe.exe`. **Topics** + [Verify the signature of the AWSTOE installation download](awstoe-verify-sig.md) + [Step 1: Install AWSTOE](#toe-start-install) + [Step 2: Set AWS credentials](#toe-start-credentials) + [Step 3: Develop component documents locally](#toe-start-develop) + [Step 4: Validate AWSTOE components](#toe-start-validate) + [Step 5: Run AWSTOE components](#toe-start-run) ## Step 1: Install AWSTOE To develop components locally, download and install the AWSTOE application. 1. **Download the AWSTOE application** To install AWSTOE, choose the appropriate download link for your architecture and platform. For the full list of application download links, see [AWSTOE downloads](toe-component-manager.md#toe-downloads) **Important** AWS is phasing out support for TLS versions 1.0 and 1.1. To access the S3 bucket for AWSTOE downloads, your client software must use TLS version 1.2 or later. For more information, see this [AWS Security Blog post](https://aws.amazon.com/blogs/security/tls-1-2-required-for-aws-endpoints/). 1. **Verify the signature** The steps for verifying your download depend on the server platform where you run the AWSTOE application after you install it. To verify your download on a Linux server, see [Verify the signature on Linux or macOS](awstoe-verify-sig.md#awstoe-verify-sig-linux). To verify your download on a Windows server, see [Verify the signature on Windows](awstoe-verify-sig.md#awstoe-verify-sig-win). **Note** AWSTOE is invoked directly from its download location. There is no need for a separate install step. This also means that AWSTOE can make changes to the local environment. To ensure that you isolate changes during component development, we recommend that you use an EC2 instance to develop and test AWSTOE components. ## Step 2: Set AWS credentials AWSTOE requires AWS credentials to connect to other AWS services, such as Amazon S3 and Amazon CloudWatch, when running tasks, such as: + Downloading AWSTOE documents from a user-provided Amazon S3 path. + Running `S3Download` or `S3Upload` action modules. + Streaming logs to CloudWatch, when enabled. If you are running AWSTOE on an EC2 instance, then running AWSTOE uses the same permissions as the IAM role attached to the EC2 instance. For more information about IAM roles for EC2, see [IAM roles for Amazon EC2](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html). The following examples show how to set AWS credentials using the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables. To set these variables on Linux, macOS, or Unix, use `export`. ``` export AWS_ACCESS_KEY_ID=your_access_key_id ``` ``` export AWS_SECRET_ACCESS_KEY=your_secret_access_key ``` To set these variables on Windows using PowerShell, use `$env`. ``` $env:AWS_ACCESS_KEY_ID=your_access_key_id ``` ``` $env:AWS_SECRET_ACCESS_KEY=your_secret_access_key ``` To set these variables on Windows using the command prompt, use `set`. ``` set AWS_ACCESS_KEY_ID=your_access_key_id ``` ``` set AWS_SECRET_ACCESS_KEY=your_secret_access_key ``` ## Step 3: Develop component documents locally Components are authored with plaintext YAML documents. For more information about document syntax, see [Use the AWSTOE component document framework for custom components](toe-use-documents.md). The following are example *Hello World* component documents to help you get started. ------ #### [ Linux ] Some of the Linux component examples in this guide refer to a component document file named `hello-world-linux.yml`. You can use the following document to get started with those examples. ``` name: Hello World description: This is hello world testing document for Linux. schemaVersion: 1.0 phases: - name: build steps: - name: HelloWorldStep action: ExecuteBash inputs: commands: - echo 'Hello World from the build phase.' - name: validate steps: - name: HelloWorldStep action: ExecuteBash inputs: commands: - echo 'Hello World from the validate phase.' - name: test steps: - name: HelloWorldStep action: ExecuteBash inputs: commands: - echo 'Hello World from the test phase.' ``` ------ #### [ Windows ] Some of the Windows component examples in this guide refer to a component document file named `hello-world-windows.yml`. You can use the following document to get started with those examples. ``` name: Hello World description: This is Hello World testing document for Windows. schemaVersion: 1.0 phases: - name: build steps: - name: HelloWorldStep action: ExecutePowerShell inputs: commands: - Write-Host 'Hello World from the build phase.' - name: validate steps: - name: HelloWorldStep action: ExecutePowerShell inputs: commands: - Write-Host 'Hello World from the validate phase.' - name: test steps: - name: HelloWorldStep action: ExecutePowerShell inputs: commands: - Write-Host 'Hello World from the test phase.' ``` ------ #### [ macOS ] Some of the macOS component examples in this guide refer to a component document file named `hello-world-macos.yml`. You can use the following document to get started with those examples. ``` name: Hello World description: This is hello world testing document for macOS. schemaVersion: 1.0 phases: - name: build steps: - name: HelloWorldStep action: ExecuteBash inputs: commands: - echo 'Hello World from the build phase.' - name: validate steps: - name: HelloWorldStep action: ExecuteBash inputs: commands: - echo 'Hello World from the validate phase.' - name: test steps: - name: HelloWorldStep action: ExecuteBash inputs: commands: - echo 'Hello World from the test phase.' ``` ------ ## Step 4: Validate AWSTOE components You can validate the syntax of AWSTOE components locally with the AWSTOE application. The following examples show the AWSTOE application `validate` command to validate the syntax of a component without running it. **Note** The AWSTOE application can validate only the component syntax for the current operating system. For example, when running `awstoe.exe` on Windows, you cannot validate the syntax for a Linux document that uses the `ExecuteBash` action module. Linux or macOS ``` awstoe validate --documents /home/user/hello-world.yml ``` Windows ``` awstoe.exe validate --documents C:\Users\user\Documents\hello-world.yml ``` ## Step 5: Run AWSTOE components The AWSTOE application can run one or more phases of specified documents using the `--phases` command line argument. Supported values for `--phases` are `build`, `validate`, and `test`. Multiple phase values can be entered as comma separated values. When you provide a list of phases, the AWSTOE application sequentially runs the specified phases of each document. For example, AWSTOE runs the `build` and `validate` phases of `document1.yaml`, followed by the `build` and `validate` phases of `document2.yaml`. To ensure that your logs are stored securely and retained for troubleshooting, we recommend configuring log storage in Amazon S3. In Image Builder, the Amazon S3 location for publishing logs is specified in the infrastructure configuration. For more information about infrastructure configuration, see [Manage Image Builder infrastructure configuration](manage-infra-config.md) If a list of phases is not provided, the AWSTOE application runs all phases in the order listed in the YAML document. To run specific phases in single or multiple documents, use the following commands. Single phase ``` awstoe run --documents hello-world.yml --phases build ``` Multiple phases ``` awstoe run --documents hello-world.yml --phases build,test ``` **Document run** Run all phases in a single document ``` awstoe run --documents documentName.yaml ``` Run all phases in multiple documents ``` awstoe run --documents documentName1.yaml,documentName2.yaml ``` Enter Amazon S3 information to upload AWSTOE logs from a user-defined local path (recommended) ``` awstoe run --documents documentName.yaml --log-s3-bucket-name amzn-s3-demo-destination-bucket --log-s3-key-prefix S3KeyPrefix --log-s3-bucket-owner S3BucketOwner --log-directory local_path ``` Run all phases in a single document, and display all logs on the console ``` awstoe run --documents documentName.yaml --trace ``` Example command ``` awstoe run --documents s3://bucket/key/doc.yaml --phases build,validate ``` Run document with unique ID ``` awstoe run --documents documentName.yaml --execution-id user-provided-id --phases build,test ``` Get help with AWSTOE ``` awstoe --help ```