# Working with multi-account experiments for AWS FIS
You can create and manage multi-account experiment templates using the AWS FIS console or the command line. You create a multi-account experiment by specifying the account targeting experiment option as `"multi-account"`, and adding target account configurations. After you create a multi-account experiment template, you can use it to run an experiment.
With a multi-account experiment, you can set up and run real-world failure scenarios on an application that spans multiple AWS accounts within a Region. You run multi-account experiments from an *orchestrator account* that impacts resources in multiple *target accounts*.
When you run a multi-account experiment, target accounts with affected resources will be notified via their AWS Health dashboards, providing awareness to users in the target accounts. With multi-account experiments, you can:
+ Run real world failure scenarios on applications that span multiple accounts with the central controls and guardrails that AWS FIS provides.
+ Control the effects of a multi-account experiment using IAM roles with fine-grained permissions and tags to define the scope of each target.
+ Centrally view the actions AWS FIS takes in each account from the AWS Management Console and through AWS FIS logs.
+ Monitor and audit API calls AWS FIS makes in each account with AWS CloudTrail.
This section helps you get started with multi-account experiments.
**Topics**
+ [
## Concepts for multi-account experiments
](#multi-account-concepts)
+ [
## Best practices for multi-account experiments
](#best-practices)
+ [
# Prerequisites for multi-account experiments
](multi-account-prerequisites.md)
+ [
# Create a multi-account experiment template
](create.md)
+ [
# Update a target account configuration
](update.md)
+ [
# Delete a target account configuration
](delete.md)
## Concepts for multi-account experiments
The following are the key concepts for multi-account experiments:
+ **Orchestrator account** ‐ The orchestrator account acts as a central account to configure and manage the experiment in the AWS FIS Console, as well as to centralize logging. The orchestrator account owns the AWS FIS experiment template and experiment.
+ **Target accounts** ‐ A target account is an individual AWS account with resources that can be affected by an AWS FIS multi-account experiment.
+ **Target account configurations** ‐ You define the target accounts that are part of an experiment by adding target account configurations to the experiment template. A target account configuration is an element of the experiment template that is required for multi-account experiments. You define one for each target account by setting an AWS account ID, IAM role, and an optional description.
## Best practices for multi-account experiments
The following are best practices to using multi-account experiments:
+ When you configure targets for multi-account experiments, we recommend targeting with consistent resource tags across all target accounts. An AWS FIS experiment will resolve resources with consistent tags in each target account. An action must resolve at least one target resource in any target account or will fail, except for experiments with `emptyTargetResolutionMode` set to `skip`. Action quotas apply per account. If you want to target resources by resource ARNs, the same single-account limit per action applies.
+ When you target resources in one or more availability zones using parameters or filters, you should specify an *AZ ID*, not an AZ name. The AZ ID is a unique and consistent identifier for an Availability Zone across accounts. To learn how to find the AZ ID for the availability zones in your account, see [Availability Zone IDs for your AWS resources](https://docs.aws.amazon.com/ram/latest/userguide/working-with-az-ids.html).
# Prerequisites for multi-account experiments
To use stop conditions for a multi-account experiment, you must first configure cross-account alarms. IAM roles are defined when you create a multi-account experiment template. You can create the necessary IAM roles before you create the template.
**Topics**
+ [
## Permissions for multi-account experiments
](#permissions)
+ [
## Stop conditions for multi-account experiments (optional)
](#alarms-and-monitoring)
+ [
## Safety levers for multi-account experiments (optional)
](#safety-levers-for-multi-accounts)
## Permissions for multi-account experiments
Multi-account experiments use IAM *role chaining* to grant permissions to AWS FIS to take actions on resources in target accounts. For multi-account experiments, you set up IAM roles in each target account and the orchestrator account. These IAM roles require a trust relationship between the target accounts and the orchestrator account, and between the orchestrator account and AWS FIS.
The IAM roles for the target accounts contain the permissions required to take action on resources and are created for an experiment template by adding target account configurations. You will create an IAM role for the orchestrator account with permission to assume the roles of target accounts and establish a trust relationship with AWS FIS. This IAM role is used as the `roleArn` for the experiment template.
To learn more about role chaining, see [Roles Terms and concepts.](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_terms-and-concepts.html) in IAM's User Guide
In the following example, you will set up permissions for an orchestrator account A to run an experiment with `aws:ebs:pause-volume-io` in target account B.
1. In account B, create an IAM role with the permissions required to run the action. For permissions required for each action, see [AWS FIS Actions reference](fis-actions-reference.md). The following example shows the permissions a target account grants to run the EBS Pause Volume IO action [aws:ebs:pause-volume-io](fis-actions-reference.md#pause-volume-io).
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:DescribeVolumes"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"ec2:PauseVolumeIO"
],
"Resource": "arn:aws:ec2:us-east-1:123456789012:volume/*"
},
{
"Effect": "Allow",
"Action": [
"tag:GetResources"
],
"Resource": "*"
}
]
}
```
------
1. Next, add a trust policy in account B that creates a trust relationship with account A. Choose a name for the IAM role for account A, which you will create in step 3.
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "AccountIdA"
},
"Action": "sts:AssumeRole",
"Condition": {
"StringLike": {
"sts:ExternalId": "arn:aws:fis:region:accountIdA:experiment/*"
},
"ArnEquals": {
"aws:PrincipalArn": "arn:aws:iam::111122223333:role/role_name"
}
}
}
]
}
```
------
1. In account A, create an IAM role. This role name must match the role you specified in the trust policy in step 2. To target multiple accounts, you grant the orchestrator permissions to assume each role. The following example shows the permissions for account A to assume account B. If you have additional target accounts, you will add additional role ARNs to this policy. You can only have one role ARN per target account.
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Resource": [
"arn:aws:iam::111122223333:role/role_name"
]
}
]
}
```
------
1. This IAM role for account A is used as the `roleArn` for the experiment template. The following example shows the trust policy required in the IAM role that grants AWS FIS permissions to assume account A, the orchestrator account.
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": [
"fis.amazonaws.com"
]
},
"Action": "sts:AssumeRole"
}
]
}
```
------
You can also use Stacksets to provision multiple IAM roles at one time. To use CloudFormation StackSets, you will need to set up the necessary StackSet permissions in your AWS accounts. To learn more, see [Working with AWS CloudFormation StackSets](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/what-is-cfnstacksets.html).
## Stop conditions for multi-account experiments (optional)
A *stop condition* is a mechanism to stop an experiment if it reaches a threshold that you define as an alarm. To set up a stop condition for your multi-account experiment, you can use cross-account alarms. You must enable sharing in each target account to make the alarm available to the orchestrator account using read-only permissions. Once shared, you can combine metrics from different target accounts using Metric Math. Then, you can add this alarm as a stop condition for the experiment.
To learn more about cross-account dashboards, see [Enabling cross-account functionality in CloudWatch.](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Cross-Account-Cross-Region.html)
## Safety levers for multi-account experiments (optional)
Safety levers are used to stop all running experiments and prevent new experiments from starting. You may want to use the safety lever to prevent FIS experiments during certain time periods or in response to application health alarms. Every AWS account has a safety lever per AWS Region. When a safety lever is engaged, it impacts all experiments running in the same account and region as the safety lever. To stop and prevent multi-account experiments, the safety lever must be engaged in the same account and region where the experiments are running.
# Create a multi-account experiment template
**To learn how to create an experiment template via the AWS Management Console**
See [Create an experiment template](create-template.md).
**To create an experiment template using the CLI**
1. Open the AWS Command Line Interface
1. To create an experiment from a saved JSON file with the account targeting experiment option set to `"multi-account"` (for example, `my-template.json`), replace the placeholder values in *italics* with your own values, and then run the following [create-experiment-template](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/fis/create-experiment-template.html) command.
```
aws fis create-experiment-template --cli-input-json file://my-template.json
```
This will return the experiment template in the response. Copy the `id` from the response, which is the ID of the experiment template.
1. Run the [create-target-account-configuration](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/fis/create-target-account-configuration.html) command to add a target account configuration to the experiment template. Replace the placeholder values in *italics* with your own values, using the `id` from step 2 as the value for the `--experiment-template-id` parameter, and then run the following. The `--description` parameter is optional. Repeat this step for each target account.
```
aws fis create-target-account-configuration --experiment-template-id EXTxxxxxxxxx --account-id 111122223333 --role-arn arn:aws:iam::111122223333:role/role-name --description "my description"
```
1. Run the [get-target-account-configuration](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/fis/get-target-account-configuration.html) command to retrieve the details for a specific target account configuration.
```
aws fis get-target-account-configuration --experiment-template-id EXTxxxxxxxxx --account-id 111122223333
```
1. Once you have added all your target account configurations, you can run the [list-target-account-configurations](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/fis/list-target-account-configurations.html) command command to see that your target account configurations have been created.
```
aws fis list-target-account-configurations --experiment-template-id EXTxxxxxxxxx
```
You can also verify that you have added target account configurations by running the [get-experiment-template](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/fis/get-experiment-template.html) command. The template will return a read-only field `targetAccountConfigurationsCount` that is a count of all the target account configurations on the experiment template.
1. When you are ready, you can run the experiment template using the [start-experiment](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/fis/start-experiment.html) command.
```
aws fis start-experiment --experiment-template-id EXTxxxxxxxxx
```
# Update a target account configuration
You can update an existing target account configuration if you want to change the role ARN or description for the the account. When you update a target account configuration, the changes do not affect any running experiments that use the template.
**To update a target account configuration using the AWS Management Console**
1. Open the AWS FIS console at [https://console.aws.amazon.com/fis/](https://console.aws.amazon.com/fis/).
1. In the navigation pane, choose **Experiment templates**
1. Select the experiment template, and choose **Actions**, **Update experiment template**.
1. In the side panel, choose **Step 3, Configure service access**.
1. Modify the **target account configurations**, and choose **Update experiment template**.
1. Select **Step 5, Review and create**.
**To update a target account configuration using the CLI**
Run the [update-target-account-configuration](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/fis/update-target-account-configuration.html) command to command, replacing the placeholder values in *italics* with your own values. The `--role-arn` and `--description` parameters are optional, and will not be updated if not included.
```
aws fis update-target-account-configuration --experiment-template-id EXTxxxxxxxxx --account-id 111122223333 --role-arn arn:aws:iam::111122223333:role/role-name --description "my description"
```
# Delete a target account configuration
If you no longer need a target account configuration, you can delete it. When you delete a target account configuration, any running experiments that use the template are not affected. The experiment continues to run until completed or stopped.
**To delete a target account configuration using the AWS Management Console**
1. Open the AWS FIS console at [https://console.aws.amazon.com/fis/](https://console.aws.amazon.com/fis/).
1. In the navigation pane, choose **Experiment templates**.
1. Select the experiment template, and choose **Actions**, **Update**.
1. In the side panel, choose **Step 3, Configure service access**.
1. Under **Target account configurations**, select **Remove** for the target account Role ARN you want to delete.
1. Select **Step 5, Review and create**.
1. Review the template and choose **Update experiment template**. When prompted for confirmation, enter `update` and choose **Update experiment template**.
**To delete a target account configuration using the CLI**
Run the [delete-target-account-configuration](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/fis/delete-target-account-configuration.html) command, replacing the placeholder values in *italics* with your own values.
```
aws fis update-target-account-configuration --experiment-template-id EXTxxxxxxxxx --account-id 111122223333
```