Summary Prerequisites and limitations Architecture Tools Best practices Epics Troubleshooting Related resources Additional information

Automate stack set deployment by using AWS CodePipeline and AWS CodeBuild

Created by Thiyagarajan Mani (AWS), Mihir Borkar (AWS), and Raghu Gowda (AWS)

Code repository: automated-code-pipeline-stackset-deployment	Environment: Production	Technologies: DevOps; DevelopmentAndTesting
AWS services: AWS CodeBuild; AWS CodeCommit; AWS CodePipeline; AWS Organizations; AWS CloudFormation

Summary

In your continuous integration and continuous delivery (CI/CD) processes, you might want to deploy applications automatically into all your existing AWS accounts and into new accounts that you add to your organization in AWS Organizations. When you architect a CI/CD solution for this requirement, the delegated stack set administrator capability of AWS CloudFormation is useful because it enables a layer of security by restricting access to the management account. However, AWS CodePipeline uses the service-managed permissions model to deploy applications into multiple acounts and Regions. You must use the AWS Organizations management account to deploy with stack sets, because AWS CodePipeline doesn’t support the delegated stack set administrator feature.

This pattern describes how you can work around this limitation. The pattern uses AWS CodeBuild and a custom script to automate stack set deployment with AWS CodePipeline. It automates these application deployment activities:

Deploy an application as stack sets into existing organizational units (OUs)
Extend the deployment of an application into additional OUs and Regions
Remove a deployed application from all or specific OUs or Regions

Prerequisites and limitations

Prerequisites

Before you follow the steps in this pattern:

Create organizations in your AWS Organizations management account. For instructions, see the AWS Organizations documentation.
Enable trusted access between AWS Organizations and CloudFormation to use service-managed permissions. For instructions, see Enable trusted access with AWS Organizations in the CloudFormation documentation.

Limitations

The code that’s supplied with this pattern has the following limitations:

You can deploy only a single CloudFormation template for an application; multiple template deployment isn’t currently supported.
Customizing the current implementation requires DevOps expertise.
This pattern doesn’t use AWS Key Management System (AWS KMS) keys. However, you can enable this functionality by reconfiguring the CloudFormation template included with this pattern.

Architecture

This architecture for the CI/CD deployment pipeline handles the following:

Restricts direct access to the management account by delegating stack set deployment responsibility to a dedicated CI/CD account as the stack set administrator for application deployments.
Uses the service-managed permission model to deploy the application automatically whenever a new account is created and mapped under an OU.
Ensures application version consistency across all accounts at the environment level.
Uses multiple approval stages at the repository and pipeline levels to provide additional layers of security and governance for the deployed application.
Overcomes the current limitation of CodePipeline by using a custom-built deployment script in CodeBuild to automatically deploy or remove stack sets and stack instances. For an illustration of the flow control and hierarchy of API calls implemented by the custom script, see the Additional information section.
Creates individual stack sets for the development, testing, and production environments. In addition, you can create stack sets that combine multiple OUs and Regions at every stage. For example, you can combine sandbox and development OUs within a development deployment stage.
Supports application deployment into, or exclusion from, a subset of accounts or list of OUs.

Automation and scale

You can use the code provided with this pattern to create a AWS CodeCommit repository and a code pipeline for your application. You can then deploy these as stack sets into multiple accounts at the OU level. The code also automates components such as Amazon Simple Notification Service (Amazon SNS) topics to notify approvers, the required AWS Identity and Access Management (IAM) roles, and the service control policy (SCP) to apply in the management account.

Tools

AWS services

AWS CloudFormation helps you set up AWS resources, provision them quickly and consistently, and manage them throughout their lifecycle across AWS accounts and Regions.
AWS CodeBuild is a fully managed build service that helps you compile source code, run unit tests, and produce artifacts that are ready to deploy.
AWS CodeCommit is a version control service that helps you privately store and manage Git repositories, without needing to manage your own source control system.
AWS CodeDeploy automates deployments to Amazon Elastic Compute Cloud (Amazon EC2) or on-premises instances, AWS Lambda functions, or Amazon Elastic Container Service (Amazon ECS) services.
AWS CodePipeline helps you quickly model and configure the different stages of a software release and automate the steps required to release software changes continuously.
AWS Organizations is an account management service that helps you consolidate multiple AWS accounts into an organization that you create and centrally manage.
Amazon Simple Notification Service (Amazon SNS) helps you coordinate and manage the exchange of messages between publishers and clients, including web servers and email addresses.

Code repository

The code for this pattern is available in the GitHub automated-code-pipeline-stackset-deployment repository. For the folder structure and other details, see the readme file for the repository.

Best practices

This pattern restricts direct access to the management account while deploying the application at the OU level. Adding multiple approval stages to the pipeline and repository process helps provide additional security and governance for the applications and components that you deploy by using this approach.

Epics

Task	Description	Skills required
Enable all features in the management account.	Enable all features in the management account for your organization by following the instructions in the AWS Organizations documentation.	AWS administrator, Platform administrator
Create a CI/CD account.	In AWS Organizations, in your organization, create a dedicated CI/CD account, and assign a team to own and control access to the account.	AWS administrator
Add a delegated administrator.	In the management account, register the CI/CD account that you created in the previous step as a delegated stack set administrator. For instructions, see the AWS CloudFormation documentation.	AWS administrator, Platform administrator

Task	Description	Skills required
Clone the code repository.	Clone the code repository that’s provided with this pattern to your computer: `git clone https://github.com/aws-samples/automated-code-pipeline-stackset-deployment.git` Review the readme file to understand the directory structure and other details.	AWS DevOps
Create SNS topics.	You can use the `sns-template.yaml` template that’s provided in the GitHub repository to create SNS topics and configure subscriptions for approval requests. On the AWS console, sign in to the CI/CD account. Open the CloudFormation console at at https://console.aws.amazon.com/cloudformation. Create a new stack with new resources (standard option). For Specify template, choose Upload a template file, Choose file, and then select the `sns-template.yaml` file from the `templates` folder of the cloned GitHub repository. Choose Next. Provide a meaningful application stack name. Specify a prefix for resources. Choose Next, Next, and Submit. When the stack has been created successfully, choose the Outputs tab, and note the Amazon Resource Names (ARNs) of the SNS topics for pull requests, the test environment, and the production environment. You’ll use this information in subsequent steps.	AWS DevOps
Create IAM roles for CI/CD components.	You can use the `cicd-role-template.yaml` template that’s provided in the GitHub repository to create IAM roles and policies required by CI/CD components. On the AWS console, sign in to the CI/CD account. Open the CloudFormation console at at https://console.aws.amazon.com/cloudformation. Create a new stack with new resources (standard option). For Specify template, choose Upload a template file, Choose file, and then select the `cicd-role-template.yaml` file from the `templates` folder of the cloned GitHub repository. Choose Next. Provide a meaningful application stack name. Enter values for the following parameters: The ARN for the permission boundary policy. You can obtain this ARN from the Policy details section of your permissions boundary policy on the IAM console. The ARN for the SNS production approval topic that you noted previously. The ARN for the SNS test approval topic that you noted previously. A prefix for resources created by the template. Choose Next, Next, and Submit. When the stack has been created successfully, choose the Outputs tab, and note the ARNs of the IAM roles that were created. You’ll use this information in subsequent steps.	AWS DevOps
Create a CodeCommit repository and a code pipeline for your application.	You can use the `cicd-pipeline-template.yaml` template that’s provided in the GitHub repository to create a CodeCommit repository and a code pipeline for your application. On the AWS console, sign in to the CI/CD account. Open the CloudFormation console at at https://console.aws.amazon.com/cloudformation. Create a new stack with new resources (standard option). For Specify template, choose Upload a template file, Choose file, and then select the `cicd-pipeline-template.yaml` file from the `templates` folder of the cloned GitHub repository. Choose Next. Provide a meaningful application stack name. Enter values for the following parameters: AppRepositoryName – The name of the CodeCommit repository that will be created for the application. AppRepositoryDescription – A brief description of the CodeCommit repository that will be created for the application. ApplicationName – The name of your application. This string is used as the name of the CodeCommit repository and as the prefix of the CI/CD pipeline. CloudWatchEventRoleARN – The ARN of the CloudWatch event role from the previous task. CodeBuildProjectRoleARN – The ARN of the CodeBuild project role from the previous task. CodePipelineRoleARN – The ARN of the CodePipeline role from the previous task. DeploymentConfigBucket – The Amazon Simple Storage Service (Amazon S3) bucket name where the deployment configuration files and script .zip file will be stored. DeploymentConfigKey – The path and .zip filename (Amazon S3 key). PRApprovalSNSARN – The ARN for the SNS topic for pull request notifications. ProdApprovalSNSARN – The ARN for the SNS topic for production approvals. TESTApprovalSNSARN – The ARN for the SNS topic for test approvals. TemplateBucket – The name of the S3 bucket in the CI/CD account where the CI/CD pipeline creation template will be stored. Choose Next, Next, and Submit. When the stack completes successfully, it creates a CodeCommit repository that has the specified name and a default directory structure, deployment configuration files, scripts, and a code pipeline for the repository.	AWS DevOps

Task	Description	Skills required
Clone the application repository.	The CI/CD pipeline template you used previously creates a sample application repository and code pipeline. To clone and verify the repository: Sign in to the CI/CD account. Find the application repository and CI/CD pipeline that you created in the previous epic. Copy the URL for the repository and use the git clone command to clone the repository on your local machine. Verify that the directory structure and files match the following: `root \|- deploy_configs \| \|- deployment_config.json \|- parameters \| \|- template-parameter-dev.json \| \|- template-parameter-test.json \| \|- template-parameter-prod.json \|- templates \| \|- template.yml \|- buildspec.yml` where the `deploy_configs` folder contains the deployment configuration file, and the `templates` and `parameters` folders include default files that you’ll replace with your own CloudFormation template and parameter files. Important: Do not customize the folder structure. Create a feature branch.	App developer, Data engineer
Add application artifacts.	Update the application repository by using a CloudFormation template. Note: This solution supports the deployment of only a single CloudFormation template. Build your CloudFormation template for deploying your application code changes, and name it `<application-name>.yaml`. Replace the `template.yml` file in the `templates` folder of the application repository with the CloudFormation template you created in step 1. Prepare parameter files for each environment (development, testing, and production). Name the parameter files by using the format `<cloudformation-template-name>-parameter-<environment-name>.json`. Replace the default parameter files in the `parameters` folder with your files from step 4.	App developer, Data engineer
Update the deployment configuration file.	Update the `deployment_config.json` file: In the application repository, navigate to the `deploy_configs` folder. Open the file `deployment_config.json`: { "deployment_action": "<deploy/delete>", "stack_set_name": "<stack set name>", "stack_set_desciption": "<stack set description>", "deployment_targets": { "dev": { "org_units": ["list of OUs"], "regions": ["list of regions"], "filter_accounts": ["list of accounts"], "filter_type": "<DIFFERENCE/INTERSECTION/UNION>" }, "test": { "org_units": ["list of OUs"], "regions": ["list of regions"], "filter_accounts": ["list of accounts"], "filter_type": "<DIFFERENCE/INTERSECTION/UNION>" }, "prod": { "org_units": ["list of OUs"], "regions": ["list of regions"], "filter_accounts": ["list of accounts"], "filter_type": "<DIFFERENCE/INTERSECTION/UNION>" } }, "cft_capabilities": ["CAPABILITY_IAM", "CAPABILITY_NAMED_IAM"], "auto_deployement": "<True/False>", "retain_stacks_on_account_removal": "<True/False>", "region_deployment_concurrency": "<SEQUENTIAL/PARALLEL>" } Update the values for deployment action, stack set name, stack set description, and deployment targets. For example, you can set `deployment_action` to `delete` to delete the entire stack set and its associated stack instances. Use `deploy` to create a new stack set, to update an existing stack set, or to add or remove stack instances for additional OUs or Regions. For more examples, see the Additional information section. This pattern creates individual stack sets for each environment by adding the environment name to the stack set name you provide in the deployment configuration file.	App developer, Data engineer
Commit changes and deploy the stack set.	Commit the changes you specified in your application template, and merge and deploy the stack set into multiple environments stage by stage: Save all your files and commit changes to the feature branch of your local application repository. Push the feature branch to the remote repository. Create a pull request to merge the changes to the main branch. When the pull request has been approved and changes have been merged to the main branch, the CI/CD pipeline will be initiated. When the Dev-deployment stage has completed successfully, check the CloudFormation console, StackSets, Service-Managed tab. You’ll see a new stack set with the suffix `dev`. Check the CodeBuild logs for the Dev-deployment stage for any issues. Deploy the stack set into the testing and production environments by asking your approvers to approve the deployments for those stages and repeating steps 5 and 6. The stack sets for the testing and production environments have the suffixes `test` and `prod`.	App developer, Data engineer

Troubleshooting

Issue	Solution
Deployment fails with the exception: Change the Name of Template Parameter file as <application name>-parameter-<evn>.json with, default names are not allowed	The CloudFormation template parameter files must follow the naming convention specified. Update the parameter file names and try again.
Deployment fails with the exception: Change the Name of CloudFormation Template as <application name>.yml, default template.yml or template.yaml are not allowed	The CloudFormation template name must follow the naming convention specified. Update the file name and try again.
Deployment fails with the exception: No valid CloudFormation Template and its Parameter File found for {environment name} environment	Check the file naming conventions for the CloudFormation template and its parameter file for the specified environment.
Deployment fails with the exception: Invalid deployment action provided in deployment config file. Valid options are 'deploy' and 'delete'.	You specified an invalid value for the `deployment_action` parameter in the deployment configuration file. The parameter has two valid values: `deploy` and `delete`. Use `deploy` to create and update the stack sets and their associated stack instances. Use `delete` only when you want to remove the entire stack set and associated stack instances.

Related resources

GitHub automated-code-pipeline-stackset-deployment repository
Enabling all features in your organization (AWS Organizations documentation)
Register a delegated administrator (AWS CloudFormation documentation)
Account level targets for service-managed Stack Sets (AWS CloudFormation documentation)

Additional information

Flow chart

The following flow chart depicts the flow control and hierarchy of API calls implemented by the custom script to automate stack set deployment.

Flow control and API calls implemented by Python script

Sample deployment configuration files

Creating a new stack set

The following deployment configuration file creates a new stack set called sample-stack-set in the AWS Region us-east-1 in three OUs.


{
     "deployment_action": "deploy",
     "stack_set_name": "sample-stack-set",
     "stack_set_desciption": "this is a sample stack set",
    "deployment_targets": {
                            "dev": {
                                        "org_units": ["dev-org-unit-1"], 
                                        "regions": ["us-east-1"],
                                        "filter_accounts": [],
                                        "filter_type": ""
                                    },
                            "test": {
                                        "org_units": ["test-org-unit-1"], 
                                        "regions": ["us-east-1"],
                                        "filter_accounts": [],
                                        "filter_type": ""
                                    },                            
                            "prod": {
                                        "org_units": ["prod-org-unit-1"], 
                                        "regions": ["us-east-1"],
                                        "filter_accounts": [],
                                        "filter_type": ""
                                    }                            
                          },
     "cft_capabilities": ["CAPABILITY_IAM", "CAPABILITY_NAMED_IAM"],
     "auto_deployement": "True",
     "retain_stacks_on_account_removal": "True",
     "region_deployment_concurrency": "PARALLEL"
 }

Deploying an existing stack set to another OU

If you deploy the configuration shown in the previous example and you want to deploy the stack set to an additional OU called dev-org-unit-2 in the development environment, the deployment configuration file might look like the following.


{
     "deployment_action": "deploy",
     "stack_set_name": "sample-stack-set",
     "stack_set_desciption": "this is a sample stack set",
    "deployment_targets": {
                            "dev": {
                                        "org_units": ["dev-org-unit-1", "dev-org-unit-2"], 
                                        "regions": ["us-east-1"],
                                        "filter_accounts": [],
                                        "filter_type": ""
                                    },
                            "test": {
                                        "org_units": ["test-org-unit-1"], 
                                        "regions": ["us-east-1"],
                                        "filter_accounts": [],
                                        "filter_type": ""
                                    },                            
                            "prod": {
                                        "org_units": ["prod-org-unit-1"], 
                                        "regions": ["us-east-1"],
                                        "filter_accounts": [],
                                        "filter_type": ""
                                    }                            
                          },
     "cft_capabilities": ["CAPABILITY_IAM", "CAPABILITY_NAMED_IAM"],
     "auto_deployement": "True",
     "retain_stacks_on_account_removal": "True",
     "region_deployment_concurrency": "PARALLEL"
 }

Deploying an existing stack set to another AWS Region

If you deploy the configuration shown in the previous example and you want to deploy the stack set to an additional AWS Region (us-east-2) in the development environment for two OUs (dev-org-unit-1 and dev-org-unit-2), the deployment configuration file might look like the following.

Note: The resources in the CloudFormation template must be valid and Region-specific.


{
     "deployment_action": "deploy",
     "stack_set_name": "sample-stack-set",
     "stack_set_desciption": "this is a sample stack set",
    "deployment_targets": {
                            "dev": {
                                        "org_units": ["dev-org-unit-1", "dev-org-unit-2"], 
                                        "regions": ["us-east-1", "us-east-2"],
                                        "filter_accounts": [],
                                        "filter_type": ""
                                    },
                            "test": {
                                        "org_units": ["test-org-unit-1"], 
                                        "regions": ["us-east-1"],
                                        "filter_accounts": [],
                                        "filter_type": ""
                                    },                            
                            "prod": {
                                        "org_units": ["prod-org-unit-1"], 
                                        "regions": ["us-east-1"],
                                        "filter_accounts": [],
                                        "filter_type": ""
                                    }                            
                          },
     "cft_capabilities": ["CAPABILITY_IAM", "CAPABILITY_NAMED_IAM"],
     "auto_deployement": "True",
     "retain_stacks_on_account_removal": "True",
     "region_deployment_concurrency": "PARALLEL"
 }

Removing a stack instance from an OU or AWS Region

Let’s say that the deployment configuration shown in the previous example has been deployed. The following configuration file removes the stack instances from both Regions of the OU dev-org-unit-2.


{
     "deployment_action": "deploy",
     "stack_set_name": "sample-stack-set",
     "stack_set_desciption": "this is a sample stack set",
    "deployment_targets": {
                            "dev": {
                                        "org_units": ["dev-org-unit-1"],
                                        "regions": ["us-east-1", "us-east-2"],
                                        "filter_accounts": [],
                                        "filter_type": ""
                                    },
                            "test": {
                                        "org_units": ["test-org-unit-1"],
                                        "regions": ["us-east-1"],
                                        "filter_accounts": [],
                                        "filter_type": ""
                                    },                            
                            "prod": {
                                        "org_units": ["prod-org-unit-1"],
                                        "regions": ["us-east-1"],
                                        "filter_accounts": [],
                                        "filter_type": ""
                                    }                            
                          },
     "cft_capabilities": ["CAPABILITY_IAM", "CAPABILITY_NAMED_IAM"],
     "auto_deployement": "True",
     "retain_stacks_on_account_removal": "True",
     "region_deployment_concurrency": "PARALLEL"
 }

The following configuration file removes the stack instance from the AWS Region us-east-1 for both OUs in the development environment.


{
     "deployment_action": "deploy",
     "stack_set_name": "sample-stack-set",
     "stack_set_desciption": "this is a sample stack set",
    "deployment_targets": {
                            "dev": {
                                        "org_units": ["dev-org-unit-1", "dev-org-unit-2"], 
                                        "regions": ["us-east-2"],
                                        "filter_accounts": [],
                                        "filter_type": ""
                                    },
                            "test": {
                                        "org_units": ["test-org-unit-1"], 
                                        "regions": ["us-east-1"],
                                        "filter_accounts": [],
                                        "filter_type": ""
                                    },                            
                            "prod": {
                                        "org_units": ["prod-org-unit-1"], 
                                        "regions": ["us-east-1"],
                                        "filter_accounts": [],
                                        "filter_type": ""
                                    }                            
                          },
     "cft_capabilities": ["CAPABILITY_IAM", "CAPABILITY_NAMED_IAM"],
     "auto_deployement": "True",
     "retain_stacks_on_account_removal": "True",
     "region_deployment_concurrency": "PARALLEL"
 }

Deleting the entire stack set

The following deployment configuration file deletes the entire stack set and all its associated stack instances.


{
     "deployment_action": “delete”,
     "stack_set_name": "sample-stack-set",
     "stack_set_desciption": "this is a sample stack set",
    "deployment_targets": {
                            "dev": {
                                        "org_units": ["dev-org-unit-1", "dev-org-unit-2"], 
                                        "regions": ["us-east-2"],
                                        "filter_accounts": [],
                                        "filter_type": ""
                                    },
                            "test": {
                                        "org_units": ["test-org-unit-1"], 
                                        "regions": ["us-east-1"],
                                        "filter_accounts": [],
                                        "filter_type": ""
                                    },                            
                            "prod": {
                                        "org_units": ["prod-org-unit-1"], 
                                        "regions": ["us-east-1"],
                                        "filter_accounts": [],
                                        "filter_type": ""
                                    }                            
                          },
     "cft_capabilities": ["CAPABILITY_IAM", "CAPABILITY_NAMED_IAM"],
     "auto_deployement": "True",
     "retain_stacks_on_account_removal": "True",
     "region_deployment_concurrency": "PARALLEL"
 }

Excluding an account from deployment

The following deployment configuration file excludes the account 111122223333, which is part of the OU dev-org-unit-1, from deployment.


 {
     "deployment_action": "deploy",
     "stack_set_name": "sample-stack-set",
     "stack_set_desciption": "this is a sample stack set",
    "deployment_targets": {
                            "dev": {
                                        "org_units": ["dev-org-unit-1"], 
                                        "regions": ["us-east-1"],
                                        "filter_accounts": ["111122223333"],
                                        "filter_type": "DIFFERENCE"
                                    },
                            "test": {
                                        "org_units": ["test-org-unit-1"], 
                                        "regions": ["us-east-1"],
                                        "filter_accounts": [],
                                        "filter_type": ""
                                    },                            
                            "prod": {
                                        "org_units": ["prod-org-unit-1"], 
                                        "regions": ["us-east-1"],
                                        "filter_accounts": [],
                                        "filter_type": ""
                                    }                            
                          },
     "cft_capabilities": ["CAPABILITY_IAM", "CAPABILITY_NAMED_IAM"],
     "auto_deployement": "True",
     "retain_stacks_on_account_removal": "True",
     "region_deployment_concurrency": "PARALLEL"
 }

Deploying the application to a subset of accounts in an OU

The following deployment configuration file deploys the application to only three accounts (111122223333, 444455556666, and 777788889999) in the OU dev-org-unit-1.


 {
     "deployment_action": "deploy",
     "stack_set_name": "sample-stack-set",
     "stack_set_desciption": "this is a sample stack set",
    "deployment_targets": {
                            "dev": {
                                        "org_units": ["dev-org-unit-1"], 
                                        "regions": ["us-east-1"],
                                        "filter_accounts": ["111122223333", "444455556666", "777788889999"],
                                        "filter_type": "INTERSECTION"
                                    },
                            "test": {
                                        "org_units": ["test-org-unit-1"], 
                                        "regions": ["us-east-1"],
                                        "filter_accounts": [],
                                        "filter_type": ""
                                    },                            
                            "prod": {
                                        "org_units": ["prod-org-unit-1"], 
                                        "regions": ["us-east-1"],
                                        "filter_accounts": [],
                                        "filter_type": ""
                                    }                            
                          },
     "cft_capabilities": ["CAPABILITY_IAM", "CAPABILITY_NAMED_IAM"],
     "auto_deployement": "True",
     "retain_stacks_on_account_removal": "True",
     "region_deployment_concurrency": "PARALLEL"
 }

Warning Javascript is disabled or is unavailable in your browser.

To use the Amazon Web Services Documentation, Javascript must be enabled. Please refer to your browser's Help pages for instructions.

Document Conventions

Automate backups from AWS CodeCommit to Amazon S3

Automatically attach a managed policy for Systems Manager to EC2 instance profiles