Automate the replication of Amazon RDS instances across AWS accounts
Created by Parag Nagwekar (AWS) and Arun Chandapillai (AWS)
Environment: Production | Technologies: Databases; DevOps; Serverless; Infrastructure | Workload: All other workloads |
AWS services: AWS Lambda; Amazon RDS; AWS SDK for Python (Boto3); AWS Step Functions; Amazon SNS |
Summary
This pattern shows you how to automate the process of replicating, tracking, and rolling back your Amazon Relational Database Service (Amazon RDS) DB instances across different AWS accounts by using AWS Step Functions and AWS Lambda. You can use this automation to perform large-scale replication of RDS DB instances without any performance impact or operational overhead—regardless of the size of your organization. You can also use this pattern to help your organization comply with mandatory data governance strategies or compliance requirements that call for your data to be replicated and redundant across different AWS accounts and AWS Regions. Cross-account replication of Amazon RDS data at scale is an inefficient and error-prone manual process that can be costly and time-consuming, but the automation in this pattern can help you achieve cross-account replication safely, effectively, and efficiently.
Prerequisites and limitations
Prerequisites
Two AWS accounts
An RDS DB instance, up and running in the source AWS account
A subnet group for the RDS DB instance in the destination AWS account
An AWS Key Management Service (AWS KMS) key created in the source AWS account and shared with the destination account (For more information about policy details, see the Additional information section of this pattern.)
An AWS KMS key in the destination AWS account to encrypt the database in the destination account
Product versions
Python 3.9 (using AWS Lambda)
PostgreSQL 11.3, 13.x, and 14.x
Architecture
Technology stack
Amazon Relational Database Service (Amazon RDS)
Amazon Simple Notification Service (Amazon SNS)
AWS Key Management Service (AWS KMS)
AWS Lambda
AWS Secrets Manager
AWS Step Functions
Target architecture
The following diagram shows an architecture for using Step Functions to orchestrate scheduled, on-demand replication of RDS DB instances from a source account (account A) to a destination account (account B).
![Replicating Amazon RDS DB instances across source and destination accounts by using Step Functions.](/images/prescriptive-guidance/latest/patterns/images/pattern-img/6310ad9b-1b1a-4a67-b684-ef605fef3e87/images/8a8beeb1-0c77-4a38-88e9-9c689a59b990.png)
In the source account (account A in the diagram), the Step Functions state machine performs the following:
Creates a snapshot from the RDS DB instance in account A.
Copies and encrypts the snapshot with an AWS KMS key from account A. To ensure encryption in transit, the snapshot is encrypted whether or not the DB instance is encrypted.
Shares the DB snapshot with account B by giving account B access to the snapshot.
Pushes a notification to the SNS topic, and then the SNS topic invokes the Lambda function in account B.
In the destination account (account B in the diagram), the Lambda function runs the Step Functions state machine to orchestrate the following:
Copies the shared snapshot from account A to account B, while using the AWS KMS key from account A to decrypt the data first and then encrypt the data by using the AWS KMS key in account B.
Reads the secret from Secrets Manager to capture the name of the current DB instance.
Restores the DB instance from the snapshot with a new name and default AWS KMS key for Amazon RDS.
Reads the endpoint of the new database and updates the secret in Secrets Manager with the new database endpoint, and then tags the previous DB instance so that it can be deleted later.
Keeps the latest N instances of the databases and deletes all the other instances.
Tools
AWS tools
Amazon Relational Database Service (Amazon RDS) helps you set up, operate, and scale a relational database in the AWS Cloud.
Amazon Simple Notification Service (Amazon SNS) helps you coordinate and manage the exchange of messages between publishers and clients, including web servers and email addresses.
AWS CloudFormation helps you set up AWS resources, provision them quickly and consistently, and manage them throughout their lifecycle across AWS accounts and Regions.
AWS Key Management Service (AWS KMS) helps you create and control cryptographic keys to help protect your data.
AWS Lambda is a compute service that helps you run code without needing to provision or manage servers. It runs your code only when needed and scales automatically, so you pay only for the compute time that you use.
AWS SDK for Python (Boto3)
is a software development kit that helps you integrate your Python application, library, or script with AWS services. AWS Secrets Manager helps you replace hardcoded credentials in your code, including passwords, with an API call to Secrets Manager to retrieve the secret programmatically.
AWS Step Functions is a serverless orchestration service that helps you combine Lambda functions and other AWS services to build business-critical applications.
Code
The code for this pattern is available in the GitHub Crossaccount RDS Replication
Epics
Task | Description | Skills required |
---|---|---|
Deploy the CloudFormation stack in the source account. |
| Cloud administrator, Cloud architect |
Deploy the CloudFormation stack in the destination account. |
| Cloud architect, DevOps engineer, Cloud administrator |
Verify the creation of the RDS DB instance in the destination account. |
| Cloud administrator, Cloud architect, DevOps engineer |
Subscribe the Lambda function to the SNS topic. | You must run the following AWS Command Line Interface (AWS CLI) commands to subscribe the Lambda function in the destination account (account B) to the SNS topic in the source account (account A). In account A, run following command:
In account B, run following command:
In account B, run following command:
| Cloud administrator, Cloud architect, DBA |
Sync the RDS DB instance from the source account with the destination account. | Initiate the on-demand database replication by starting the Step Functions state machine in the source account.
Note: A scheduler is in place to help you run the replication automatically on schedule, but the scheduler is turned off by default. You can find the name of the Amazon CloudWatch rule for the scheduler in the Resources tab of the CloudFormation stack in the destination account. For instructions on how to modify the CloudWatch Events rule, see Deleting or Disabling a CloudWatch Events Rule in the CloudWatch User Guide. | Cloud architect, DevOps engineer, Cloud administrator |
Roll back your database to any of the previous copies when needed. |
| Cloud administrator, DBA, DevOps engineer |
Related resources
Cross-Region read replicas (Amazon RDS User Guide)
Blue/Green Deployments (Amazon RDS User Guide)
Additional information
You can use the following example policy to share your AWS KMS key across AWS accounts.
{ "Version": "2012-10-17", "Id": "cross-account-rds-kms-key", "Statement": [ { "Sid": "Enable user permissions", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::<SourceAccount>:root" }, "Action": "kms:*", "Resource": "*" }, { "Sid": "Allow administration of the key", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::<DestinationAccount>:root" }, "Action": [ "kms:Create*", "kms:Describe*", "kms:Enable*", "kms:List*", "kms:Put*", "kms:Update*", "kms:Revoke*", "kms:Disable*", "kms:Get*", "kms:Delete*", "kms:ScheduleKeyDeletion", "kms:CancelKeyDeletion" ], "Resource": "*" }, { "Sid": "Allow use of the key", "Effect": "Allow", "Principal": { "AWS": [ "arn:aws:iam::<DestinationAccount>:root", "arn:aws:iam::<SourceAccount>:root" ] }, "Action": [ "kms:Encrypt", "kms:Decrypt", "kms:ReEncrypt*", "kms:GenerateDataKey*", "kms:DescribeKey", "kms:CreateGrant" ], "Resource": "*" } ] }