Optimizing AWS Database Migration Service Performance with Amazon Redshift as Target

Publication date: July 15, 2022 (Document revisions)

Abstract

This whitepaper is intended for Amazon Web Services (AWS) customers who are migrating data from any supported source database to Amazon Redshift data warehouse using AWS Database Migration Service (AWS DMS). AWS DMS helps customers migrate databases to the AWS Cloud quickly and securely by replicating data from any supported source to any supported target. AWS DMS also supports continuous data capture (CDC) functionality, where it replicates data from source to target on an ongoing basis.

A data migration that is not well planned or well tested, whether it is a one-time event or ongoing, may cause issues with respect to AWS DMS source and target latencies, which can potentially delay or stall the migration efforts. This whitepaper presents information about important migration considerations, highlights the best practices to be followed during the migration process, and provides an optimal strategy for a successful database migration to the Amazon Redshift data warehouse.

Are you well-architected?

The AWS Well-Architected Framework helps you understand the pros and cons of the decisions you make when building systems in the cloud. The six pillars of the Framework allow you to learn architectural best practices for designing and operating reliable, secure, efficient, cost-effective, and sustainable systems. Using the AWS Well-Architected Tool, available at no charge in the AWS Management Console, you can review your workloads against these best practices by answering a set of questions for each pillar.

For more expert guidance and best practices for your cloud architecture—reference architecture deployments, diagrams, and whitepapers—refer to the AWS Architecture Center.

Introduction

It is no secret that customers have strained relationships with their traditional database vendors. The pattern is the same: those databases are expensive, proprietary, designed for lock-in, and come with punitive licensing terms. That’s why so many customers have accelerated their migration to AWS Cloud databases. They want the benefits of reduced capital and operational costs, increased IT staff productivity, scalability, a modern and open architecture, a pay-as-you-go model that charges only for services used, and the business value made possible by the unequaled pace of innovation at AWS.

In the past few years, hundreds of thousands of customers have migrated their databases using AWS Database Migration Service (AWS DMS). AWS DMS is a fully managed service that allows customers to migrate their relational databases, non-relational databases, and data warehouses to AWS with virtually no downtime.

As of June 2022, more than 685,544 databases have been migrated to AWS using AWS Database Migration Service. One of the common targets for AWS DMS for data migration is Amazon Redshift. Amazon Redshift is a fully managed, petabyte-scale data warehouse service in the cloud. With an Amazon Redshift database as a target, you can migrate data from all of the supported source databases.

Database migrations tend to be complex and error-prone, so AWS advocates taking a planned and well-tested approach. Various factors such as database performance, network throughput, AWS DMS resources such as memory, CPU, number of instances, Amazon Redshift queue and memory management, load on the Amazon Redshift cluster at the time of migration, and other conditions influence the migration process. The most common issues seen during the migration process are AWS DMS source and target latency issues, where the process of capturing changes from the source database or the process of applying the changes to the target is delayed due to various reasons, affecting the entire migration process.

This whitepaper provides an optimal path for AWS DMS migration with Amazon Redshift as a target. With any deployment on AWS, there are many different considerations and options, so your approach may vary slightly from the approach in this paper. The whitepaper explains the various phases involved in the migration process as well as storage and networking considerations, and provides pointers on optimizing the performance of different components in the architecture such as source database, connectivity, AWS DMS, and Amazon Redshift as the target. Lastly, the whitepaper emphasizes the importance of performing tests in your staging environments to understand how these components work under load. It includes information on the approach you can use to troubleshoot issues within your environments.