# Auto migrating EC2 databases to Amazon RDS using AWS Database Migration Service
You can use the RDS console to migrate an EC2 database to RDS. RDS uses AWS Database Migration Service (AWS DMS) to migrate your source EC2 database. AWS DMS allows you to migrate relational databases into your AWS Cloud. For more information about AWS Database Migration Service, see [What is AWS Database Migration Service?](https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html) in the *AWS Database Migration Service User Guide*.
To begin the migration, you must create an equivalent RDS DB instance to migrate the data into. After you create your target database, you can import your EC2 database into it. For source databases smaller than 1TiB, this migration action reduces the time and resources required to migrate your data into RDS.
## Overview
The RDS console allows you to migrate EC2 databases into equivalent RDS databases. You must create an RDS database to enable migration from the console.
You can migrate EC2 databases for the following databases engines:
+ MySQL
+ MariaDB
+ PostgreSQL
The migration process involves the following steps:
+ Create an equivalent database in RDS. For the databases to be equivalent, they must have the same database engine and compatible engine versions. They must also be in the same VPC. For instructions on creating your database, see [Creating an Amazon RDS DB instance](USER_CreateDBInstance.md).
+ Choose the type of replication for your database:
+ **Full load migration** – RDS copies the complete source database to the target database, creating new tables in the target when necessary.
**Note**
This option causes an outage in your RDS database.
+ **Full load and change data capture (CDC) migration** – Similar to full load migration, with this option, RDS copies over the complete source database to the target database. However, after the full load migration, RDS applies any captured changes in the source to the target database. Change data capture collects changes to the database logs by using the database engine's native API.
**Note**
This option causes an outage in your RDS database.
+ **Change data capture (CDC)** – Use this option to keep your target database available through the migration. RDS migrates ongoing changes in your source database to the target database.
+ RDS creates the necessary networking resources to facilitate the migration. Once RDS creates the required resources, it notifies you about the resources created and allows you to initiate the data transfer.
The time required to complete the migration depends on the type of replication and the size of the source database.
## Prerequisites
### MySQL and MariaDB
Before you begin to work with a MySQL or MariaDB database as the source database, make sure that you have the following prerequisites. These prerequisites apply to AWS-managed sources.
You must have an account for AWS DMS that has the Replication Admin role. The role needs the following privileges:
+ **REPLICATION CLIENT** – This privilege is required for CDC tasks only. In other words, full-load-only tasks don't require this privilege.
+ **REPLICATION SLAVE** – This privilege is required for CDC tasks only. In other words, full-load-only tasks don't require this privilege.
The AWS DMS user must also have SELECT privileges for the source tables designated for replication.
Grant the following privileges if you use MySQL-specific premigration assessments.
```
grant select on mysql.user to ;
grant select on mysql.db to ;
grant select on mysql.tables_priv to ;
grant select on mysql.role_edges to #only for MySQL version 8.0.11 and higher
```
### PostgreSQL
Before migrating data from an AWS-managed PostgreSQL source database, do the following:
+ We recommend that you use an AWS user account with the minimum required permissions for the PostgreSQL DB instance as the user account for the PostgreSQL source endpoint for AWS DMS. Using the master account is not recommended. The account must have the `rds_superuser` role and the `rds_replication` role. The `rds_replication` role grants permissions to manage logical slots and to stream data using logical slots.
**Note**
Some AWS DMS transactions are idle for some time before the DMS engine uses them again. By using the parameter `idle_in_transaction_session_timeout` in PostgreSQL versions 9.6 and higher, you can cause idle transactions to time out and fail.
## Limitations
The following limitations apply to the auto-migrate process:
+ Your target database status must be **Available** to begin source database migration.
+ When migrating from a MySQL source database, your RDS account must have the Replication Admin role. You must also have the proper privileges applied for that role.
+ Your EC2 instance and target database must be in the same VPC.
+ You can't migrate your EC2 database to the following target databases when using the **Migrate data from EC2 database** action:
+ Database that is a member of a cluster
+ Oracle, SQL Server, and Db2 databases
+ Databases with MySQL version lower than 5.7
+ Databases with PostgreSQL version lower than 10.4
+ Databases with MariaDB version lower than 10.2
# Creating IAM resources for homogeneous migrations
RDS uses AWS DMS to migrate your data. To access your databases and to migrate data, AWS DMS creates a serverless environment for homogeneous data migrations. In this environment, AWS DMS requires access to VPC peering, route tables, security groups, and other AWS resources. Also, AWS DMS stores logs, metrics, and progress for each data migration in Amazon CloudWatch. To create a data migration project, AWS DMS needs access to these services.
Also, AWS DMS requires access to the secrets that respresent a set of user credentials to authenticate the database connection for the source and target connection.
**Note**
By using the **Migrate data from EC2 instance** action, you can use the RDS console to generate these IAM resources. Skip this step if you use the console generated IAM resources.
You need the following IAM resources for this process:
**Topics**
+ [Creating an IAM policy for homogeneous data migrations](#USER_DMS_migration-IAM.iam-policy)
+ [Creating an IAM role for homogeneous data migrations](#USER_DMS_migration-IAM.iam-role)
+ [Creating a secret access policy and role](USER_DMS_migration-IAM.secret-iam-role-policy.md)
+ [Creating an IAM role for AWS DMS to manage Amazon VPC](USER_DMS_migration-IAM.dms-vpc-role.md)
## Creating an IAM policy for homogeneous data migrations
In this step, you create an IAM policy that provides AWS DMS with access to Amazon EC2 and CloudWatch resources. Next, create an IAM role and attach this policy.
**To create an IAM policy for data migration**
1. Sign in to the AWS Management Console and open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/).
1. In the navigation pane, choose **Policies**.
1. Choose **Create policy**.
1. In the **Create policy** page, choose the **JSON** tab.
1. Paste the following JSON into the editor.
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:DescribeRouteTables",
"ec2:DescribeSecurityGroups",
"ec2:DescribeVpcPeeringConnections",
"ec2:DescribeVpcs",
"ec2:DescribePrefixLists",
"logs:DescribeLogGroups"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"servicequotas:GetServiceQuota"
],
"Resource": "arn:aws:servicequotas:*:*:vpc/L-0EA8095F"
},
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:DescribeLogStreams"
],
"Resource": "arn:aws:logs:*:*:log-group:dms-data-migration-*"
},
{
"Effect": "Allow",
"Action": [
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": "arn:aws:logs:*:*:log-group:dms-data-migration-*:log-stream:dms-data-migration-*"
},
{
"Effect": "Allow",
"Action": "cloudwatch:PutMetricData",
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"ec2:CreateRoute",
"ec2:DeleteRoute"
],
"Resource": "arn:aws:ec2:*:*:route-table/*"
},
{
"Effect": "Allow",
"Action": [
"ec2:CreateTags"
],
"Resource": [
"arn:aws:ec2:*:*:security-group/*",
"arn:aws:ec2:*:*:security-group-rule/*",
"arn:aws:ec2:*:*:route-table/*",
"arn:aws:ec2:*:*:vpc-peering-connection/*",
"arn:aws:ec2:*:*:vpc/*"
]
},
{
"Effect": "Allow",
"Action": [
"ec2:AuthorizeSecurityGroupEgress",
"ec2:AuthorizeSecurityGroupIngress"
],
"Resource": "arn:aws:ec2:*:*:security-group-rule/*"
},
{
"Effect": "Allow",
"Action": [
"ec2:AuthorizeSecurityGroupEgress",
"ec2:AuthorizeSecurityGroupIngress",
"ec2:RevokeSecurityGroupEgress",
"ec2:RevokeSecurityGroupIngress"
],
"Resource": "arn:aws:ec2:*:*:security-group/*"
},
{
"Effect": "Allow",
"Action": [
"ec2:AcceptVpcPeeringConnection",
"ec2:ModifyVpcPeeringConnectionOptions"
],
"Resource": "arn:aws:ec2:*:*:vpc-peering-connection/*"
},
{
"Effect": "Allow",
"Action": "ec2:AcceptVpcPeeringConnection",
"Resource": "arn:aws:ec2:*:*:vpc/*"
}
]
}
```
------
1. Choose **Next: Tags** and **Next: Review.**
1. Enter **HomogeneousDataMigrationsPolicy** for **Name\$1**, and choose **Create policy**.
## Creating an IAM role for homogeneous data migrations
In this step, you create an IAM role that provides access to AWS Secrets Manager, Amazon EC2, and CloudWatch.
**To create an IAM role for data migrations**
1. Sign in to the AWS Management Console and open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/).
1. In the navigation pane, choose **Roles**.
1. Choose **Create role**.
1. On the **Select trusted entity** page, for **Trusted entity type**, choose **AWS Service**. For **Use cases for other AWS services**, choose **DMS**.
1. Select the **DMS** check box and choose **Next**.
1. On the **Add permissions** page, choose **HomogeneousDataMigrationsPolicy** that you created before. Choose **Next**.
1. On the **Name, review, and create** page, enter **HomogeneousDataMigrationsRole** for **Role name**, and choose **Create role**.
1. On the **Roles** page, enter **HomogeneousDataMigrationsRole** for **Role name**. Choose **HomogeneousDataMigrationsRole**.
1. On the **HomogeneousDataMigrationsRole** page, choose the **Trust relationships** tab. Choose **Edit trust policy**.
1. On the **Edit trust policy** page, paste the following JSON into the editor, replacing the existing text.
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Service": [
"dms-data-migrations.amazonaws.com",
"dms.your_region.amazonaws.com"
]
},
"Action": "sts:AssumeRole"
}
]
}
```
------
In the preceding example, replace *your\$1region* with the name of your AWS Region.
The preceding resource-based policy provides AWS DMS service principals with permissions to perform tasks according to the customer managed **HomogeneousDataMigrationsPolicy** policy.
1. Choose **Update policy**.
# Creating a secret access policy and role
Follow the procedures below to create your secret access policy and role which allow DMS to access the user credentials for your source and target databases.
**To create the secret access policy and role, which allows Amazon RDS to access AWS Secrets Manager to access your appropriate secret**
1. Sign in to the AWS Management Console and open the AWS Identity and Access Management (IAM) console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/).
1. Choose **Policies**, then choose **Create policy**.
1. Choose **JSON** and enter the following policy to enable access to and decryption of your secret.
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "secretsmanager:GetSecretValue",
"Resource": "arn:aws:secretsmanager:us-east-1:111122223333:secret:SecretName-ABCDEF"
},
{
"Effect": "Allow",
"Action": [
"kms:Decrypt",
"kms:DescribeKey"
],
"Resource": "arn:aws:kms:us-east-1:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab"
}
]
}
```
------
Here, `secret_arn` is the ARN of your secret, which you can get from either `SecretsManagerSecretId` as appropriate, and `kms_key_arn` is the ARN of the AWS KMS key that you are using to encrypt your secret, as in the following example.
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "secretsmanager:GetSecretValue",
"Resource": "arn:aws:secretsmanager:us-east-2:123456789012:secret:MySQLTestSecret-qeHamH"
},
{
"Effect": "Allow",
"Action": [
"kms:Decrypt",
"kms:DescribeKey"
],
"Resource": "arn:aws:kms:us-east-2:123456789012:key/761138dc-0542-4e58-947f-4a3a8458d0fd"
}
]
}
```
------
**Note**
If you use the default encryption key created by AWS Secrets Manager, you do not have to specify the AWS KMS permissions for `kms_key_arn`.
If you want your policy to provide access to both secrets, simply specify an additional JSON resource object for the other *secret\$1arn*.
1. Review and create the policy with a friendly name and optional description.
1. Choose **Roles**, then choose **Create role**.
1. Choose **AWS service** as the type of trusted entity.
1. Choose **DMS** from the list of services as the trusted service, then choose **Next: Permissions**.
1. Look up and attach the policy you created in step 4, then proceed through adding any tags and review your role. At this point, edit the trust relationships for the role to use your Amazon RDS regional service principal as the trusted entity. This principal has the following format.
```
dms.region-name.amazonaws.com
```
Here, *`region-name`* is the name of your region, such as `us-east-1`. Thus, an Amazon RDS regional service principal for this region follows.
```
dms.us-east-1.amazonaws.com
dms-data-migrations.amazonaws.com
```
# Creating an IAM role for AWS DMS to manage Amazon VPC
You must create an IAM role for AWS DMS to manage the VPC settings for your resources. This role must be available for successful migration.
**Creating the `dms-vpc-role` for database migration**
1. Sign in to the AWS Management Console and open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/).
1. In the navigation pane of the console, choose **Roles** and then choose **Create role**.
1. Choose the **AWS service** option for the **Select trusted entity** option.
For **Use case**, select **DMS**.
1. For the **Add permissions** step, select `AmazonDMSVPCManagementRole` and choose **Next**.
1. In the **Name, review, and create** page, set the **Role name** to `dms-vpc-role` and choose **Create role**.
This creates the role for the DMS to manage the VPC settings for the migration.
# Setting up data migration for EC2 database
To begin migrating data from your EC2 source database, you must create an equivalent RDS database. For instructions on creating your database, see [Creating an Amazon RDS DB instance](USER_CreateDBInstance.md).
After creating your target database, use the following steps to set up the data migration:
**Set up data migration project**
1. Select the target database on the **Databases** page in the RDS console.
1. Choose the **Actions** dropdown and select the **Migrate data from EC2 database** option. To see the supported target databases, see [Limitations](USER_DMS_migration.md#USER_DMS_migration-Limitations).
1. Under the **Select source EC2 database** section:
1. Check the **Engine type** and make sure it is the same as your source database.
Also, check if the engine versions are compatible.
1. For **EC2 instance**, choose the EC2 instance where your source database resides.
1. For **Port**, enter the port on which your source database allows traffic.
1. For **Secret**, choose **Create and use a new secret** if you don't have an existing secret. Enter the **Username** and **Password** for your source database. Also choose the KMS key with which to encrypt your secret.
If you have an existing secret, select **Use an existing secret** and then choose secret from the dropdown.
1. For **IAM role for secret**, if you have an existing IAM role, select **Use an existing IAM role** and choose an IAM role from the dropdown that can access the secret ID from the previous step.
If you don't have existing IAM role, choose **Create and use new IAM role**. Enter the new name for your role for **IAM role name. You can see the permissions associated with this role in the link below.**
1. Under the **View target RDS database** section:
1. Confirm the settings of your target database at the top of the section.
1. For **Secret**, choose **Create and use a new secret** if you don't have an existing secret that holds your target database credentials.
If you have an existing secret, select the secret from the dropdown.
1. For **IAM role for secret**, select an IAM role that can access the secret from the previous step. You can also create a new IAM role if you don't have existing IAM role.
If the dropdown doesn't populate the IAM roles, specify the **IAM role ARN** in the format `arn:aws:iam:account_id:role/roleName`.
1. Under the **Configure data migration** section:
1. Select the type of data migration by selecting between **Full load**, **Full load and change data capture (CDC)**, or **Change data capture (CDC)**. For more information about these options, see [Overview](USER_DMS_migration.md#USER_DMS_migration-overview).
You can't modify the migration type afer the migration starts.
1. For **IAM role for data migration**, if you have an existing IAM role, select **Use an existing IAM role** and choose an IAM role from the dropdown that grants DMS the permissions to create the resources required for the migration. If you don't have existing IAM role, choose **Create and use new IAM role**.
1. Confirm that the **View migration settings** tab shows the required settings for your data migration to be set up successfully.
1. Select **Migrate** to complete the migration set up.
After completing these steps, you can see the resources being set up for the data migration by choosing **View details** in the progress banner in the console. Once the required resources are set up, the migration automatically starts. If you create
To migrate multiple databases into the target database, start this process again with details about the new EC2 database.
# Managing data migrations
After using the **Migrate data from EC2 database** action from the RDS console, RDS starts the migration automatically.
If you used the AWS DMS console to create the migration resources, you can start the migration process.
## Starting the data migration
Follow these steps to start data migration:
**Starting a data migration**
1. Choose the target database on the **Databases** page in the RDS console.
1. In the database details page, choose the **Data migrations** tab.
1. Under the **Data migrations** tab, the **Associated data migrations** lists the available data migrations.
Migrations set up using the RDS console start automatically once the required resources are set up.
Migrations set up using the DMS console are set to **Ready**.
To begin these migrations, select the **Actions** drop down and select **Start**.
1. This begins the data migration for your EC2 database.
## Stopping the data migration
For data migrations whose replication type is full load, stopping the migration causes the process to stop and can't be resumed. Once stopped, you must restart the migration.
For migrations with replication type set to change data capture (CDC) or full load and CDC, you can stop the continuous replication process, and resume the process later.
**Stopping a data migration**
1. Choose the target database on the **Databases** page in the RDS console.
1. In the database details page, choose the **Data migrations** tab.
1. Under the **Data migrations** tab, the **Associated data migrations** lists the ongoing data migrations.
To stop a migration, select a data migration and select **Stop** in the **Actions** drop down.
1. This stops the data migration for your EC2 database.
## Resuming the data migration
For data migrations whose replication type is full load and change data capture (CDC) or change data capture (CDC) migration, you can resume the CDC process from the last stop point.
**Resuming a data migration**
1. Choose the target database on the **Databases** page in the RDS console.
1. In the database details page, choose the **Data migrations** tab.
1. Under the **Data migrations** tab, the **Associated data migrations** lists the stopped data migrations.
To resume a migration, select a data migration and select **Resume processing** in the **Actions** drop down.
1. This resume the data migration for your EC2 database.
## Deleting the data migration
To delete an associated data migration, use the following instructions
**Deleting a data migration**
1. Choose the target database on the **Databases** page in the RDS console.
1. In the database details page, choose the **Data migrations** tab.
1. To delete a migration, select a data migration and select **Delete** in the **Actions** drop down.
1. This deletes the data migration.
Deleting a data migration that was in progress doesn't impact any data that has already been loaded to the target database.
## Restarting the data migration
To restart an associated data migration from a CDC start point, use the following instructions
**Restarting a data migration**
1. Choose the target database on the **Databases** page in the RDS console.
1. In the database details page, choose the **Data migrations** tab.
1. To restart a migration, select a data migration and select **Restart** in the **Actions** drop down.
1. This restarts the data migration from a CDC start point.
Restarting a data migration that was in progress doesn't impact any data that has already been loaded to the target database.
# Monitoring your data migrations
After the data migrations starts, you can monitor its status and progress. Data migrations of large data sets take hours to complete. To maintain the reliability, availability, and high performance of your data migration, monitor its progress regularly.
**To check the status and progress of your data migration**
1. Choose the target database on the **Databases** page in the RDS console.
1. In the database details page, choose the **Data migrations** tab.
1. The **Associated data migrations** section lists your data migrations. Check the **Status** column.
1. For running data migrations, the **Migration process** column displays the percentage of migrated data.
1. To monitor the process in CloudWatch, use the link in the in **CloudWatch** column.
## Migration statuses
For each data migration that you run, the RDS console displays the **Status**. The following list includes the statuses:
+ `Ready`: The data migration is ready to start.
+ `Starting`: RDS is creating the serverless environment for your data migration.
+ `Load running`: RDS is performing the full load migration.
+ `Load complete, replication ongoing`: RDS completed the full load and now replicates the ongoing changes. This status only applies for full load and CDC type migrations.
+ `Replication ongoing`: RDS is replicating ongoing changes. This status only applies to CDC type migrations.
+ `Stopping`: RDS is stopping the data migrations. This status applies when you choose to stop the data migration from the **Actions** menu.
+ `Stopped`: RDS has stopped the data migration.
+ `Failed`: The data migration has failed. For more information, see the log files.
+ `Restarting`: The data migration has restarted an ongoing data replication from a CDC start point.