# Running homogeneous data migrations in AWS DMS
You can use [Homogeneous data migrations](data-migrations.md) in AWS DMS to migrate data from your source database to the equivalent engine on Amazon Relational Database Service (Amazon RDS) , Amazon Aurora, or Amazon DocumentDB. AWS DMS automates the data migration process by using native database tools in your source and target databases.
After you create an instance profile and compatible data providers for homogeneous data migrations, create a migration project. For more information, see [ Creating migration projects](migration-projects-create.md).
The following sections describe how to create, configure, and run homogeneous data migrations.
**Topics**
+ [Creating a data migration in AWS DMS](dm-migrating-data-create.md)
+ [Selection rules for homogeneous data migrations](dm-migrating-data-selectionrules.md)
+ [Managing data migrations in AWS DMS](dm-migrating-data-manage.md)
+ [Monitoring data migrations in AWS DMS](dm-migrating-data-monitoring.md)
+ [Statuses of homogeneous data migrations in AWS DMS](dm-migrating-data-statuses.md)
+ [Migrating data from MySQL databases with homogeneous data migrations in AWS DMS](dm-migrating-data-mysql.md)
+ [Migrating data from PostgreSQL databases with homogeneous data migrations in AWS DMS](dm-migrating-data-postgresql.md)
+ [Migrating data from MongoDB databases with homogeneous data migrations in AWS DMS](dm-migrating-data-mongodb.md)
+ [Target table preparation mode](dm-migrating-data-table-prep.md)
# Creating a data migration in AWS DMS
After you create a migration project with compatible data providers of the same type, you can use this project for homogeneous data migrations. For more information, see [ Creating migration projects](migration-projects-create.md).
To start using homogeneous data migrations, create a new data migration. You can create several homogeneous data migrations of different types in a single migration project.
AWS DMS has the maximum number of homogeneous data migrations that you can create for your AWS account. See the following section for information about AWS DMS service quotas [Quotas for AWS Database Migration Service](CHAP_Limits.md).
Before you create a data migration, make sure that you set up the required resources such as your source and target databases, an IAM policy and role, an instance profile, and data providers. For more information, see [Creating IAM resources](dm-iam-resources.md), [Creating instance profiles](instance-profiles.md), and [ Creating data providers](data-providers-create.md).
Also, we recommend that you don't use homogeneous data migrations to migrate data from a higher database version to a lower database version. Check the versions of databases that you use for source and target data providers, and upgrade your target database version, if needed.
**To create a data migration**
1. Sign in to the AWS Management Console and open the AWS DMS console at [https://console.aws.amazon.com/dms/v2/](https://console.aws.amazon.com/dms/v2/).
1. Choose **Migration projects**. The **Migration projects** page opens.
1. Choose your migration project, and on the **Data migrations** tab, choose **Create data migration**.
1. For **Name**, enter a name for your data migration. Make sure that you use a unique name for your data migration so that you can easily identify it.
1. For **Replication type**, choose the type of data migration that you want to configure. You can choose one of the following options.
+ **Full load** — Migrates your existing source data.
+ **Full load and change data capture (CDC)** — Migrates your existing source data and replicates ongoing changes.
+ **Change data capture (CDC)** — Replicates ongoing changes.
1. Select the check box for **Turn on CloudWatch logs** to store data migration logs in Amazon CloudWatch. If you don't choose this option, then you can't see the log files when your data migration fails.
1. (Optional) Expand **Advanced settings**. For **Number of jobs**, enter the number of parallel threads that AWS DMS can use to migrate your source data to the target.
1. For **IAM service role**, choose the IAM role that you created in prerequisites. For more information, see [Creating an IAM role for homogeneous data migrations in AWS DMS](dm-iam-resources.md#dm-resources-iam-role).
1. Configure the **Start mode** for data migrations of the **Change data capture (CDC)** type. You can choose one of the following options.
+ **Immediately** — Starts the ongoing replication when you start your data migration.
+ **Using a native start point** — Starts the ongoing replication from the specified point.
For PostgreSQL databases, enter the name of the logical replication slot for **Slot name** and enter the transaction log sequence number for **Native start point**.
For MySQL databases, enter the transaction log sequence number for **Log sequence number (LSN)**.
1. Configure the **Stop mode** for data migrations of the **Change data capture (CDC)** or **Full load and change data capture (CDC)** type. You can choose one of the following options.
+ **Don’t stop CDC** — AWS DMS continues the ongoing replication until you stop your data migration.
+ **Using a server time point** — AWS DMS stops the ongoing replication at the specified time.
If you choose this option, then for **Stop date and time**, enter the date and time when you want to automatically stop the ongoing replication.
1. Choose **Create data migration**.
AWS DMS creates your data migration and adds it to the list on the **Data migrations** tab in your migration project. Here you can see the status of your data migration. For more information, see [Migration statuses](dm-migrating-data-statuses.md).
**Important**
For data migrations of the **Full load** and **Full load and change data capture (CDC)** type, AWS DMS deletes all data, tables, and other database objects on your target database. Make sure you have a backup of your target database.
After AWS DMS creates your data migration, the status of this data migration is set to **Ready**. To migrate your data, you must start the data migration manually. To do so, choose your data migration from the list. Next, for **Actions**, choose **Start**. For more information, see [Managing data migrations](dm-migrating-data-manage.md).
The first launch of a homogeneous data migration requires some setup. AWS DMS creates a serverless environment for your data migration. This process takes up to 15 minutes. After you stop and restart your data migration, AWS DMS doesn't create the environment again, and you can access your data migration faster.
# Selection rules for homogeneous data migrations
You can use selection rules to choose the schema, tables, or both that you want to include in your replication.
When creating data migration task, choose **Add selection rule**.
For the rule settings, provide the following values:
+ **Schema**: Choose **Enter a schema**.
+ **Schema name**: Provide the name of the schema you want to replicate, or use **%** as a wildcard.
+ **Table name**: : Provide the name of the table you want to replicate, or use **%** as a wildcard.
By default, the only rule-action that DMS supports is `Include`, and the only wildcard character that DMS supports is `%`.
**Note**
The support for selection rules AWS DMS for homogeneous data migrations varies based on the combination of the source database engine and the migration type chosen. PostgreSQL and MongoDB-compatible sources allow selection rules for all migration types, while MySQL sources only support selection rules for the Full Load migration type.
**Example Migrate all tables in a schema**
The following example migrates all tables from a schema named `dmsst` in your source to your target endpoint.
```
{
"rules": [
{
"rule-type": "selection",
"rule-action": "include",
"object-locator": {
"schema-name": "dmsst",
"table-name": "%"
},
"filters": [],
"rule-id": "1",
"rule-name": "1"
}
]
}
```
**Example Migrate some tables in a schema**
The following example migrates all tables with a name starting with `collectionTest`, from a schema named `dmsst` in your source to your target endpoint.
```
{
"rules": [
{
"rule-type": "selection",
"rule-action": "include",
"object-locator": {
"schema-name": "dmsst",
"table-name": "collectionTest%"
},
"filters": [],
"rule-id": "1",
"rule-name": "1"
}
]
}
```
**Example Migrate specific tables from multiple schemas**
The following example migrates some of the tables from multiple schemas named `dmsst` and `Test` in your source to your target endpoint.
```
{
"rules": [
{
"rule-type": "selection",
"rule-action": "include",
"object-locator": {
"schema-name": "dmsst",
"table-name": "collectionTest1"
},
"filters": [],
"rule-id": "1",
"rule-name": "1"
},
{
"rule-type": "selection",
"rule-action": "include",
"object-locator": {
"schema-name": "Test",
"table-name": "products"
},
"filters": [],
"rule-id": "2",
"rule-name": "2"
}
]
}
```
# Managing data migrations in AWS DMS
After you create a data migration, AWS DMS doesn't automatically start migrating data. You start a data migration manually when needed.
Before you start a data migration, you can modify all settings of your data migration. After you start your data migration, you can't change the replication type. To use another replication type, create a new data migration.
**To start a data migration**
1. Sign in to the AWS Management Console and open the AWS DMS console at [https://console.aws.amazon.com/dms/v2/](https://console.aws.amazon.com/dms/v2/).
1. Choose **Migration projects**. The **Migration projects** page opens.
1. Choose your migration project. On the **Data migrations** tab, choose your data migration. The **Summary** page for your data migration opens.
1. For **Actions**, choose **Start**.
After this, AWS DMS creates a serverless environment for your data migration. This process takes up to 15 minutes.
After you start a data migration, AWS DMS sets its status to **Starting**. The next status that AWS DMS uses for your data migration, depends on the type of replication that you choose in the data migration settings. For more information, see [Migration statuses](dm-migrating-data-statuses.md).
**To modify a data migration**
1. Sign in to the AWS Management Console and open the AWS DMS console at [https://console.aws.amazon.com/dms/v2/](https://console.aws.amazon.com/dms/v2/).
1. Choose **Migration projects**. The **Migration projects** page opens.
1. Choose your migration project. On the **Data migrations** tab, choose your data migration. The **Summary** page for your data migration opens.
1. Choose **Modify**.
1. Configure the settings for your data migration.
**Important**
If you have started a data migration, then you can't change the replication type.
1. To view your data migration logs in Amazon CloudWatch, select the check box for **Turn on CloudWatch logs**.
1. Choose **Save changes**.
After AWS DMS starts a data migration, you can stop it. To do so, choose your data migration on the **Data migrations** tab. Next, for **Actions**, choose **Stop**.
After you stop a data migration, AWS DMS sets its status to **Stopping**. Next, AWS DMS sets the status of this data migration to **Stopped**. After AWS DMS stops a data migration, you can modify, resume, restart, or delete your data migration.
To continue the data replication, choose the data migration that you stopped on the **Data migrations** tab. Next, for **Actions**, choose **Resume processing**.
To restart the data load, choose the data migration that you stopped on the **Data migrations** tab. Next, for **Actions**, choose **Restart**. AWS DMS deletes all data from your target database and starts the data migration from scratch.
You can delete a data migration that you have stopped or that you haven't started. To delete a data migration, choose it on the **Data migrations** tab. Next, for **Actions**, choose **Delete**. To delete your migration project, stop and delete all data migrations.
# Monitoring data migrations in AWS DMS
After you start your homogeneous data migration, you can monitor its status and progress. Data migrations of large data sets such as hundreds of gigabytes take hours to complete. To maintain the reliability, availability, and high performance of your data migration, monitor its progress regularly.
**To check the status and progress of your data migration**
1. Sign in to the AWS Management Console and open the AWS DMS console at [https://console.aws.amazon.com/dms/v2/](https://console.aws.amazon.com/dms/v2/).
1. Choose **Migration projects**. The **Migration projects** page opens.
1. Choose your migration project and navigate to the **Data migrations** tab.
1. For your data migration, see the **Status** column. For more information about values in this column, see [Migration statuses](dm-migrating-data-statuses.md).
1. For a running data migration, the **Migration progress** column displays the percentage of migrated data.
**To check the details of your data migration**
1. Sign in to the AWS Management Console and open the AWS DMS console at [https://console.aws.amazon.com/dms/v2/](https://console.aws.amazon.com/dms/v2/).
1. Choose **Migration projects**. The **Migration projects** page opens.
1. Choose your migration project. On the **Data migrations** tab, choose your data migration.
1. On the **Details** tab, you can see the migration progress. Particularly, you can see the following metrics.
+ **Public IP address** – The public IP address of your data migration. You need this value to configure a network. For more information, see [Setting up a network](dm-network.md).
+ **Tables loaded** – The number of successfully loaded tables.
+ **Tables loading** – The number of tables currently loading.
+ **Tables queued** – The number of tables currently waiting to be loaded.
+ **Tables errored** – The number of tables that failed to load.
+ **Elapsed time** – The amount of time that passed after the start of your data migration.
+ **CDC latency** – The average time that passes between when a change occurs on a source table and when AWS DMS applies this change to the target table.
+ **Migration started** – The time when you started this data migration.
+ **Migration stopped** – The time when you stopped this data migration.
1. To view the log files for your data migration, choose **View CloudWatch logs** under **Homogeneous data migration settings**. You can **Turn on CloudWatch logs** when you create or modify a data migration. For more information, see [Creating a data migration](dm-migrating-data-create.md) and [Managing data migrations](dm-migrating-data-manage.md).
You can use Amazon CloudWatch alarms or events to closely track your data migration. For more information, see [What are Amazon CloudWatch, Amazon CloudWatch Events, and Amazon CloudWatch Logs?](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html) in the *Amazon CloudWatch User Guide*. Note that there is a charge for using Amazon CloudWatch.
For homogeneous data migrations, AWS DMS includes the following metrics in Amazon CloudWatch.
| Metric | Description |
| --- | --- |
| OverallCDCLatency | The overall latency during the CDC phase. For MySQL databases, this metric shows the number of seconds that passes between the change in the source binary log and the replication of this change. For PostgreSQL databases, this metric shows the number of seconds that passes between `last_msg_receipt_time` and `last_msg_send_time` from the `pg_stat_subscription` view. Units: Seconds |
| StorageConsumption | The storage that your data migration consumes. Units: Bytes |
# Statuses of homogeneous data migrations in AWS DMS
For each data migration that you run, AWS DMS displays the **Status** in the AWS DMS console. The following list includes the available statuses.
+ `Creating` – AWS DMS is creating the data migration.
+ `Ready` – The data migration is ready to start.
+ `Starting` – AWS DMS is creating the serverless environment for your data migration. This process takes up to 15 minutes.
+ `Load running` – AWS DMS is performing the full load migration.
+ `Load complete, replication ongoing` – AWS DMS completed the full load and now replicates ongoing changes. AWS DMS uses this status only for data migrations of the full load and change data capture (CDC) type.
+ `Replication ongoing` – AWS DMS is replicating ongoing changes. AWS DMS uses this status only for migrations of the change data capture (CDC) type.
+ `Reloading target` – AWS DMS is restarting a data migration and performs the specified migration type.
+ `Stopping` – AWS DMS is stopping the data migration. AWS DMS sets this status after you choose to stop the data migration on the **Actions** menu.
+ `Stopped` – AWS DMS has stopped the data migration.
+ `Failed` – The data migration has failed. For more information, see the log files.
To view the log files, choose your data migration on the **Data migrations** tab. Next, choose **View CloudWatch logs** under **Homogeneous data migration settings**.
**Important**
You can view log files if you select the check box for **Turn on CloudWatch logs** when you create your data migration.
+ `Deleting` – AWS DMS is deleting the data migration. AWS DMS sets this status after you choose to delete the data migration on the **Actions** menu.
+ `Maintenance` – AWS DMS puts a task in maintenance mode status when new image is deployed on the underlying serverless container associated with your data migration task.
# Migrating data from MySQL databases with homogeneous data migrations in AWS DMS
You can use [Homogeneous data migrations](data-migrations.md) to migrate a self-managed MySQL database to RDS for MySQL or Aurora MySQL. AWS DMS creates a serverless environment for your data migration. For different types of data migrations, AWS DMS uses different native MySQL database tools.
For homogeneous data migrations of the **Full load** type, AWS DMS uses mydumper to read data from your source database and store it on the disk attached to the serverless environment. After AWS DMS reads all your source data, it uses myloader in the target database to restore your data.
For homogeneous data migrations of the **Full load and change data capture (CDC)** type, AWS DMS uses mydumper to read data from your source database and store it on the disk attached to the serverless environment. After AWS DMS reads all your source data, it uses myloader in the target database to restore your data. After AWS DMS completes the full load, it sets up the binlog replication with the binlog position set to the start of the full load.
For homogeneous data migrations of the **Change data capture (CDC)** type, AWS DMS requires the **Native CDC start point** to start the replication. If you provide the native CDC start point, then AWS DMS captures changes from that point. Alternatively, choose **Immediately** in the data migration settings to automatically capture the start point for the replication when the actual data migration starts.
**Note**
For a CDC-only migration to work properly, all source database schemas and objects must already be present on the target database. The target may have objects that are not present on the source, however.
You can use the following code example to get the current log sequence number (LSN) in your MySQL database.
```
show master status
```
This query returns a binlog file name and the position. For the native start point, use a combination of the binlog file name and the position. For example, `mysql-bin-changelog.000024:373`. In this example, `mysql-bin-changelog.000024` is the binlog file name and `373` is the position where AWS DMS starts capturing changes.
The following diagram shows the process of using homogeneous data migrations in AWS DMS to migrate a MySQL database to RDS for MySQL or Aurora MySQL.
![\[An architecture diagram of the MySQL data migration with DMS Homogeneous Data Migrations.\]](http://docs.aws.amazon.com/dms/latest/userguide/images/data-migrations-mysql.png)
# Migrating data from PostgreSQL databases with homogeneous data migrations in AWS DMS
You can use [Homogeneous data migrations](data-migrations.md) to migrate a self-managed PostgreSQL database to RDS for PostgreSQL or Aurora PostgreSQL. AWS DMS creates a serverless environment for your data migration. For different types of data migrations, AWS DMS uses different native PostgreSQL database tools.
For homogeneous data migrations of the **Full load** type, AWS DMS uses pg\$1dump to read data from your source database and store it on the disk attached to the serverless environment. After AWS DMS reads all your source data, it uses pg\$1restore in the target database to restore your data.
For homogeneous data migrations of the **Full load and change data capture (CDC)** type, AWS DMS uses `pg_dump` to read schema objects without table data from your source database and store them on the disk attached to the serverless environment. It then uses `pg_restore` in the target database to restore your schema objects. After AWS DMS completes the `pg_restore` process, it automatically switches to a publisher and subscriber model for logical replication with the `Initial Data Synchronization` option to copy initial table data directly from the source database to the target database, and then initiates ongoing replication. In this model, one or more subscribers subscribe to one or more publications on a publisher node.
For homogeneous data migrations of the **Change data capture (CDC)** type, AWS DMS requires the native start point to start the replication. If you provide the native start point, then AWS DMS captures changes from that point. Alternatively, choose **Immediately** in the data migration settings to automatically capture the start point for the replication when the actual data migration starts.
**Note**
For a CDC-only migration to work properly, all source database schemas and objects must already be present on the target database. The target may have objects that are not present on the source, however.
You can use the following code example to get the native start point in your PostgreSQL database.
```
select confirmed_flush_lsn from pg_replication_slots where slot_name=‘migrate_to_target';
```
This query uses the `pg_replication_slots` view in your PostgreSQL database to capture the log sequence number (LSN) value.
After AWS DMS sets the status of your PostgreSQL homogeneous data migration to **Stopped**, **Failed**, or **Deleted**, the publisher and replication aren't removed. If you don't want to resume the migration, then delete the replication slot and the publisher by using the following command.
```
SELECT pg_drop_replication_slot('migration_subscriber_{ARN}');
DROP PUBLICATION publication_{ARN};
```
The following diagram shows the process of using homogeneous data migrations in AWS DMS to migrate a PostgreSQL database to RDS for PostgreSQL or Aurora PostgreSQL.
![\[An architecture diagram of the PostgreSQL data migration with DMS Homogeneous Data Migrations.\]](http://docs.aws.amazon.com/dms/latest/userguide/images/data-migrations-postgresql.png)
## Best practices for using a PostgreSQL database as a source for homogeneous data migrations
+ To speed up initial data syncing on the subscriber side for FLCDC task, you must adjust `max_logical_replication_workers` and `max_sync_workers_per_subscription`. Increasing these values enhances table synchronization speed.
+ **max\$1logical\$1replication\$1workers** – Specifies maximum number of logical replication workers. This includes both the apply workers on the subscriber side and the table synchronization workers.
+ **max\$1sync\$1workers\$1per\$1subscription** – Increasing `max_sync_workers_per_subscription` only affects the number of tables that are synchronized in parallel, not the number of workers per table.
**Note**
`max_logical_replication_workers` should not exceed `max_worker_processes`, and `max_sync_workers_per_subscription` should be less than or equal to `max_logical_replication_workers`.
+ For migrating large tables, consider dividing them into separate tasks using selection rules. For example, you can divide large tables into separate individual tasks and small tables into another single task.
+ Monitor disk and CPU usage on the subscriber side to maintain optimal performance.
# Migrating data from MongoDB databases with homogeneous data migrations in AWS DMS
You can use [Homogeneous data migrations](data-migrations.md) to migrate a self-managed MongoDB database to Amazon DocumentDB. AWS DMS creates a serverless environment for your data migration. For different types of data migrations, AWS DMS uses different native MongoDB database tools.
For homogeneous data migrations of the **Full load** type, AWS DMS uses `mongodump` to read data from your source database and store it on the disk attached to the serverless environment. After AWS DMS reads all your source data, it uses `mongorestore` in the target database to restore your data.
For homogeneous data migrations of the **Full load and change data capture (CDC)** type, AWS DMS uses `mongodump` to read data from your source database and store it on the disk attached to the serverless environment. After AWS DMS reads all your source data, it uses `mongorestore` in the target database to restore your data. After AWS DMS completes the full load, it automatically switches to a publisher and subscriber model for logical replication. In this model, we recommend sizing the oplog to retain changes for at least 24 hours.
For homogeneous data migrations of the **Change data capture (CDC)** type, choose `immediately` in the data migration settings to automatically capture the start point for the replication when the actual data migration starts.
**Note**
For any new or renamed collection, you need to create a new data migration task for those collections as homogeneous data migrations. For a MongoDB-compatible source, AWS DMS doesn't support `create`, `rename` and `drop collection` operations.
The following diagram shows the process of using homogeneous data migrations in AWS DMS to migrate a MongoDB database to Amazon DocumentDB.
![\[An architecture diagram of the MongoDB data migration with DMS Homogeneous Data Migrations.\]](http://docs.aws.amazon.com/dms/latest/userguide/images/data-migrations-mongodb.png)
# Target table preparation mode
You can select Target table preparation mode when you choose to a create data migration task under the **Advanced Settings** tab in the AWS DMS console for PostgreSQL, MongoDB, and Amazon DocumentDB migrations.
## Drop tables on target
In Drop tables on target mode, AWS DMS homogeneous migration drops the target tables and recreates them before starting the migration. This approach ensures that the target tables are empty at the start of the migration. During homogeneous migrations, AWS DMS creates all secondary objects, including indexes defined in the source table metadata, before loading the data to ensure efficient data migration.
When using the Drop tables on target mode, you might need to configure the target database. For example, with a PostgreSQL target, AWS DMS cannot create a schema user for security reasons. In this case, you must pre-create the schema user to match the source, allowing AWS DMS to create the tables and assign them to a similar role as the source when the migration begins.
## Truncate
In Truncate mode, AWS DMS homogeneous migration truncates all existing target tables before the migration begins. This preserves the table structure. This mode is suitable for full load or full load plus CDC migrations where the target schema is pre-created. For an Amazon DocumentDB target, if the collection does not exist, AWS DMS creates the collection without indexes during the migration.
## Do nothing
In Do nothing mode, AWS DMS homogeneous migration assumes that the target tables are pre-created. If the target tables are not empty, data conflicts may occur during migration, potentially causing a DMS task error. In this mode, the table structure remains unchanged, and any existing data is preserved. Do nothing mode is suitable for CDC-only tasks when the target tables have been backfilled from the source, and ongoing replication is used to synchronize the source and target. For an Amazon DocumentDB target, if the collection does not exist, AWS DMS creates the collection without secondary indexes. Additionally, Do nothing mode can be used during the Full Load phase when migrating data from a MongoDB sharded collection to Amazon DocumentDB.