# Replicating existing objects with Batch Replication
S3 Batch Replication differs from live replication, which continuously and automatically replicates new objects across Amazon S3 buckets. Instead, S3 Batch Replication occurs on demand on existing objects. You can use S3 Batch Replication to replicate the following types of objects:
+ Objects that existed before a replication configuration was in place
+ Objects that have previously been replicated
+ Objects that have failed replication
You can replicate these objects on demand by using a Batch Operations job.
To get started with Batch Replication, you can:
+ **Initiate Batch Replication for a new replication rule or destination** – You can create a one-time Batch Replication job when you're creating the first rule in a new replication configuration or when you're adding a new destination bucket to an existing configuration through the Amazon S3 console.
+ **Initiate Batch Replication for an existing replication configuration** – You can create a new Batch Replication job by using S3 Batch Operations through the Amazon S3 console, the AWS Command Line Interface (AWS CLI), the AWS SDKs, or the Amazon S3 REST API.
When the Batch Replication job finishes, you receive a completion report. For more information about how to use this report to examine the job, see [Tracking job status and completion reports](batch-ops-job-status.md).
## S3 Batch Replication considerations
Before using S3 Batch Replication, review the following list of considerations:
+ Your source bucket must have an existing replication configuration. To enable replication, see [Setting up live replication overview](replication-how-setup.md) and [Examples for configuring live replication](replication-example-walkthroughs.md).
+ If you have S3 Lifecycle configured for your bucket, we recommend disabling your lifecycle rules while the Batch Replication job is active. Doing so helps ensure parity between the source and destination buckets. Otherwise, these buckets could diverge, and the destination bucket won't be an exact replica of the source bucket. For example, consider the following scenario:
+ Your source bucket has multiple versions of an object and a delete marker on that object.
+ Your source and destination buckets have a lifecycle configuration to remove expired delete markers.
In this scenario, Batch Replication might replicate the delete marker to the destination bucket before replicating the object versions. This behavior could result in your lifecycle configuration marking the delete marker as expired and the delete marker being removed from the destination bucket before the object versions are replicated.
+ The AWS Identity and Access Management (IAM) role that you specify to run the Batch Operations job must have the necessary permissions to perform the underlying Batch Replication operation. For more information about creating IAM roles, see [Configuring an IAM role for S3 Batch Replication](s3-batch-replication-policies.md).
+ Batch Replication requires a manifest, which can be generated by Amazon S3. The generated manifest must be stored in the same AWS Region as the source bucket. If you choose not to generate the manifest, you can supply an Amazon S3 Inventory report or CSV file that contains the objects that you want to replicate. For more information, see [Specifying a manifest for a Batch Replication job](#batch-replication-manifest).
+ Batch Replication doesn't support re-replicating objects that were deleted by specifying the version ID of the object from the destination bucket. To re-replicate these objects, you can copy the source objects in place with a Batch Copy job. Copying those objects in place creates new versions of the objects in the source bucket and automatically initiates replication to the destination bucket. Deleting and recreating the destination bucket doesn't initiate replication.
For more information about Batch Copy, see [Examples that use Batch Operations to copy objects](batch-ops-examples-copy.md).
+ If you're using a replication rule on the source bucket, make sure to [update your replication configuration](https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication-walkthrough-2.html) by granting the IAM role that's attached to the replication rule the proper permissions to replicate objects. This IAM role must have the necessary permissions to perform replication on both the source and destination buckets.
+ If you submit multiple Batch Replication jobs for the same bucket within a short time frame, Amazon S3 runs those jobs concurrently.
+ If you submit multiple Batch Replication jobs for two different buckets, be aware that Amazon S3 might not run all jobs concurrently. If you exceed the number of Batch Replication jobs that can run at one time on your account, Amazon S3 pauses the lower priority jobs to work on the higher priority ones. After the higher priority jobs are completed, any paused jobs become active again.
+ Batch Replication isn't supported for objects that are stored in the S3 Glacier Flexible Retrieval and S3 Glacier Deep Archive storage classes.
+ To batch replicate S3 Intelligent-Tiering objects that are stored in the Archive Access or Deep Archive Access storage tiers, you must first initiate a [restore](https://docs.aws.amazon.com/AmazonS3/latest/userguide/intelligent-tiering-managing.html#restore-data-from-int-tier-archive) request and wait until the objects are moved to the Frequent Access tier.
+ A single Batch Replication job can support a manifest with up to 20 billion objects.
+ If you use S3 Batch Replication to replicate datasets cross region and your objects previously had their server-side encryption type updated from SSE-S3 to SSE-KMS, you may need additional permissions. On the source region bucket, you must have `kms:decrypt` permissions. Then, you will need the `kms:decrypt` and `kms:encrypt` permissions for the bucket in the destination region. For more information, see [Replicating encrypted objects](replication-config-for-kms-objects.md).
## Specifying a manifest for a Batch Replication job
A manifest is an Amazon S3 object that contains the object keys that you want Amazon S3 to act upon. If you want to create a Batch Replication job, you must supply either a user-generated manifest or have Amazon S3 generate a manifest based on your replication configuration.
If you supply a user-generated manifest, it must be in the form of an Amazon S3 Inventory report or a CSV file. If the objects in your manifest are in a versioned bucket, you must specify the version IDs for the objects. Only the object with the version ID that's specified in the manifest will be replicated. To learn more about specifying a manifest, see [Specifying a manifest](batch-ops-create-job.md#specify-batchjob-manifest).
If you choose to have Amazon S3 generate a manifest file on your behalf, the objects listed use the same source bucket, prefix, and tags as your replication configurations on the source bucket. With a generated manifest, Amazon S3 replicates all eligible versions of your objects.
**Note**
If you choose to have Amazon S3 generate the manifest, the manifest must be stored in the same AWS Region as the source bucket.
## Filters for a Batch Replication job
When creating your Batch Replication job, you can optionally specify additional filters, such as the object creation date and replication status, to reduce the scope of the job.
You can filter objects to replicate based on the `ObjectReplicationStatuses` value, by providing one or more of the following values:
+ `"NONE"` – Indicates that Amazon S3 has never attempted to replicate the object before.
+ `"FAILED"` – Indicates that Amazon S3 has attempted, but failed, to replicate the object before.
+ `"COMPLETED"` – Indicates that Amazon S3 has successfully replicated the object before.
+ `"REPLICA"` – Indicates that this object is a replica that Amazon S3 has replicated from another source bucket.
For more information about replication statuses, see [Getting replication status information](replication-status.md).
If you don't filter your Batch Replication job, Batch Operations attempts to replicate all objects (no matter their `ObjectReplicationStatus`) in your manifest that match the rules in your replication configuration, except for certain objects that aren't replicated by default. For more information, see [What isn't replicated with replication configurations?](replication-what-is-isnot-replicated.md#replication-what-is-not-replicated)
Depending on your goal, you might set `ObjectReplicationStatuses` to one or more of the following values:
+ To replicate only existing objects that have never been replicated, only include `"NONE"`.
+ To retry replicating only objects that previously failed to replicate, only include `"FAILED"`.
+ To both replicate existing objects and retry replicating objects that previously failed to replicate, include both `"NONE"` and `"FAILED"`.
+ To backfill a destination bucket with objects that have been replicated to another destination, include `"COMPLETED"`.
+ To replicate objects that were previously replicated, include `"REPLICA"`.
## Batch Replication completion report
When you create a Batch Replication job, you can request a CSV completion report. This report shows the objects, replication success or failure codes, outputs, and descriptions. For more information about job tracking and completion reports, see [Completion reports](batch-ops-job-status.md#batch-ops-completion-report).
For a list of replication failure codes and descriptions, see [Amazon S3 replication failure reasons](replication-metrics-events.md#replication-failure-codes).
For information about troubleshooting Batch Replication, see [Batch Replication errors](replication-troubleshoot.md#troubleshoot-batch-replication-errors).
## Getting started with Batch Replication
To learn more about how to use Batch Replication, see [Tutorial: Replicating existing objects in your Amazon S3 buckets with S3 Batch Replication](https://aws.amazon.com/getting-started/hands-on/replicate-existing-objects-with-amazon-s3-batch-replication/).
# Configuring an IAM role for S3 Batch Replication
Because Amazon S3 Batch Replication is a type of Batch Operations job, you must create an AWS Identity and Access Management (IAM) role to grant Batch Operations permissions to perform actions on your behalf. You also must attach a Batch Replication IAM policy to the Batch Operations IAM role.
Use the following procedures to create a policy and an IAM role that give Batch Operations permission to initiate a Batch Replication job.
**To create a policy for Batch Replication**
1. Sign in to the AWS Management Console and open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/).
1. Under **Access management**, choose **Policies**.
1. Choose **Create policy**.
1. On the **Specify permissions** page, choose **JSON**.
1. Insert one of the following policies, depending on whether your manifest is generated by Amazon S3 or whether you are supplying your own manifest. For more information about manifests, see [Specifying a manifest for a Batch Replication job](s3-batch-replication-batch.md#batch-replication-manifest).
Before using these policies, replace the `user input placeholders` in the following policies with the names of your replication source bucket, manifest bucket, and completion report bucket.
**Note**
Your IAM role for Batch Replication needs different permissions, depending on whether you are generating a manifest or supplying one, so make sure that you choose the appropriate policy from the following examples.
**Policy if using and storing an Amazon S3 generated manifest**
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Action": [
"s3:InitiateReplication"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::amzn-s3-demo-source-bucket/*"
]
},
{
"Action": [
"s3:GetReplicationConfiguration",
"s3:PutInventoryConfiguration"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::amzn-s3-demo-source-bucket"
]
},
{
"Action": [
"s3:GetObject",
"s3:GetObjectVersion"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::amzn-s3-demo-manifest-bucket/*"
]
},
{
"Effect": "Allow",
"Action": [
"s3:PutObject"
],
"Resource": [
"arn:aws:s3:::amzn-s3-demo-completion-report-bucket/*",
"arn:aws:s3:::amzn-s3-demo-manifest-bucket/*"
]
}
]
}
```
------
**Policy if using a user-supplied manifest**
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Action": [
"s3:InitiateReplication"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::amzn-s3-demo-source-bucket/*"
]
},
{
"Action": [
"s3:GetObject",
"s3:GetObjectVersion"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::amzn-s3-demo-manifest-bucket/*"
]
},
{
"Effect": "Allow",
"Action": [
"s3:PutObject"
],
"Resource": [
"arn:aws:s3:::amzn-s3-demo-completion-report-bucket/*"
]
}
]
}
```
------
1. Choose **Next**.
1. Specify a name for the policy, and then choose **Create policy**.
**To create an IAM role for Batch Replication**
1. Sign in to the AWS Management Console and open the IAM console at [https://console.aws.amazon.com/iam/](https://console.aws.amazon.com/iam/).
1. Under **Access management**, choose **Roles**.
1. Choose **Create role**.
1. Choose **AWS service** as the type of trusted entity. In the **Use case** section, choose **S3** as the service, and **S3 Batch Operations** as the use case.
1. Choose **Next**. The **Add permissions** page appears. In the search box, search for the policy that you created in the preceding procedure. Select the checkbox next to the policy name, then choose **Next**.
1. On the **Name, review, and create** page, specify a name for your IAM role.
1. In the **Step 1: Trust identities** section, verify that your IAM role is using the following trust policy:
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement":[
{
"Effect":"Allow",
"Principal":{
"Service":"batchoperations.s3.amazonaws.com"
},
"Action":"sts:AssumeRole"
}
]
}
```
------
1. In the **Step 2: Add permissions** section, verify that your IAM role is using the policy that you created earlier.
1. Choose **Create role**.
# Create a Batch Replication job for new replication rules or destinations
In Amazon S3, live replication doesn't replicate any objects that already existed in your source bucket before you created a replication configuration. Live replication automatically replicates only new and updated objects that are written to the bucket after the replication configuration is created. To replicate already existing objects, you can use S3 Batch Replication to replicate these objects on demand.
When you create the first rule in a new live replication configuration or add a new destination bucket to an existing replication configuration through the Amazon S3 console, you can optionally create a Batch Replication job. You can use this Batch Replication job to replicate existing objects in the source bucket to the destination bucket.
To use Batch Replication for an existing configuration without adding a new destination bucket, see [Create a Batch Replication job for existing replication rules](s3-batch-replication-existing-config.md).
**Prerequisites**
Before creating your Batch Replication job, you must create a Batch Operations AWS Identity and Access Management (IAM) role to grant Amazon S3 permissions to perform actions on your behalf. For more information, see [Configuring an IAM role for S3 Batch Replication](s3-batch-replication-policies.md).
## Using Batch Replication for a new replication rule or destination through the Amazon S3 console
When you create the first rule in a new replication configuration or add a new destination bucket to an existing configuration through the Amazon S3 console, you can choose to create a Batch Replication job to replicate existing objects in the source bucket.
**To create a Batch Replication job when creating or updating a replication configuration**
1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/).
1. In the left navigation pane, choose **Buckets**.
1. In the **General purpose buckets** list, choose the name of the bucket that contains the objects that you want to replicate.
1. To create a new replication rule or edit an existing rule, choose the **Management** tab, and scroll down to **Replication rules**:
+ To create a new replication rule, choose **Create replication rule**. For examples of how to set up a basic replication rule, see [Examples for configuring live replication](replication-example-walkthroughs.md).
+ To edit an existing replication rule, select the option button next to the rule name, and then choose **Edit rule**.
1. Create your new replication rule or edit the destination for your existing replication rule, and choose **Save**.
After you create the first rule in a new replication configuration or edit an existing configuration to add a new destination, a **Replicate existing objects?** dialog appears, giving you the option to create a Batch Replication job.
1. If you want to create and run this job now, choose **Yes, replicate existing objects**.
If you want to create a Batch Replication job at a later time, choose **No, do not replicate existing objects**.
1. If you chose **Yes, replicate existing objects**, the **Create Batch Operations job** page appears. The S3 Batch Replication job has the following settings:
**Job run options**
If you want the S3 Batch Replication job to run immediately, choose **Automatically run the job when it's ready**. If you want to run the job at a later time, choose **Wait to run the job when it's ready**.
If you choose **Automatically run the job when it's ready**, you won't be able to create and save a Batch Operations manifest. To save the Batch Operations manifest, choose **Wait to run the job when it's ready**.
**Batch Operations manifest**
If you chose **Wait to run the job when it's ready**, the **Batch Operations manifest** section appears. The manifest is a list of all of the objects that you want to run the specified action on. You can choose to save the manifest. Similar to S3 Inventory files, the manifest will be saved as a CSV file and stored in a bucket. To learn more about Batch Operations manifests, see [Specifying a manifest](batch-ops-create-job.md#specify-batchjob-manifest).
**Completion report**
S3 Batch Operations executes one task for each object specified in the manifest. Completion reports provide an easy way to view the results of your tasks in a consolidated format with no additional setup required. You can request a completion report for all tasks or only for failed tasks. To learn more about completion reports, see [Completion reports](batch-ops-job-status.md#batch-ops-completion-report).
**Permissions**
One of the most common causes of replication failures is insufficient permissions in the provided AWS Identity and Access Management (IAM) role. For information about creating this role, see [Configuring an IAM role for S3 Batch Replication](s3-batch-replication-policies.md). Make sure that you create or choose an IAM role that has the required permissions for Batch Replication.
1. Choose **Save**.
# Create a Batch Replication job for existing replication rules
In Amazon S3, live replication doesn't replicate any objects that already existed in your source bucket before you created a replication configuration. Live replication automatically replicates only new and updated objects that are written to the bucket after the replication configuration is created. To replicate already existing objects, you can use S3 Batch Replication to replicate these objects on demand.
You can configure S3 Batch Replication for an existing replication configuration by using the AWS SDKs, AWS Command Line Interface (AWS CLI), or the Amazon S3 console. For an overview of Batch Replication, see [Replicating existing objects with Batch Replication](s3-batch-replication-batch.md).
When the Batch Replication job finishes, you receive a completion report. For more information about how to use the report to examine the job, see [Tracking job status and completion reports](batch-ops-job-status.md).
**Prerequisites**
Before creating your Batch Replication job, you must create a Batch Operations AWS Identity and Access Management (IAM) role to grant Amazon S3 permissions to perform actions on your behalf. For more information, see [Configuring an IAM role for S3 Batch Replication](s3-batch-replication-policies.md).
## Using the S3 console
1. Sign in to the AWS Management Console and open the Amazon S3 console at [https://console.aws.amazon.com/s3/](https://console.aws.amazon.com/s3/).
1. In the left navigation pane, choose **Batch Operations**.
1. Choose **Create job**.
1. Verify that the **AWS Region** section shows the Region where you want to create your job.
1. In the **Manifest** section, specify the manifest format that you want to use. The manifest is a list of all of the objects that you want to run the specified action on. To learn more about Batch Operations manifests, see [Specifying a manifest](batch-ops-create-job.md#specify-batchjob-manifest).
+ If you have a manifest prepared, choose **S3 inventory report (manifest.json)** or **CSV**. If your manifest is in a versioned bucket, you can specify the version ID for the manifest. If you don't specify a version ID, Batch Operations uses the current version of your manifest. For more information about creating a manifest, see [Specifying a manifest](batch-ops-create-job.md#specify-batchjob-manifest).
**Note**
If the objects in your manifest are in a versioned bucket, you must specify the version IDs for the objects. For more information, see [Specifying a manifest](batch-ops-create-job.md#specify-batchjob-manifest).
+ To create a manifest based on your replication configuration, choose **Create manifest using S3 Replication configuration**. Then choose the source bucket of your replication configuration.
1. (Optional) If you chose **Create manifest using S3 Replication configuration**, you can include additional filters, such as the object creation date and replication status. For examples of how to filter by replication status, see [Specifying a manifest for a Batch Replication job](s3-batch-replication-batch.md#batch-replication-manifest).
1. (Optional) If you chose **Create manifest using S3 Replication configuration**, you can save the generated manifest. To save this manifest, select **Save Batch Operations manifest**. Then specify the destination bucket for the manifest and choose whether to encrypt the manifest.
**Note**
The generated manifest must be stored in the same AWS Region as the source bucket.
1. Choose **Next**.
1. On the **Operations** page, choose **Replicate**, then choose **Next**.
1. (Optional) Provide a **Description**.
1. Adjust the **Priority** of the job if needed. Higher numbers indicate higher priority. Amazon S3 attempts to run higher priority jobs before lower priority jobs. For more information about job priority, see [Assigning job priority](batch-ops-job-priority.md).
1. (Optional) Generate a completion report. To generate this report, select **Generate completion report**.
If you choose to generate a completion report, you must choose either to report **Failed tasks only** or **All tasks**, and provide a destination bucket for the report.
1. In the **Permissions** section, make sure that you choose an IAM role that has the required permissions for Batch Replication. One of the most common causes of replication failures is insufficient permissions in the provided IAM role. For information about creating this role, see [Configuring an IAM role for S3 Batch Replication](s3-batch-replication-policies.md).
1. (Optional) Add job tags to the Batch Replication job.
1. Choose **Next**.
1. Review your job configuration, and then choose **Create job**.
## Using the AWS CLI with an S3 manifest
The following example `create-job` command creates an S3 Batch Replication job by using an S3 generated manifest for the AWS account `111122223333`. This example replicates existing objects and objects that previously failed to replicate. For information about filtering by replication status, see [Specifying a manifest for a Batch Replication job](s3-batch-replication-batch.md#batch-replication-manifest).
To use this command, replace the *`user input placeholders`* with your own information. Replace the IAM role `role/batch-Replication-IAM-policy` with the IAM role that you previously created. For more information, see [Configuring an IAM role for S3 Batch Replication](s3-batch-replication-policies.md).
```
aws s3control create-job --account-id 111122223333 \
--operation '{"S3ReplicateObject":{}}' \
--report '{"Bucket":"arn:aws:s3:::amzn-s3-demo-completion-report-bucket",\
"Prefix":"batch-replication-report", \
"Format":"Report_CSV_20180820","Enabled":true,"ReportScope":"AllTasks"}' \
--manifest-generator '{"S3JobManifestGenerator": {"ExpectedBucketOwner": "111122223333", \
"SourceBucket": "arn:aws:s3:::amzn-s3-demo-source-bucket", \
"EnableManifestOutput": false, "Filter": {"EligibleForReplication": true, \
"ObjectReplicationStatuses": ["NONE","FAILED"]}}}' \
--priority 1 \
--role-arn arn:aws:iam::111122223333:role/batch-Replication-IAM-policy \
--no-confirmation-required \
--region source-bucket-region
```
**Note**
You must initiate the job from the same AWS Region as the replication source bucket.
After you have successfully initiated a Batch Replication job, you receive the job ID as the response. You can monitor this job by using the following `describe-job` command. To use this command, replace the *`user input placeholders`* with your own information.
```
aws s3control describe-job --account-id 111122223333 --job-id job-id --region source-bucket-region
```
## Using the AWS CLI with a user-provided manifest
The following example creates an S3 Batch Replication job by using a user-defined manifest for AWS account `111122223333`. If the objects in your manifest are in a versioned bucket, you must specify the version IDs for the objects. Only the object with the version ID specified in the manifest will be replicated. For more information about creating a manifest, see [Specifying a manifest](batch-ops-create-job.md#specify-batchjob-manifest).
To use this command, replace the *`user input placeholders`* with your own information. Replace the IAM role `role/batch-Replication-IAM-policy` with the IAM role that you previously created. For more information, see [Configuring an IAM role for S3 Batch Replication](s3-batch-replication-policies.md).
```
aws s3control create-job --account-id 111122223333 \
--operation '{"S3ReplicateObject":{}}' \
--report '{"Bucket":"arn:aws:s3:::amzn-s3-demo-completion-report-bucket",\
"Prefix":"batch-replication-report", \
"Format":"Report_CSV_20180820","Enabled":true,"ReportScope":"AllTasks"}' \
--manifest '{"Spec":{"Format":"S3BatchOperations_CSV_20180820",\
"Fields":["Bucket","Key","VersionId"]},\
"Location":{"ObjectArn":"arn:aws:s3:::amzn-s3-demo-manifest-bucket/manifest.csv",\
"ETag":"Manifest Etag"}}' \
--priority 1 \
--role-arn arn:aws:iam::111122223333:role/batch-Replication-IAM-policy \
--no-confirmation-required \
--region source-bucket-region
```
**Note**
You must initiate the job from the same AWS Region as the replication source bucket.
After you have successfully initiated a Batch Replication job, you receive the job ID as the response. You can monitor this job by using the following `describe-job` command.
```
aws s3control describe-job --account-id 111122223333 --job-id job-id --region source-bucket-region
```