# Protecting your data with on-demand replication
Amazon FSx for OpenZFS supports on-demand data replication, enabling you to transfer snapshots of data between file systems within and across AWS Regions and accounts. You can use on-demand data replication for a variety of tasks such as:
+ Synchronizing or distributing data to your development or test environments.
+ Establishing and maintaining read replicas to provide scale-out read performance.
+ Maintaining a passive standby file system for use in disaster recovery cases.
With on-demand data replication, Amazon FSx automatically establishes and maintains network connectivity between file systems to handle interruptions and resume data transfer as needed. Amazon FSx also encrypts data in transit and at rest and integrates with AWS RAM to authorize accesss to volumes for data replication across AWS accounts. For more information, see [Shareable AWS resources](https://docs.aws.amazon.com/ram/latest/userguide/shareable.html#shareable-fsx) in the *AWS RAM User Guide*.
On-demand data replication is available for all deployment types in AWS Regions where Amazon FSx for OpenZFS is available. For more information, see [Availability by AWS Region](available-aws-regions.md).
**Topics**
+ [Prerequisites for using on-demand data replication](#access-data-replication)
+ [Performance considerations for on-demand data replication](#on-demand-replication-performance)
+ [Using on-demand data replication](#how-to-use-data-replication)
+ [Monitoring progress of on-demand data replication](#how-to-monitor-data-replication)
+ [Setting up ongoing periodic data replication](ongoing-periodic-data-replication.md)
## Prerequisites for using on-demand data replication
Before using on-demand data replication, make sure that you have met the following prerequisites.
+ Single-AZ 1 (non-HA and HA) file systems must have a provisioned throughput capacity of 256 MBps or above. It is also recommended that Single-AZ 1 (non-HA and HA) file systems have a provisioned SSD IOPS level of 6,000 or above.
+ Single-AZ 2 (non-HA and HA) and Multi-AZ (HA) file systems must have a provisioned throughput capacity of 160 MBps or above. It is also recommended that Single-AZ 2 (non-HA) and Multi-AZ (HA) file systems have a provisioned SSD IOPS level of 6,000 or above.
+ Users or roles must have permission to take the [CreateVolume](https://docs.aws.amazon.com/fsx/latest/APIReference/API_CreateVolume.html) and [CopySnapshotAndUpdateVolume](https://docs.aws.amazon.com/fsx/latest/APIReference/API_CopySnapshotAndUpdateVolume.html) actions in an AWS account. You can control these permissions by using AWS Identity and Access Management (IAM) policies. For more information, see [Actions, resources, and condition keys for Amazon FSx](https://docs.aws.amazon.com/service-authorization/latest/reference/list_amazonfsx.html#amazonfsx-actions-as-permissions.html) in the *Service Authorization Reference*.
+ To replicate data across file systems in different AWS accounts, the source account must have, at minimum, permission to take the **fsx:PutResourcePolicy**, **fsx:GetResourcePolicy**, and **fsx:DeleteResourcePolicy** actions. The source account must also have permissions to share resources on AWS RAM. To grant these permissions, you can directly attach the [AmazonFSxFullAccess](security-iam-awsmanpol.md#security-iam-awsmanpol-AmazonFSxFullAccess), [AmazonFSxConsoleFullAccess](security-iam-awsmanpol.md#security-iam-awsmanpol-AmazonFSxConsoleFullAccess), and [AWSResourceAccessManagerFullAccess](https://docs.aws.amazon.com/ram/latest/userguide/security-iam-managed-policies.html#security-iam-managed-policies-AWSResourceAccessManagerFullAccess) AWS managed policies to your IAM roles, groups, and users. The destination account must have the [AWSResourceAccessManagerResourceShareParticipantAccess](https://docs.aws.amazon.com/ram/latest/userguide/security-iam-managed-policies.html#security-iam-managed-policies-AWSResourceAccessManagerResourceShareParticipantAccess) AWS managed policy attached to its IAM roles, groups, and users.
## Performance considerations for on-demand data replication
On-demand data replication shares provisioned throughput with other file system clients. To accommodate data replication activity without impacting other workloads, we recommend provisioning twice the level of throughput capacity that your workload normally needs. You can use Amazon CloudWatch metrics with FSx for OpenZFS to monitor your file system’s performance utilization and scale up your file system’s performance as needed to avoid slowing down your ongoing workloads. For more information, see [Using Amazon FSx for OpenZFS CloudWatch metrics](how_to_use_metrics.md).
## Using on-demand data replication
On-demand data replication only transfers data from the indicated source snapshot, which does not include data from child volumes. To transfer data from child volumes, you must initiate additional data replication jobs using source snapshots from the child volumes.
Each file system can only be used as the source file system or the destination file system for one on-demand data replication task at a time. You must wait until the first on-demand replication task is completed or cancelled before initating another request. You can only have a maximum of twenty concurrent cross-file system replication jobs per account, per AWS Region.
### Replicating data across file systems on the same account
You can create or update a replica volume across file systems that are on the same AWS account by using the Amazon FSx Console, API, or CLI.
#### To update a volume from a snapshot (Console)
1. Open the Amazon FSx console at [https://console.aws.amazon.com/fsx/](https://console.aws.amazon.com/fsx/).
1. In the left navigation pane, choose **Volumes**, and then choose the volume that you would like to use as your destination volume.
1. For **Actions**, choose **Update volume with snapshot**. The **Copy snapshot and update volume** panel displays.
1. Choose the source region of the snapshot
1. Choose the snapshot that you would like to update the volume from.
1. For **Source snapshot copy strategy**, choose **Incremental copy** or **Full copy**. An incremental copy returns the destination volume to the most recent common ancestor that it shares with the source volume and then updates the destination volume, transferring only the data that is not already included in the most recent common ancestor. A full copy will remove any clones, snapshots, and intermediate data on the destination volume and transfer all of the data from the source volume. During incremental copy, your destination volume will be read-only. During full copy, your destination volume will be unmounted and automatically remounted after the transfer is completed.
1. If the destination volume has any **intermediate clones**, **dependent snapshots**, or **intermediate data**, select the checkboxes to delete them. If you are using incremental copy, you must delete all descendent data for the update to succeed.
1. Choose **Update** to update the volume.
#### To update a volume from a snapshot (CLI)
+ To update an FSx for OpenZFS volume with a snapshot, use the [copy-snapshot-and-update-volume](https://docs.aws.amazon.com/cli/latest/reference/fsx/copy-snapshot-and-update-volume.html) CLI command, or the equivalent [CopySnapshotAndUpdateVolume](https://docs.aws.amazon.com/fsx/latest/APIReference/API_CopySnapshotAndUpdateVolume.html) API command, and specify the following properties:
+ `--volume-id` – The ID of the volume that you would like to update.
+ `--source-snapshot-arn` – The ARN of the source snapshot.
+ `--options` – Any intermediate clones, dependent snapshots, or intermediate data that need to be deleted. Valid values are `DELETE_INTERMEDIATE_SNAPSHOTS`, `DELETE_CLONED_VOLUMES`, and `DELETE_INTERMEDIATE_DATA`.
+ `--copy-strategy` – Strategy used to copy data from the source volume. Value values are `FULL_COPY` and `INCREMENTAL_COPY`.
The following example shows how to update a volume with a snapshot using incremental copy and deleting all intermediate clones, dependent snapshots, and intermediate data.
```
aws fsx copy-snapshot-and-update-volume \
--volume-id fsvol-1234567890abcdef0 \
--source-snapshot-arn arn:aws:fsx:555555555555:snapshot/fsvol-1234567890abcdef0/fsvolsnap-021345abcdef6789\
--options DELETE_INTERMEDIATE_SNAPSHOTS DELETE_CLONED_VOLUMES DELETE_INTERMEDIATE_DATA\
--copy-strategy INCREMENTAL_COPY
```
The example above returns the following response.
```
{
"VolumeId": "fsvol-1234567890abcdef0",
"Lifecycle": "AVAILABLE",
"AdministrativeActions": [
{
"AdministrativeActionType": "VOLUME_UPDATE_WITH_SNAPSHOT",
"FailureDetails": {
"Message": "string"
},
"ProgressPercent": 80,
"RequestTime": 2023-11-03T09:26:55-07:00,
"Status": "IN_PROGRESS",
"TargetVolumeValues": {
"OpenZFSConfiguration": {
"RecordSizeKiB": 128,
"DataCompressionType": "ZSTD",
"DeleteIntermediateSnaphots": false,
"DeleteClonedVolumes": false,
"DeleteIntermediateData": true,
"SourceSnapshotARN": "arn:aws:fsx:us-east-1:854733241892:snapshot/fsvol-018a3d05b4d9fc768/fsvolsnap-03b43bd1942a51637",
"DestinationSnapshot": "fsvolsnap-0f753e290e20cc974" }"
}
}]
}
```
### Replicating data across file systems on different AWS accounts using AWS RAM
FSx for OpenZFS integrates with AWS Resource Access Manager (RAM) to allow you to replicate data across file systems that are on different AWS accounts. In the AWS Resource Access Manager (RAM) console, the owner of the source account must first enable resource sharing, and then share the source FSx for OpenZFS volume with the destination account. For more information on enabling and creating a resource share, see [Enable resource sharing within AWS Organizations](https://docs.aws.amazon.com/ram/latest/userguide/getting-started-sharing.html#getting-started-sharing-orgs) and [Creating a resource share](https://docs.aws.amazon.com/ram/latest/userguide/working-with-sharing-create.html) in the *AWS RAM User Guide*.
You will receive a shared resource invitation when the source volume has been shared with your account. Once you accept the invitation, all snapshots associated with the source volume will appear in the list of snapshots that you can replicate to a volume in the FSx for OpenZFS console. For more information, see [To update a volume from a snapshot (Console)](#update-volume-from-snapshot-console). After you’ve created a replica volume, you can continue to update it with any of the subsequent snapshots in the source volume, as long as the source volume continues to be shared.
## Monitoring progress of on-demand data replication
You can monitor the progress of your data replication using the AWS Management Console on the **Volume details** page. When you initiate a replication task, the destination snapshot will enter the **CREATING** state. Once the data transfer is complete, the destination snapshot will become **AVAILABLE**.
You can also use the AWS CLI or Amazon FSx API to track more detailed progress of your replication by using the [describe-volumes](https://docs.aws.amazon.com/cli/latest/reference/fsx/describe-volumes.html) AWS CLI command or the [DescribeVolumes](https://docs.aws.amazon.com/fsx/latest/APIReference/API_DescribeVolumes.html) API operation. to display the `AdministrativeActions` for the destination volume. The `AdministrativeActions` array lists the 10 most recent update actions for each administrative action type. When you initiate an on-demand data replication, a `VOLUME_UPDATE_WITH_SNAPSHOT` action is generated. Progress will be reported using the `ProgressPercent` property.
The following example shows the response for an incremental copy on-demand data replication task.
```
{
"VolumeId": "fsvol-1234567890abcdef0",
"Lifecycle": "AVAILABLE",
"AdministrativeActions": [
{
"AdministrativeActionType": "VOLUME_UPDATE_WITH_SNAPSHOT",
"FailureDetails": {
"Message": "string"
},
"ProgressPercent": 80,
"RequestTime": 2023-11-03T09:26:55-07:00,
"Status": "IN_PROGRESS",
"TotalTransferBytes": 107483152368,
"RemainingTransferBytes": 0
"TargetVolumeValues": {
"OpenZFSConfiguration": {
"SourceSnapshotARN": "stringarn:aws:fsx:555555555555:snapshot/fsvol-1234567890abcdef0/fsvolsnap-021345abcdef6789",
"DestinationSnapshot": "fsvolsnap-021345abcdef6789"
}
}
}]
}
```
When Amazon FSx processes the request successfully, the status changes to `COMPLETED`. If the on-demand data replication task fails, the status changes to `FAILED`, and the `FailureDetails` property provides information about the failure.
# Setting up ongoing periodic data replication
With ongoing periodic data replication, you can set up a schedule that automatically takes a snapshot of a source volume and performs an incremental replication of that snapshot on a destination volume at a certain interval, for example every 15 minutes. You can schedule ongoing periodic data replication between two volumes on FSx for OpenZFS file systems within or across AWS Regions and accounts by using the solution provided in this section.
**Topics**
+ [Architecture overview](#architecture-overview-periodic-replication)
+ [Required permissions](#required-permissions-periodic-replication)
+ [Step 1: Initializing and deploying the application](#step-1-periodic-replication)
+ [Step 2: Monitoring periodic replication](#step2-periodic-replication)
## Architecture overview
Deploying this solution builds the following resources in the AWS Cloud.
![\[Architecture of the periodic data replication solution.\]](http://docs.aws.amazon.com/fsx/latest/OpenZFSGuide/images/openzfs-periodic-data-replication-architecture.PNG)
The diagram illustrates the following periodic replication workflow.
1. AWS Serverless Application Model (SAM) automates the deployment of the FSx for OpenZFS periodic replication solution. For more information about AWS SAM, see [What is the AWS Serverless Application Model (AWS SAM)?](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/what-is-sam.html) in the *AWS Serverless Application Model* User Guide.
1. The SAM template deploys an Amazon EventBridge scheduler, an AWS Lambda function, an Amazon SNS queue, and an IAM role. The IAM role gives the Lambda function permission to call the necessary Amazon FSx API operations.
1. The EventBridge scheduler runs on a schedule you specify as a cron pattern during the initial deployment. For more information about cron patterns, see [Creating an Amazon EventBridge rule that runs on a schedule](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-create-rule-schedule.html) in the *Amazon EventBridge* User Guide. The scheduler invokes a Lambda function that calls the Amazon FSx `CreateSnapshot` API operation to create a snapshot of the source volume.
1. Once the snapshot is available, the Lambda function calls the Amazon FSx `CopySnapshotAndUpdateVolume` API operation to start replicating the source snapshot data to the destination volume.
1. The Lambda function sends a notification message to the Amazon SNS queue when replication starts, if you choose to be notified during the initial deployment. A notification is always sent when a snapshot cannot be created or the replication cannot be initiated.
## Required permissions
The following permissions are required to use the custom snapshot schedule CloudFormation template.
+ `AmazonS3FullAccess`
+ `AWSCloudFormationFullAccess`
+ `AmazonEventBridgeFullAccess`
+ `IAMFullAccess`
+ `AmazonSNSFullAccess`
+ `AWSKeyManagementServicePowerUser`
+ `AWSLambda_FullAccess`
For more information about using IAM to set up permissions, see [How Amazon FSx for OpenZFS works with IAM](security_iam_service-with-iam.md).
## Step 1: Initializing and deploying the application
The following procedure configures and deploys the periodic replication solution. It takes about five minutes to deploy. Before you begin this step, make sure that you have the ID of the source and destination volumes that you would like to initiate the replication between. For more information on these resources, see [Creating an Amazon FSx for OpenZFS volume](creating-volumes.md), [Creating a snapshot](snapshots-openzfs.md#creating-snapshots), and [Using on-demand data replication](on-demand-replication.md#how-to-use-data-replication).
**Note**
Implementing this solution incurs billing for the associated AWS services. For more information, see the pricing details pages for those services.
**To launch the periodic replication solution stack**
1. Follow the instructions on the [Replicate FSx-OpenZFS volumes across file systems](https://serverlessland.com/patterns/eventbridge-lambda-fsx-openzfs-periodic-replication) page to download the serverless pattern.
1. For **Parameters**, review the following parameters for the template and modify them for the needs of your periodic replication. This solution uses the following default values.
[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/fsx/latest/OpenZFSGuide/ongoing-periodic-data-replication.html)
1. In the AWS SAM CLI, run the following command to deploy the resources specified in the SAM template.
```
sam deploy --guided \
--stack-name fsxz-periodic-replication \
--template-file fsx-openzfs-periodic-replication.yaml \
--capabilities CAPABILITY_AUTO_EXPAND CAPABILITY_IAM CAPABILITY_NAMED_IAM
```
You will be asked if you would like to update any parameters.
1. Choose **Enter** to deploy the template.
## Step 2: Monitoring periodic replication
You can monitor the status of the periodic replication workflow using the Amazon FSx Console, AWS CLI, and API. For more information on how to monitor periodic replication using the Amazon FSx Console, see [Monitoring progress of on-demand data replication](on-demand-replication.md#how-to-monitor-data-replication).
To use the AWS CLI or API to track the progress of your replication, call the [describe-volumes](https://docs.aws.amazon.com/cli/latest/reference/fsx/describe-volumes.html) CLI command or the [DescribeVolumes](https://docs.aws.amazon.com/fsx/latest/APIReference/API_DescribeVolumes.html) API operation to view the `AdministrativeActions` array for the destination volume. The following example shows the response for an incremental copy on-demand data replication task.
```
"AdministrativeActions": [
{
"AdministrativeActionType": "VOLUME_UPDATE_WITH_SNAPSHOT",
"ProgressPercent": 100,
"RequestTime": 1699997847.438,
"Status": "COMPLETED",
"TargetVolumeValues": {
"OpenZFSConfiguration": {
"RecordSizeKiB": 128,
"DataCompressionType": "ZSTD",
"DeleteIntermediateSnaphots": true,
"DeleteClonedVolumes": false,
"DeleteIntermediateData": true,
"SourceSnapshotARN": "arn:aws:fsx:us-east-1:609492434915:snapshot/fsvol-0e1ab09de954a352f/fsvolsnap-01dda47dcbb24ddd0",
"DestinationSnapshot": "fsvolsnap-0afef62088c7c9060"
}
},
"TotalTransferBytes": 44144,
"RemainingTransferBytes": 0
},
```