# Understanding MemoryDB replication
MemoryDB implements replication with data partitioned across up to 500 shards.
Each shard in a cluster has a single read/write primary node and up to 5 read-only replica nodes. Each primary node can sustain up to 100 MB/s. You can create a cluster with higher number of shards and lower number of replicas totaling up to 500 nodes per cluster. This cluster configuration can range from 500 shards and 0 replicas to 100 shards and 4 replicas, which is the maximum number of replicas allowed.
# Consistency
In MemoryDB, primary nodes are strongly consistent. Successful write operations are durably stored in a distributed Multi-AZ transactional logs before returning to clients. Read operations on primaries always return the most up-to-date data reflecting the effects from all prior successful write operations. Such strong consistency is preserved across primary failovers.
In MemoryDB, replica nodes are eventually consistent. Read operations from replicas (using `READONLY` command) might not always reflect the effects of the most recent successful write operations, with lag metrics published to CloudWatch. However, read operations from a single replica are sequentially consistent. Successful write operations take effect on each replica in the same order they were executed on the primary.
## Replication in a cluster
Each read replica in a shard maintains a copy of the data from the shard's primary node. Asynchronous replication mechanisms using the transaction logs are used to keep the read replicas synchronized with the primary. Applications can read from any node in the cluster. Applications can write only to the primary nodes. Read replicas enhance read scalability. Since MemoryDB stores the data in durable transaction logs, there is no risk that data will be lost. Data is partitioned across the shards in a MemoryDB cluster.
Applications use the MemoryDB cluster's *cluster endpoint* to connect with the nodes in the cluster. For more information, see [Finding connection endpoints](endpoints.md).
MemoryDB clusters are regional and can contain nodes only from one Region. To improve fault tolerance, you must provision primaries and read replicas across multiple Availability Zones within that region.
Using replication, which provides you with Multi-AZ, is strongly recommended for all MemoryDB clusters. For more information, see [Minimizing downtime in MemoryDB with Multi-AZ](autofailover.md).
# Minimizing downtime in MemoryDB with Multi-AZ
There are a number of instances where MemoryDB may need to replace a primary node; these include certain types of planned maintenance and the unlikely event of a primary node or Availability Zone failure.
The response to node failure depends on which node has failed. However, in all cases, MemoryDB ensures that no data is lost during node replacements or failover. For example, if a replica fails, the failed node is replaced and data is synced from the transaction log. If the primary node fails, a failover is triggered to a consistent replica which ensures no data is lost during failover. The writes are now served from the new primary node. The old primary node is then replaced and synced from the transaction log.
If a primary node fails on a single node shard (no replicas), MemoryDB stops accepting writes until the primary node is replaced and synced from the transaction log.
Node replacement may result in some downtime for the cluster, but if Multi-AZ is active, the downtime is minimized. The role of primary node will automatically fail over to one of the replicas. There is no need to create and provision a new primary node, because MemoryDB will handle this transparently. This failover and replica promotion ensure that you can resume writing to the new primary as soon as promotion is complete.
In case of planned node replacements initiated due to maintenance updates or service updates, be aware the planned node replacements complete while the cluster serves incoming write requests.
Multi-AZ on your MemoryDB clusters improves your fault tolerance. This is true particularly in cases where your cluster's primary nodes become unreachable or fail for any reason. Multi-AZ on MemoryDB clusters requires each shard to have more than one node, and is automatically enabled.
**Topics**
+ [Failure scenarios with Multi-AZ responses](#autofailover.scenarios)
+ [Testing automatic failover](#auto-failover-test)
## Failure scenarios with Multi-AZ responses
If Multi-AZ is active, a failed primary node fails over to an available replica. The replica is automatically synchronized with the transaction log and becomes primary, which is much faster than creating and reprovisioning a new primary node. This process usually takes just a few seconds until you can write to the cluster again.
When Multi-AZ is active, MemoryDB continually monitors the state of the primary node. If the primary node fails, one of the following actions is performed depending on the type of failure.
**Topics**
+ [Failure scenarios when only the primary node fails](#autofailover.scenarios.primaryonly)
+ [Failure scenarios when the primary node and some replicas fail](#autofailover.scenarios.primaryandeplicas)
+ [Failure scenarios when the entire cluster fails](#autofailover.scenarios.allfail)
### Failure scenarios when only the primary node fails
If only the primary node fails, a replica will automatically become primary. A replacement replica is then created and provisioned in the same Availability Zone as the failed primary.
When only the primary node fails, MemoryDB Multi-AZ does the following:
1. The failed primary node is taken offline.
1. An up-to-date replica automatically become primary.
Writes can resume as soon as the failover process is complete, typically just a few seconds.
1. A replacement replica is launched and provisioned.
The replacement replica is launched in the Availability Zone that the failed primary node was in so that the distribution of nodes is maintained.
1. The replica syncs with the transaction log.
For information about finding the endpoints of a cluster, see the following topics:
+ [Finding the Endpoint for a MemoryDB Cluster (MemoryDB API)](endpoints.md#endpoints.find.api.clusters)
### Failure scenarios when the primary node and some replicas fail
If the primary and at least one replica fails, an up-to-date replica is promoted to primary cluster. New replicas are also created and provisioned in the same Availability Zones as the failed nodes.
When the primary node and some replicas fail, MemoryDB Multi-AZ does the following:
1. The failed primary node and failed replicas are taken offline.
1. An available replica will become the primary node.
Writes can resume as soon as the failover is complete, typically just a few seconds.
1. Replacement replicas are created and provisioned.
The replacement replicas are created in the Availability Zones of the failed nodes so that the distribution of nodes is maintained.
1. All nodes sync with the transaction log.
For information about finding the endpoints of a cluster, see the following topics:
+ [Finding the Endpoint for a MemoryDB Cluster (AWS CLI)](endpoints.md#endpoints.find.cli)
+ [Finding the Endpoint for a MemoryDB Cluster (MemoryDB API)](endpoints.md#endpoints.find.api.clusters)
### Failure scenarios when the entire cluster fails
If everything fails, all the nodes are recreated and provisioned in the same Availability Zones as the original nodes.
There is no data loss in this scenario as the data was persisted in the transaction log.
When the entire cluster fails, MemoryDB Multi-AZ does the following:
1. The failed primary node and replicas are taken offline.
1. A replacement primary node is created and provisioned, syncing with the transaction log.
1. Replacement replicas are created and provisioned, syncing with the transaction log.
The replacements are created in the Availability Zones of the failed nodes so that the distribution of nodes is maintained.
For information about finding the endpoints of a cluster, see the following topics:
+ [Finding the Endpoint for a MemoryDB Cluster (AWS CLI)](endpoints.md#endpoints.find.cli)
+ [Finding the Endpoint for a MemoryDB Cluster (MemoryDB API)](endpoints.md#endpoints.find.api.clusters)
## Testing automatic failover
You can test automatic failover using the MemoryDB console, the AWS CLI, and the MemoryDB API.
When testing, note the following:
+ You can use this operation up to five times in any 24-hour period.
+ If you call this operation on shards in different clusters, you can make the calls concurrently.
+ In some cases, you might call this operation multiple times on different shards in the same MemoryDB cluster. In such cases, the first node replacement must complete before a subsequent call can be made.
+ To determine whether the node replacement is complete, check events using the MemoryDB console, the AWS CLI, or the MemoryDB API. Look for the following events related to `FailoverShard`, listed here in order of likely occurrence:
1. cluster message: `FailoverShard API called for shard `
1. cluster message: `Failover from primary node to replica node completed`
1. cluster message: `Recovering nodes `
1. cluster message: `Finished recovery for nodes `
For more information, see the following:
+ [DescribeEvents](https://docs.aws.amazon.com/memorydb/latest/APIReference/API_DescribeEvents.html) in the *MemoryDB API Reference*
+ This API is designed for testing the behavior of your application in case of MemoryDB failover. It is not designed to be an operational tool for initiating a failover to address an issue with the cluster. Moreover, in certain conditions such as large-scale operational events, AWS may block this API.
**Topics**
+ [Testing automatic failover using the AWS Management Console](#auto-failover-test-con)
+ [Testing automatic failover using the AWS CLI](#auto-failover-test-cli)
+ [Testing automatic failover using the MemoryDB API](#failovershard-test-api)
### Testing automatic failover using the AWS Management Console
Use the following procedure to test automatic failover with the console.
****
1. Sign in to the AWS Management Console and open the MemoryDB console at [https://console.aws.amazon.com/memorydb/](https://console.aws.amazon.com/memorydb/).
1. Choose the radio button to the left of the cluster you want to test. This cluster must have at least one replica node.
1. In the **Details** area, confirm that this cluster is Multi-AZ enabled. If the cluster isn't Multi-AZ enabled, either choose a different cluster or modify this cluster to enable Multi-AZ. For more information, see [Modifying a MemoryDB cluster](clusters.modify.md).
1. Choose the cluster's name.
1. On the **Shards and nodes** page, for the shard on which you want to test failover, choose the shard's name.
1. For the node, choose **Failover Primary**.
1. Choose **Continue** to fail over the primary, or **Cancel** to cancel the operation and not fail over the primary node.
During the failover process, the console continues to show the node's status as *available*. To track the progress of your failover test, choose **Events** from the console navigation pane. On the **Events** tab, watch for events that indicate your failover has started (`FailoverShard API called`) and completed (`Recovery completed`).
### Testing automatic failover using the AWS CLI
You can test automatic failover on any Multi-AZ enabled cluster using the AWS CLI operation [failover-shard](https://docs.aws.amazon.com/cli/latest/reference/memorydb/failover-shard.html).
**Parameters**
+ `--cluster-name` – Required. The cluster that is to be tested.
+ `--shard-name` – Required. The name of the shard you want to test automatic failover on. You can test a maximum of five shards in a rolling 24-hour period.
The following example uses the AWS CLI to call `failover-shard` on the shard `0001` in the MemoryDB cluster `my-cluster`.
For Linux, macOS, or Unix:
```
aws memorydb failover-shard \
--cluster-name my-cluster \
--shard-name 0001
```
For Windows:
```
aws memorydb failover-shard ^
--cluster-name my-cluster ^
--shard-name 0001
```
To track the progress of your failover, use the AWS CLI `describe-events` operation.
It will return the following JSON response:
```
{
"Events": [
{
"SourceName": "my-cluster",
"SourceType": "cluster",
"Message": "Failover to replica node my-cluster-0001-002 completed",
"Date": "2021-08-22T12:39:37.568000-07:00"
},
{
"SourceName": "my-cluster",
"SourceType": "cluster",
"Message": "Starting failover for shard 0001",
"Date": "2021-08-22T12:39:10.173000-07:00"
}
]
}
```
For more information, see the following:
+ [failover-shard](https://docs.aws.amazon.com/cli/latest/reference/memorydb/failover-shard.html)
+ [describe-events](https://docs.aws.amazon.com/cli/latest/reference/memorydb/describe-events.html)
### Testing automatic failover using the MemoryDB API
The following example calls `FailoverShard` on the shard `0003` in the cluster `memorydb00`.
**Example Testing automatic failover**
```
https://memory-db.us-east-1.amazonaws.com/
?Action=FailoverShard
&ShardName=0003
&ClusterName=memorydb00
&Version=2021-01-01
&SignatureVersion=4
&SignatureMethod=HmacSHA256
&Timestamp=20210801T192317Z
&X-Amz-Credential=
```
To track the progress of your failover, use the MemoryDB `DescribeEvents` API operation.
For more information, see the following:
+ [FailoverShard](https://docs.aws.amazon.com/memorydb/latest/APIReference/API_FailoverShard.html)
+ [DescribeEvents](https://docs.aws.amazon.com/memorydb/latest/APIReference/API_DescribeEvents.html)
# Changing the number of replicas
You can dynamically increase or decrease the number of read replicas in your MemoryDB cluster using the AWS Management Console, the AWS CLI, or the MemoryDB API. All shards must have the same number of replicas.
## Increasing the number of replicas in a cluster
You can increase the number of replicas in a MemoryDB cluster up to a maximum of five per shard. You can do so using the AWS Management Console, the AWS CLI, or the MemoryDB API.
**Topics**
+ [Using the AWS Management Console](#increase-replica-count-con)
+ [Using the AWS CLI](#increase-replica-count-cli)
+ [Using the MemoryDB API](#increase-replica-count-api)
### Using the AWS Management Console
To increase the number of replicas in a MemoryDB cluster (console), see [Adding / Removing nodes from a cluster](clusters.deletenode.md).
### Using the AWS CLI
To increase the number of replicas in a MemoryDB cluster, use the `update-cluster` command with the following parameters:
+ `--cluster-name` – Required. Identifies which cluster you want to increase the number of replicas in.
+ `--replica-configuration` – Required. Allows you to set the number of replicas. To increase the replica count, set the `ReplicaCount` property to the number of replicas that you want in this shard at the end of this operation.
**Example**
The following example increases the number of replicas in the cluster `my-cluster` to 2.
For Linux, macOS, or Unix:
```
aws memorydb update-cluster \
--cluster-name my-cluster \
--replica-configuration \
ReplicaCount=2
```
For Windows:
```
aws memorydb update-cluster ^
--cluster-name my-cluster ^
--replica-configuration ^
ReplicaCount=2
```
It returns the following JSON response:
```
{
"Cluster": {
"Name": "my-cluster",
"Status": "updating",
"NumberOfShards": 1,
"ClusterEndpoint": {
"Address": "clustercfg.my-cluster.xxxxx.memorydb.us-east-1.amazonaws.com",
"Port": 6379
},
"NodeType": "db.r6g.large",
"EngineVersion": "6.2",
"EnginePatchVersion": "6.2.6",
"ParameterGroupName": "default.memorydb-redis6",
"ParameterGroupStatus": "in-sync",
"SubnetGroupName": "my-sg",
"TLSEnabled": true,
"ARN": "arn:aws:memorydb:us-east-1:xxxxxxexamplearn:cluster/my-cluster",
"SnapshotRetentionLimit": 0,
"MaintenanceWindow": "wed:03:00-wed:04:00",
"SnapshotWindow": "04:30-05:30",
"DataTiering": "false",
"AutoMinorVersionUpgrade": true
}
}
```
To view the details of the updated cluster once its status changes from *updating* to *available*, use the following command:
For Linux, macOS, or Unix:
```
aws memorydb describe-clusters \
--cluster-name my-cluster
--show-shard-details
```
For Windows:
```
aws memorydb describe-clusters ^
--cluster-name my-cluster
--show-shard-details
```
It will return the following JSON response:
```
{
"Clusters": [
{
"Name": "my-cluster",
"Status": "available",
"NumberOfShards": 1,
"Shards": [
{
"Name": "0001",
"Status": "available",
"Slots": "0-16383",
"Nodes": [
{
"Name": "my-cluster-0001-001",
"Status": "available",
"AvailabilityZone": "us-east-1a",
"CreateTime": "2021-08-21T20:22:12.405000-07:00",
"Endpoint": {
"Address": "clustercfg.my-cluster.xxxxxx.memorydb.us-east-1.amazonaws.com",
"Port": 6379
}
},
{
"Name": "my-cluster-0001-002",
"Status": "available",
"AvailabilityZone": "us-east-1b",
"CreateTime": "2021-08-21T20:22:12.405000-07:00",
"Endpoint": {
"Address": "clustercfg.my-cluster.xxxxxx.memorydb.us-east-1.amazonaws.com",
"Port": 6379
}
},
{
"Name": "my-cluster-0001-003",
"Status": "available",
"AvailabilityZone": "us-east-1a",
"CreateTime": "2021-08-22T12:59:31.844000-07:00",
"Endpoint": {
"Address": "clustercfg.my-cluster.xxxxxx.memorydb.us-east-1.amazonaws.com",
"Port": 6379
}
}
],
"NumberOfNodes": 3
}
],
"ClusterEndpoint": {
"Address": "clustercfg.my-cluster.xxxxxx.memorydb.us-east-1.amazonaws.com",
"Port": 6379
},
"NodeType": "db.r6g.large",
"EngineVersion": "6.2",
"EnginePatchVersion": "6.2.6",
"ParameterGroupName": "default.memorydb-redis6",
"ParameterGroupStatus": "in-sync",
"SubnetGroupName": "my-sg",
"TLSEnabled": true,
"ARN": "arn:aws:memorydb:us-east-1:xxxxxxexamplearn:cluster/my-cluster",
"SnapshotRetentionLimit": 0,
"MaintenanceWindow": "wed:03:00-wed:04:00",
"SnapshotWindow": "04:30-05:30",
"ACLName": "my-acl",
"DataTiering": "false",
"AutoMinorVersionUpgrade": true
}
]
}
```
For more information about increasing the number of replicas using the CLI, see [update-cluster](https://docs.aws.amazon.com/memorydb/latest/APIReference/API_UpdateCluster.html) in the *AWS CLI Command Reference.*
### Using the MemoryDB API
To increase the number of replicas in a MemoryDB shard, use the `UpdateCluster` action with the following parameters:
+ `ClusterName` – Required. Identifies which cluster you want to increase the number of replicas in.
+ `ReplicaConfiguration` – Required. Allows you to set the number of replicas. To increase the replica count, set the `ReplicaCount` property to the number of replicas that you want in this shard at the end of this operation.
**Example**
The following example increases the number of replicas in the cluster `sample-cluster` to three. When the example is finished, there are three replicas in each shard. This number applies whether this is a MemoryDB cluster with a single shard or a MemoryDB cluster with multiple shards.
```
https://memory-db.us-east-1.amazonaws.com/
?Action=UpdateCluster
&ReplicaConfiguration.ReplicaCount=3
&ClusterName=sample-cluster
&Version=2021-01-01
&SignatureVersion=4
&SignatureMethod=HmacSHA256
&Timestamp=20210802T192317Z
&X-Amz-Credential=
```
For more information about increasing the number of replicas using the API, see [UpdateCluster](https://docs.aws.amazon.com/memorydb/latest/APIReference/API_UpdateCluster.html).
## Decreasing the number of replicas in a cluster
You can decrease the number of replicas in a cluster for MemoryDB. You can decrease the number of replicas to zero, but you can't failover to a replica if your primary node fails.
You can use the AWS Management Console, the AWS CLI or the MemoryDB API to decrease the number of replicas in a cluster.
**Topics**
+ [Using the AWS Management Console](#decrease-replica-count-con)
+ [Using the AWS CLI](#decrease-replica-count-cli)
+ [Using the MemoryDB API](#decrease-replica-count-api)
### Using the AWS Management Console
To decrease the number of replicas in a MemoryDB cluster (console), see [Adding / Removing nodes from a cluster](clusters.deletenode.md).
### Using the AWS CLI
To decrease the number of replicas in a MemoryDB cluster, use the `update-cluster` command with the following parameters:
+ `--cluster-name` – Required. Identifies which cluster you want to decrease the number of replicas in.
+ `--replica-configuration` – Required.
`ReplicaCount` – Set this property to specify the number of replica nodes you want.
**Example**
The following example uses `--replica-configuration` to decrease the number of replicas in the cluster `my-cluster` to the value specified.
For Linux, macOS, or Unix:
```
aws memorydb update-cluster \
--cluster-name my-cluster \
--replica-configuration \
ReplicaCount=1
```
For Windows:
```
aws memorydb update-cluster ^
--cluster-name my-cluster ^
--replica-configuration ^
ReplicaCount=1 ^
```
It will return the following JSON response:
```
{
"Cluster": {
"Name": "my-cluster",
"Status": "updating",
"NumberOfShards": 1,
"ClusterEndpoint": {
"Address": "clustercfg.my-cluster.xxxxxx.memorydb.us-east-1.amazonaws.com",
"Port": 6379
},
"NodeType": "db.r6g.large",
"EngineVersion": "6.2",
"EnginePatchVersion": "6.2.6",
"ParameterGroupName": "default.memorydb-redis6",
"ParameterGroupStatus": "in-sync",
"SubnetGroupName": "my-sg",
"TLSEnabled": true,
"ARN": "arn:aws:memorydb:us-east-1:xxxxxxexamplearn:cluster/my-cluster",
"SnapshotRetentionLimit": 0,
"MaintenanceWindow": "wed:03:00-wed:04:00",
"SnapshotWindow": "04:30-05:30",
"DataTiering": "false",
"AutoMinorVersionUpgrade": true
}
}
```
To view the details of the updated cluster once its status changes from *updating* to *available*, use the following command:
For Linux, macOS, or Unix:
```
aws memorydb describe-clusters \
--cluster-name my-cluster
--show-shard-details
```
For Windows:
```
aws memorydb describe-clusters ^
--cluster-name my-cluster
--show-shard-details
```
It will return the following JSON response:
```
{
"Clusters": [
{
"Name": "my-cluster",
"Status": "available",
"NumberOfShards": 1,
"Shards": [
{
"Name": "0001",
"Status": "available",
"Slots": "0-16383",
"Nodes": [
{
"Name": "my-cluster-0001-001",
"Status": "available",
"AvailabilityZone": "us-east-1a",
"CreateTime": "2021-08-21T20:22:12.405000-07:00",
"Endpoint": {
"Address": "clustercfg.my-cluster.xxxxxx.memorydb.us-east-1.amazonaws.com",
"Port": 6379
}
},
{
"Name": "my-cluster-0001-002",
"Status": "available",
"AvailabilityZone": "us-east-1b",
"CreateTime": "2021-08-21T20:22:12.405000-07:00",
"Endpoint": {
"Address": "clustercfg.my-cluster.xxxxxx.memorydb.us-east-1.amazonaws.com",
"Port": 6379
}
}
],
"NumberOfNodes": 2
}
],
"ClusterEndpoint": {
"Address": "clustercfg.my-cluster.xxxxxx.memorydb.us-east-1.amazonaws.com",
"Port": 6379
},
"NodeType": "db.r6g.large",
"EngineVersion": "6.2",
"EnginePatchVersion": "6.2.6",
"ParameterGroupName": "default.memorydb-redis6",
"ParameterGroupStatus": "in-sync",
"SubnetGroupName": "my-sg",
"TLSEnabled": true,
"ARN": "arn:aws:memorydb:us-east-1:xxxxxxexamplearn:cluster/my-cluster",
"SnapshotRetentionLimit": 0,
"MaintenanceWindow": "wed:03:00-wed:04:00",
"SnapshotWindow": "04:30-05:30",
"ACLName": "my-acl",
"DataTiering": "false",
"AutoMinorVersionUpgrade": true
}
]
}
```
For more information about decreasing the number of replicas using the CLI, see [update-cluster](https://docs.aws.amazon.com/cli/latest/reference/memorydb/update-cluster.html) in the *AWS CLI Command Reference.*
### Using the MemoryDB API
To decrease the number of replicas in a MemoryDB cluster, use the `UpdateCluster` action with the following parameters:
+ `ClusterName` – Required. Identifies which cluster you want to decrease the number of replicas in.
+ `ReplicaConfiguration` – Required. Allows you to set the number of replicas.
`ReplicaCount` – Set this property to specify the number of replica nodes you want.
**Example**
The following example uses `ReplicaCount` to decrease the number of replicas in the cluster `sample-cluster` to one. When the example is finished, there is one replica in each shard. This number applies whether this is a MemoryDB cluster with a single shard or a MemoryDB cluster with multiple shards.
```
https://memory-db.us-east-1.amazonaws.com/
?Action=UpdateCluster
&ReplicaConfiguration.ReplicaCount=1
&ClusterName=sample-cluster
&Version=2021-01-01
&SignatureVersion=4
&SignatureMethod=HmacSHA256
&Timestamp=20210802T192317Z
&X-Amz-Credential=
```
For more information about decreasing the number of replicas using the API, see [UpdateCluster](https://docs.aws.amazon.com/memorydb/latest/APIReference/API_UpdateCluster.html).