# Amazon DocumentDB High availability and replication
You can achieve high availability and read scaling in Amazon DocumentDB (with MongoDB compatibility) by using replica instances. A single Amazon DocumentDB cluster supports a single primary instance and up to 15 replica instances. These instances can be distributed across Availability Zones within the cluster's Region. The primary instance accepts read and write traffic, and replica instances accept only read requests.
The cluster volume is made up of multiple copies of the data for the cluster. However, the data in the cluster volume is represented as a single, logical volume to the primary instance and to Amazon DocumentDB replicas in the cluster. Replica instances are eventually consistent. They return query results with minimal replica lag—usually much less than 100 milliseconds after the primary instance has written an update. Replica lag varies depending on the rate of database change. That is, during periods in which a large number of write operations occur for the database, you might see an increase in the replica lag.
## Read scaling
Amazon DocumentDB replicas work well for read scaling because they are fully dedicated to read operations on your cluster volume. Write operations are managed by the primary instance. The cluster volume is shared among all instances in your cluster. Therefore, you don't have to replicate and maintain a copy of the data for each Amazon DocumentDB replica.
## High availability
When you create an Amazon DocumentDB cluster, depending upon the number of Availability Zones in the subnet group (there must be at least two), Amazon DocumentDB provisions instances across the Availability Zones. When you create instances in the cluster, Amazon DocumentDB automatically distributes the instances across the Availability Zones in a subnet group to balance the cluster. This action also prevents all instances from being located in the same Availability Zone.
**Example**
To illustrate the point, consider an example where you create a cluster that has a subnet group with three Availability Zones: *AZ1*, *AZ2*, and *AZ3*.
When the first instance in the cluster is created, it is the primary instance and is located in one of the Availability Zones. In this example, it's in *AZ1*. The second instance created is a replica instance and is located in one of the other two Availability Zones, say *AZ2*. The third instance created is a replica instance and is located in the remaining Availability Zone, *AZ3*. If you create more instances, they are distributed across the Availability Zones so that you achieve balance in the cluster.
If a failure occurs in the primary instance (AZ1), a failover is triggered, and one of the existing replicas is promoted to primary. When the old primary recovers, it becomes a replica in the same Availability Zone in which it was provisioned (AZ1). When you provision a three-instance cluster, Amazon DocumentDB continues to preserve that three-instance cluster. Amazon DocumentDB automatically handles detection, failover, and recovery of instance failures without any manual intervention.
When Amazon DocumentDB performs a failover and recovers an instance, the recovered instance remains in the Availability Zone in which it was originally provisioned. However, the role of the instance might change from primary to replica. Doing this prevents the scenario in which a series of failovers could result in all instances being in the same Availability Zone.
You can specify Amazon DocumentDB replicas as failover targets. That is, if the primary instance fails, the specified Amazon DocumentDB replica or replica from a tier is promoted to the primary instance. There is a brief interruption during which read and write requests made to the primary instance fail with an exception. If your Amazon DocumentDB cluster doesn't include any Amazon DocumentDB replicas, when the primary instance fails, it is re-created. Promoting an Amazon DocumentDB replica is much faster than re-creating the primary instance.
For high availability scenarios, we recommend that you create one or more Amazon DocumentDB replicas. These replicas should be of the same instance class as the primary instance and in different Availability Zones for your Amazon DocumentDB cluster.
For more information, see the following:
+ [Understanding Amazon DocumentDB cluster fault tolerance](db-cluster-fault-tolerance.md)
+ [Amazon DocumentDB Failover](failover.md)
+ [Controlling the failover target](failover.md#failover-target_control)
### High availability with global clusters
For high availability across multiple AWS Regions, you can set up [Amazon DocumentDB global clusters](https://docs.aws.amazon.com/documentdb/latest/developerguide/global-clusters.html). Each global cluster spans multiple regions, enabling low latency global reads and disaster recovery from outages across an AWS Region. Amazon DocumentDB automatically handles replicating all data and updates from the primary region to each of the secondary regions.
## Adding replicas
The first instance added to the cluster is the primary instance. Every instance that is added after the first instance is a replica instance. A cluster can have up to 15 replica instances in addition to the primary.
When you create a cluster using the AWS Management Console, a primary instance is automatically created at the same time. To create a replica at the same time as you create the cluster and the primary instance, choose **Create replica in different zone**. For more information, see step 4.d in [Creating an Amazon DocumentDB cluster](db-cluster-create.md). To add more replicas to an Amazon DocumentDB cluster, see [Adding an Amazon DocumentDB instance to a cluster](db-instance-add.md).
When using the AWS CLI to create your cluster, you must explicitly create your primary and replica instances. For more information, see the "Using the AWS CLI" section in the following topics:
+ [Creating an Amazon DocumentDB cluster](db-cluster-create.md)
+ [Adding an Amazon DocumentDB instance to a cluster](db-instance-add.md)
# Amazon DocumentDB Failover
In certain cases, such as certain types of planned maintenance, or in the unlikely event of a primary node or Availability Zone failure, Amazon DocumentDB (with MongoDB compatibility) detects the failure and replaces the primary node. During a failover, write down time is minimized. This is because the role of primary node fails over to one of the read replicas instead of having to create and provision a new primary node. This failure detection and replica promotion ensure that you can resume writing to the new primary as soon as promotion is complete.
For failover to function, your cluster must have at least two instances — a primary and at least one replica instance.
**Note**
This topic only applies to original Amazon DocumentDB instance-based clusters. It does not apply to elastic or global clusters.
## Controlling the failover target
Amazon DocumentDB provides you with failover tiers as a means to control which replica instance is promoted to primary when a failover occurs.
**Failover Tiers**
Each replica instance is associated with a failover tier (0–15). When a failover occurs due to maintenance or an unlikely hardware failure, the primary instance fails over to a replica with the highest priority (the lowest numbered tier). If multiple replicas have the same priority tier, the primary fails over to that tier's replica that is the closest in size to the previous primary.
By setting the failover tier for a group of select replicas to `0` (the highest priority), you can ensure that a failover will promote one of the replicas in that group. You can effectively prevent specific replicas from being promoted to primary in case of a failover by assigning a low-priority tier (high number) to these replicas. This is useful in cases where specific replicas are receiving heavy use by an application and failing over to one of them would negatively impact a critical application.
You can set the failover tier of an instance when you create it or later by modifying it. Setting an instance failover tier by modifying the instance does not trigger a failover. For more information see the following topics:
+ [Adding an Amazon DocumentDB instance to a cluster](db-instance-add.md)
+ [Modifying an Amazon DocumentDB instance](db-instance-modify.md)
When manually initiating a failover, you have two means to control which replica instance is promoted to primary: the failover tiers as previously described, and the `--target-db-instance-identifier` parameter.
**--`target-db-instance-identifier`**
For testing, you can force a failover event using the `failover-db-cluster` operation. You can use the `--target-db-instance-identifier` parameter to specify which replica to promote to primary. Using the `--target-db-instance-identifier` parameter supersedes the failover priority tier. If you do not specify the `--target-db-instance-identifier` parameter, the primary failover is in accordance with the failover priority tier.
## What happens during a failover
Failover is automatically handled by Amazon DocumentDB so that your applications can resume database operations as quickly as possible without administrative intervention.
+ If you have an Amazon DocumentDB replica instance in the same or different Availability Zone when failing over: Amazon DocumentDB flips the canonical name record (CNAME) for your instance to point at the healthy replica, which is, in turn, promoted to become the new primary. Failover typically completes within 30 seconds from start to finish.
+ If you don't have an Amazon DocumentDB replica instance (for example, a single instance cluster): Amazon DocumentDB will attempt to create a new instance in the same Availability Zone as the original instance. This replacement of the original instance is done on a best-effort basis and may not succeed if, for example, there is an issue that is broadly affecting the Availability Zone.
Your application should retry database connections in the event of a connection loss.
## Testing failover
A failover for a cluster promotes one of the Amazon DocumentDB replicas (read-only instances) in the cluster to be the primary instance (the cluster writer).
When the primary instance fails, Amazon DocumentDB automatically fails over to an Amazon DocumentDB replica, if one exists. You can force a failover when you want to simulate a failure of a primary instance for testing. Each instance in a cluster has its own endpoint address. Therefore, you need to clean up and re-establish any existing connections that use those endpoint addresses when the failover is complete.
To force a failover, use the `failover-db-cluster` operation with these parameters.
+ `--db-cluster-identifier`—Required. The name of the cluster to fail over.
+ `--target-db-instance-identifier`—Optional. The name of the instance to be promoted to the primary instance.
**Example**
The following operation forces a failover of the `sample-cluster` cluster. It does not specify which instance to make the new primary instance, so Amazon DocumentDB chooses the instance according to failover tier priority.
For Linux, macOS, or Unix:
```
aws docdb failover-db-cluster \
--db-cluster-identifier sample-cluster
```
For Windows:
```
aws docdb failover-db-cluster ^
--db-cluster-identifier sample-cluster
```
The following operation forces a failover of the `sample-cluster` cluster, specifying that `sample-cluster-instance` is to be promoted to the primary role. (Notice `"IsClusterWriter": true` in the output.)
For Linux, macOS, or Unix:
```
aws docdb failover-db-cluster \
--db-cluster-identifier sample-cluster \
--target-db-instance-identifier sample-cluster-instance
```
For Windows:
```
aws docdb failover-db-cluster ^
--db-cluster-identifier sample-cluster ^
--target-db-instance-identifier sample-cluster-instance
```
Output from this operation looks something like the following (JSON format).
```
{
"DBCluster": {
"HostedZoneId": "Z2SUY0A1719RZT",
"Port": 27017,
"EngineVersion": "3.6.0",
"PreferredMaintenanceWindow": "thu:04:05-thu:04:35",
"BackupRetentionPeriod": 1,
"ClusterCreateTime": "2018-06-28T18:53:29.455Z",
"AssociatedRoles": [],
"DBSubnetGroup": "default",
"MasterUsername": "master-user",
"Engine": "docdb",
"ReadReplicaIdentifiers": [],
"EarliestRestorableTime": "2018-08-21T00:04:10.546Z",
"DBClusterIdentifier": "sample-cluster",
"ReaderEndpoint": "sample-cluster.node.us-east-1.docdb.amazonaws.com",
"DBClusterMembers": [
{
"DBInstanceIdentifier": "sample-cluster-instance",
"DBClusterParameterGroupStatus": "in-sync",
"PromotionTier": 1,
"IsClusterWriter": true
},
{
"DBInstanceIdentifier": "sample-cluster-instance-00",
"DBClusterParameterGroupStatus": "in-sync",
"PromotionTier": 1,
"IsClusterWriter": false
},
{
"DBInstanceIdentifier": "sample-cluster-instance-01",
"DBClusterParameterGroupStatus": "in-sync",
"PromotionTier": 1,
"IsClusterWriter": false
}
],
"AvailabilityZones": [
"us-east-1b",
"us-east-1c",
"us-east-1a"
],
"DBClusterParameterGroup": "default.docdb3.6",
"Endpoint": "sample-cluster.node.us-east-1.docdb.amazonaws.com",
"IAMDatabaseAuthenticationEnabled": false,
"AllocatedStorage": 1,
"LatestRestorableTime": "2018-08-22T21:57:33.904Z",
"PreferredBackupWindow": "00:00-00:30",
"StorageEncrypted": false,
"MultiAZ": true,
"Status": "available",
"DBClusterArn": "arn:aws:rds:us-east-1:123456789012:cluster:sample-cluster",
"VpcSecurityGroups": [
{
"Status": "active",
"VpcSecurityGroupId": "sg-12345678"
}
],
"DbClusterResourceId": "cluster-ABCDEFGHIJKLMNOPQRSTUVWXYZ"
}
}
```
## Replication lag
Replication lag is typically 50ms or less. The most common reasons for increased replica lag are:
+ A high write rate on the primary that causes the read replicas to fall behind the primary.
+ Contention on the read replicas between long running queries (e.g., large sequential scans, aggregation queries) and incoming write replication.
+ Very large number of concurrent queries on the read replicas.
To minimize replication lag, try these troubleshooting techniques:
+ If you have a high write rate or high CPU utilization, we recommend that you scale up the instances in your cluster.
+ If there are long running queries on your read replicas, and very frequent updates to the documents being queried, consider altering your long running queries, or running them against the primary/write replica to avoid contention on the read replicas.
+ If there is a very large number of concurrent queries or high CPU utilization only on the read replicas, another option is to scale out the number of read replicas to spread out the workload.
+ Because replication lag is a result of high write throughput and long running queries, we recommend troubleshooting the replication lag by utilizing the DBClusterReplicaLagMaximum CW metric in combination with the slow query logger and `WriteThroughput`/`WriteIOPS` metrics.
In general, we recommend that all your replicas are of the same instance type, so that a cluster failover will not cause a degradation in performance.
If you are choosing between scaling up and scaling out (eg. six smaller instances vs three larger instances), we generally recommend trying to scale up first (larger instances) before scaling out, as you will get a larger buffer cache per DB instance.
Proactively, you should set a replication lag alarm and set its threshold to a value that you feel is the upper bound for how far behind (or “stale”) your data on replica instances can be before it starts affecting the functionality of your application. In general, we would advise that the replication lag threshold be exceeded for several data points before alarming, due to transient workloads.
**Note**
In addition, we recommend that you set another alarm for replication lags that exceed 10 seconds. If you surpass this threshold for multiple data points, we recommend that you scale up your instances or reduce your write throughput on the primary instance.