Minimizing downtime in ElastiCache by using Multi-AZ with Valkey and Redis OSS

There are a number of instances where ElastiCache for Valkey and Redis OSS may need to replace a primary node; these include certain types of planned maintenance and the unlikely event of a primary node or Availability Zone failure.

This replacement results in some downtime for the cluster, but if Multi-AZ is enabled, the downtime is minimized. The role of primary node will automatically fail over to one of the read replicas. There is no need to create and provision a new primary node, because ElastiCache will handle this transparently. This failover and replica promotion ensure that you can resume writing to the new primary as soon as promotion is complete.

ElastiCache also propagates the Domain Name Service (DNS) name of the promoted replica. It does so because then if your application is writing to the primary endpoint, no endpoint change is required in your application. If you are reading from individual endpoints, make sure that you change the read endpoint of the replica promoted to primary to the new replica's endpoint.

In case of planned node replacements initiated due to maintenance updates or self-service updates, be aware of the following:

You can enable Multi-AZ using the ElastiCache Management Console, the AWS CLI, or the ElastiCache API.

Enabling ElastiCache Multi-AZ on your Valkey or Redis OSS cluster (in the API and CLI, replication group) improves your fault tolerance. This is true particularly in cases where your cluster's read/write primary cluster becomes unreachable or fails for any reason. Multi-AZ is only supported on Valkey and Redis OSS clusters with more than one node in each shard.

Enabling Multi-AZ

You can enable Multi-AZ when you create or modify a cluster (API or CLI, replication group) using the ElastiCache console, AWS CLI, or the ElastiCache API.

You can enable Multi-AZ only on Valkey or Redis OSS (cluster mode disabled) clusters that have at least one available read replica. Clusters without read replicas do not provide high availability or fault tolerance. For information about creating a cluster with replication, see Creating a Valkey or Redis OSS replication group. For information about adding a read replica to a cluster with replication, see Adding a read replica for Valkey or Redis OSS (Cluster Mode Disabled).

Enabling Multi-AZ (Console)

You can enable Multi-AZ using the ElastiCache console when you create a new Valkey or Redis OSS cluster or by modifying an existing cluster with replication.

Multi-AZ is enabled by default on Valkey or Redis OSS (cluster mode enabled) clusters.

Enabling Multi-AZ when creating a cluster using the ElastiCache console

For more information on this process, see Creating a Valkey (cluster mode disabled) cluster (Console). Be sure to have one or more replicas and enable Multi-AZ.

Enabling Multi-AZ on an existing cluster (Console)

For more information on this process, see Modifying a Cluster Using the ElastiCache AWS Management Console.

Enabling Multi-AZ (AWS CLI)

The following code example uses the AWS CLI to enable Multi-AZ for the replication group redis12.

For Linux, macOS, or Unix:

aws elasticache modify-replication-group \
    --replication-group-id redis12 \
    --automatic-failover-enabled \
    --multi-az-enabled \
    --apply-immediately

For Windows:

aws elasticache modify-replication-group ^
    --replication-group-id redis12 ^
    --automatic-failover-enabled ^
    --multi-az-enabled ^
    --apply-immediately

The JSON output from this command should look something like the following.

{
    "ReplicationGroup": {
        "Status": "modifying", 
        "Description": "One shard, two nodes", 
        "NodeGroups": [
            {
                "Status": "modifying", 
                "NodeGroupMembers": [
                    {
                        "CurrentRole": "primary", 
                        "PreferredAvailabilityZone": "us-west-2b", 
                        "CacheNodeId": "0001", 
                        "ReadEndpoint": {
                            "Port": 6379, 
                            "Address": "redis12-001.v5r9dc.0001.usw2.cache.amazonaws.com"
                        }, 
                        "CacheClusterId": "redis12-001"
                    }, 
                    {
                        "CurrentRole": "replica", 
                        "PreferredAvailabilityZone": "us-west-2a", 
                        "CacheNodeId": "0001", 
                        "ReadEndpoint": {
                            "Port": 6379, 
                            "Address": "redis12-002.v5r9dc.0001.usw2.cache.amazonaws.com"
                        }, 
                        "CacheClusterId": "redis12-002"
                    }
                ], 
                "NodeGroupId": "0001", 
                "PrimaryEndpoint": {
                    "Port": 6379, 
                    "Address": "redis12.v5r9dc.ng.0001.usw2.cache.amazonaws.com"
                }
            }
        ], 
        "ReplicationGroupId": "redis12", 
        "SnapshotRetentionLimit": 1, 
        "AutomaticFailover": "enabling", 
        "MultiAZ": "enabled", 
        "SnapshotWindow": "07:00-08:00", 
        "SnapshottingClusterId": "redis12-002", 
        "MemberClusters": [
            "redis12-001", 
            "redis12-002"
        ], 
        "PendingModifiedValues": {}
    }
}

For more information, see these topics in the AWS CLI Command Reference:

Enabling Multi-AZ (ElastiCache API)

The following code example uses the ElastiCache API to enable Multi-AZ for the replication group redis12.

https://elasticache.us-west-2.amazonaws.com/
    ?Action=ModifyReplicationGroup
    &ApplyImmediately=true
    &AutoFailover=true
    &MultiAZEnabled=true
    &ReplicationGroupId=redis12
    &Version=2015-02-02
    &SignatureVersion=4
    &SignatureMethod=HmacSHA256
    &Timestamp=20140401T192317Z
    &X-Amz-Credential=<credential>

For more information, see these topics in the ElastiCache API Reference:

Failure scenarios with Multi-AZ responses

Before the introduction of Multi-AZ, ElastiCache detected and replaced a cluster's failed nodes by recreating and reprovisioning the failed node. If you enable Multi-AZ, a failed primary node fails over to the replica with the least replication lag. The selected replica is automatically promoted to primary, which is much faster than creating and reprovisioning a new primary node. This process usually takes just a few seconds until you can write to the cluster again.

When Multi-AZ is enabled, ElastiCache continually monitors the state of the primary node. If the primary node fails, one of the following actions is performed depending on the type of failure.

Failure scenarios when only the primary node fails

If only the primary node fails, the read replica with the least replication lag is promoted to primary. A replacement read replica is then created and provisioned in the same Availability Zone as the failed primary.

When only the primary node fails, ElastiCache Multi-AZ does the following:

After the new replica is available, be aware of these effects:

For information about finding the endpoints of a cluster, see the following topics:

Failure scenarios when the primary node and some read replicas fail

If the primary and at least one read replica fails, the available replica with the least replication lag is promoted to primary cluster. New read replicas are also created and provisioned in the same Availability Zones as the failed nodes and replica that was promoted to primary.

When the primary node and some read replicas fail, ElastiCache Multi-AZ does the following:

Make the following changes to your application after the new nodes are available:

Failure scenarios when the entire cluster fails

If everything fails, all the nodes are recreated and provisioned in the same Availability Zones as the original nodes.

In this scenario, all the data in the cluster is lost due to the failure of every node in the cluster. This occurrence is rare.

When the entire cluster fails, ElastiCache Multi-AZ does the following:

Because each of the replacement nodes has the same endpoint as the node it's replacing, you don't need to make any endpoint changes in your application.

For information about finding the endpoints of a replication group, see the following topics:

We recommend that you create the primary node and read replicas in different Availability Zones to raise your fault tolerance level.

Testing automatic failover

After you enable automatic failover, you can test it using the ElastiCache console, the AWS CLI, and the ElastiCache API.

When testing, note the following:

Testing automatic failover using the AWS Management Console

Use the following procedure to test automatic failover with the console.

Testing automatic failover using the AWS CLI

You can test automatic failover on any Multi-AZ enabled cluster using the AWS CLI operation test-failover.

The following example uses the AWS CLI to test automatic failover on the node group redis00-0003 in the Valkey or Redis OSS (cluster mode enabled) cluster redis00.

aws elasticache test-failover \
   --replication-group-id redis00 \
   --node-group-id redis00-0003

aws elasticache test-failover ^
   --replication-group-id redis00 ^
   --node-group-id redis00-0003

Output from the preceding command looks something like the following.

{
    "ReplicationGroup": {
        "Status": "available", 
        "Description": "1 shard, 3 nodes (1 + 2 replicas)", 
        "NodeGroups": [
            {
                "Status": "available", 
                "NodeGroupMembers": [
                    {
                        "CurrentRole": "primary", 
                        "PreferredAvailabilityZone": "us-west-2c", 
                        "CacheNodeId": "0001", 
                        "ReadEndpoint": {
                            "Port": 6379, 
                            "Address": "redis1x3-001.7ekv3t.0001.usw2.cache.amazonaws.com"
                        }, 
                        "CacheClusterId": "redis1x3-001"
                    }, 
                    {
                        "CurrentRole": "replica", 
                        "PreferredAvailabilityZone": "us-west-2a", 
                        "CacheNodeId": "0001", 
                        "ReadEndpoint": {
                            "Port": 6379, 
                            "Address": "redis1x3-002.7ekv3t.0001.usw2.cache.amazonaws.com"
                        }, 
                        "CacheClusterId": "redis1x3-002"
                    }, 
                    {
                        "CurrentRole": "replica", 
                        "PreferredAvailabilityZone": "us-west-2b", 
                        "CacheNodeId": "0001", 
                        "ReadEndpoint": {
                            "Port": 6379, 
                            "Address": "redis1x3-003.7ekv3t.0001.usw2.cache.amazonaws.com"
                        }, 
                        "CacheClusterId": "redis1x3-003"
                    }
                ], 
                "NodeGroupId": "0001", 
                "PrimaryEndpoint": {
                    "Port": 6379, 
                    "Address": "redis1x3.7ekv3t.ng.0001.usw2.cache.amazonaws.com"
                }
            }
        ], 
        "ClusterEnabled": false, 
        "ReplicationGroupId": "redis1x3", 
        "SnapshotRetentionLimit": 1, 
        "AutomaticFailover": "enabled", 
        "MultiAZ": "enabled",
        "SnapshotWindow": "11:30-12:30", 
        "SnapshottingClusterId": "redis1x3-002", 
        "MemberClusters": [
            "redis1x3-001", 
            "redis1x3-002", 
            "redis1x3-003"
        ], 
        "CacheNodeType": "cache.m3.medium", 
        "DataTiering": "disabled",
        "PendingModifiedValues": {}
    }
}

To track the progress of your failover, use the AWS CLI describe-events operation.

For more information, see the following:

Testing automatic failover using the ElastiCache API

You can test automatic failover on any cluster enabled with Multi-AZ using the ElastiCache API operation TestFailover.

The following example tests automatic failover on the node group redis00-0003 in the replication group (on the console, cluster) redis00.

https://elasticache.us-west-2.amazonaws.com/
    ?Action=TestFailover
    &NodeGroupId=redis00-0003
    &ReplicationGroupId=redis00
    &Version=2015-02-02
    &SignatureVersion=4
    &SignatureMethod=HmacSHA256
    &Timestamp=20140401T192317Z
    &X-Amz-Credential=<credential>

To track the progress of your failover, use the ElastiCache DescribeEvents API operation.

For more information, see the following:

Limitations on Multi-AZ

Be aware of the following limitations for Multi-AZ: