Dedicated manager nodes in Amazon OpenSearch Service
Amazon OpenSearch Service uses dedicated manager nodes to increase cluster stability. A dedicated manager node performs cluster management tasks, but does not hold data or respond to data upload requests. This offloading of cluster management tasks increases the stability of your domain. Just like all other node types, you pay an hourly rate for each dedicated manager node.
Dedicated manager nodes perform the following cluster management tasks:
-
Track all nodes in the cluster.
-
Track the number of indexes in the cluster.
-
Track the number of shards belonging to each index.
-
Maintain routing information for nodes in the cluster.
-
Update the cluster state after state changes, such as creating an index and adding or removing nodes in the cluster.
-
Replicate changes to the cluster state across all nodes in the cluster.
-
Monitor the health of all cluster nodes by sending heartbeat signals, periodic signals that monitor the availability of the data nodes in the cluster.
The following illustration shows an OpenSearch Service domain with 10 instances. Seven of the instances are data nodes and three are dedicated manager nodes. Only one of the dedicated manager nodes is active. The two gray dedicated manager nodes wait as backup in case the active dedicated manager node fails. All data upload requests are served by the seven data nodes, and all cluster management tasks are offloaded to the active dedicated manager node.
Choosing the number of dedicated manager nodes
We recommend that you use Multi-AZ with Standby, which adds three dedicated manager nodes to each production OpenSearch Service domain. If you deploy with Multi-AZ without Standby or single-AZ, we still recommend three dedicated manager nodes. Never choose an even number of dedicated manager nodes. Consider the following when choosing the number of dedicated manager nodes:
-
One dedicated manager node is explicitly prohibited by OpenSearch Service because you have no backup in the event of a failure. You receive a validation exception if you try to create a domain with only one dedicated manager node.
-
If you have two dedicated manager nodes, your cluster doesn't have the necessary quorum of nodes to elect a new manager node in the event of a failure.
A quorum is the number of dedicated manager nodes / 2 + 1 (rounded down to the nearest whole number). In this case, 2 / 2 + 1 = 2. Because one dedicated manager node has failed and only one backup exists, the cluster doesn't have a quorum and can't elect a new manager.
-
Three dedicated manager nodes, the recommended number, provides two backup nodes in the event of a manager node failure and the necessary quorum (2) to elect a new manager.
-
Four dedicated manager nodes are not better than three and can cause issues if you use multiple Availability Zones.
-
If one manager node fails, you have the quorum (3) to elect a new manager. If two nodes fail, you lose that quorum, just as you do with three dedicated manager nodes.
-
In a three Availability Zone configuration, two AZs have one dedicated manager node, and one AZ has two. If that AZ experiences a disruption, the remaining two AZs don't have the necessary quorum (3) to elect a new manager.
-
-
Having five dedicated manager nodes works as well as three and allows you to lose two nodes while maintaining a quorum. But because only one dedicated manager node is active at any given time, this configuration means that you pay for four idle nodes. Many users find this level of failover protection excessive.
If a cluster has an even number of manager-eligible nodes, OpenSearch and Elasticsearch versions 7.x and later ignore one node so that the voting configuration is always an odd number. In this case, four dedicated manager nodes are essentially equivalent to three (and two to one).
Note
If your cluster doesn't have the necessary quorum to elect a new manager node, write and read requests to the cluster both fail. This behavior differs from the OpenSearch default.
Choosing instance types for dedicated manager nodes
OpenSearch Service domain and instance quotas
Although dedicated manager nodes don't process search and query requests, their size is highly correlated with the instance size and number of instances, indexes, and shards that they can manage. For production clusters, we recommend, at a minimum, the following instance types for dedicated manager nodes.
These recommendations are based on typical workloads and can vary based on your needs. Clusters with many shards or field mappings can benefit from larger instance types. For more information, see Recommended CloudWatch alarms for Amazon OpenSearch Service to determine if you need to use a larger instance type.
| RAM | Max Node Support for Elasticsearch and OpenSearch Service 1.x to 2.15 | Max Shard Support for Elasticsearch and OpenSearch Service 2.15 and above | Max Node Support for Elasticsearch and OpenSearch Service 1.x to 2.15 | Max Shard Support for Elasticsearch and OpenSearch Service 2.17 and above |
|---|---|---|---|---|
| 2 GB | Not applicable | Not applicable | 10 | 1K |
| 4 GB | Not applicable | Not applicable | 10 | 5K |
| 8 GB | 10 | 10K | 30 | 15K |
| 16 GB | 30 | 30K | 60 | 30K |
| 32 GB | 75 | 40K | 120 | 60K |
| 64 GB | 125 | 75K | 240 | 120K |
| 128 GB | 200 | 75K | 480 | 240K |
| 256 GB | Not applicable | Not applicable | 1002 | 500K |
