# Creating and managing Amazon OpenSearch Service domains This chapter describes how to create and manage Amazon OpenSearch Service domains. A domain is the AWS-provisioned equivalent of an open source OpenSearch cluster. When you create a domain, you specify its settings, instance types, instance counts, and storage allocation. For more information about open source clusters, see [Creating a cluster](https://opensearch.org/docs/latest/tuning-your-cluster/) in the OpenSearch documentation. Unlike the brief instructions in the [Getting started tutorial](gsg.md), this chapter describes all options and provides relevant reference information. You can complete each procedure by using instructions for the OpenSearch Service console, the AWS Command Line Interface (AWS CLI), or the AWS SDKs. ## Creating OpenSearch Service domains This section describes how to create OpenSearch Service domains by using the OpenSearch Service console or by using the AWS CLI with the `create-domain` command. ### Creating OpenSearch Service domains (console) Use the following procedure to create an OpenSearch Service domain by using the console. **To create an OpenSearch Service domain (console)** 1. Go to [https://aws.amazon.com](https://aws.amazon.com) and choose **Sign In to the Console**. 1. Under **Analytics**, choose **Amazon OpenSearch Service**. 1. Choose **Create domain**. 1. For **Domain name**, enter a domain name. The name must meet the following criteria: + Unique to your account and AWS Region + Starts with a lowercase letter + Contains between 3 and 28 characters + Contains only lowercase letters a-z, the numbers 0-9, and the hyphen (-) 1. For the domain creation method, choose **Standard create**. 1. For **Templates**, choose the option that best matches the purpose of your domain: + **Production** domains for workloads that need high-availability and performance. These domains use Multi-AZ (with or without standby) and dedicated master nodes for higher availability. + **Dev/test** for development or testing. These domains can use Multi-AZ (with or without standby) or a single Availability Zone. **Important** Different deployment types present different options on subsequent pages. These steps include all options. 1. For **Deployment Option(s)**, choose **Domain with standby** to configure a 3-AZ domain, with nodes in one of the zones are reserved as standby. This option enforces a number of best practices, such as a specified data node count, master node count, instance type, replica count, and software update settings. 1. For **Version**, choose the version of OpenSearch or legacy Elasticsearch OSS to use. We recommend that you choose the latest version of OpenSearch. For more information, see [Supported versions of Elasticsearch and OpenSearch](what-is.md#choosing-version). (Optional) If you chose an OpenSearch version for your domain, select **Enable compatibility mode** to make OpenSearch report its version as 7.10, which allows certain Elasticsearch OSS clients and plugins that check the version before connecting to continue working with the service. 1. For **Instance type**, choose an instance type for your data nodes. For more information, see [Supported instance types in Amazon OpenSearch Service](supported-instance-types.md). **Note** Not all Availability Zones support all instance types. If you choose Multi-AZ with or without Standby, we recommend choosing current-generation instance types, such as R5 or I3. 1. For **Number of nodes**, choose the number of data nodes. For maximum values, see [OpenSearch Service domain and instance quotas](https://docs.aws.amazon.com/general/latest/gr/opensearch-service.html#opensearch-limits-domain). Single-node clusters are fine for development and testing, but should not be used for production workloads. For more guidance, see [Sizing Amazon OpenSearch Service domains](sizing-domains.md) and [Configuring a multi-AZ domain in Amazon OpenSearch Service](managedomains-multiaz.md). **Note** (Optional)Dedicated coordinator nodes support all OpenSearch versions and ElasticSearch versions 6.8 through 7.10. Dedicated coordinator nodes are available for use with domains that have a dedicated cluster manager enabled. To enable dedicated coordinator nodes, you will select the instance type and count. As a best practice, you should keep the instance family for your dedicated coordinator node the same as your data nodes (Intel based instances or Graviton based instances). 1. For **Storage type**, select Amazon EBS. The volume types available in the list depend on the instance type that you've chosen. For guidance on creating especially large domains, see [Petabyte scale in Amazon OpenSearch Service](petabyte-scale.md). 1. For **EBS** storage, configure the following additional settings. Some settings might not appear depending on the type of volume you choose. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/opensearch-service/latest/developerguide/createupdatedomains.html) 1. (Optional) If you selected a `gp3` volume type, expand **Advanced settings** and specify additional IOPS (up to 16,000 for every 3 TiB volume size provisioned per data node) and throughput (up to 1,000 MiB/s for every 3 TiB volume size provisioned per data node) beyond what is included with the price of storage, for an additional cost. For more information, see the [Amazon OpenSearch Service pricing](https://aws.amazon.com/opensearch-service/pricing/). 1. (Optional) To enable [UltraWarm storage](ultrawarm.md), choose **Enable UltraWarm data nodes**. Each instance type has a [maximum amount of storage](limits.md#limits-ultrawarm) that it can address. Multiply that amount by the number of warm data nodes for the total addressable warm storage. 1. (Optional) To enable [cold storage](cold-storage.md), choose **Enable cold storage**. You must enable UltraWarm to enable cold storage. 1. If you use Multi-AZ with Standby, three [dedicated master nodes](managedomains-dedicatedmasternodes.md) are aleady enabled. Choose the type of master nodes that you want. If you chose a Multi-AZ without Standby domain, select **Enable dedicated master nodes** and choose the type and number of master nodes that you want. Dedicated master nodes increase cluster stability and are required for domains that have instance counts greater than 10. We recommend three dedicated master nodes for production domains. **Note** You can choose different instance types for your dedicated master nodes and data nodes. For example, you might select general purpose or storage-optimized instances for your data nodes, but compute-optimized instances for your dedicated master nodes. 1. (Optional) For domains running OpenSearch or Elasticsearch 5.3 and later, the **Snapshot configuration** is irrelevant. For more information about automated snapshots, see [Creating index snapshots in Amazon OpenSearch Service](managedomains-snapshots.md). 1. If you want to use a custom endpoint rather than the standard one of `https://search-mydomain-1a2a3a4a5a6a7a8a9a0a9a8a7a.us-east-1.es.amazonaws.com` , choose **Enable custom endpoint** and provide a name and certificate. For more information, see [Creating a custom endpoint for Amazon OpenSearch Service](customendpoint.md). 1. Under **Network**, choose either **VPC access** or **Public access**. If you choose **Public access**, skip to the next step. If you choose **VPC access**, make sure you meet the [prerequisites](vpc.md#prerequisites-vpc-endpoints), then configure the following settings: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/opensearch-service/latest/developerguide/createupdatedomains.html) 1. Enable or disable fine-grained access control: + If you want to use IAM for user management, choose **Set IAM ARN as master user** and specify the ARN for an IAM role. + If you want to use the internal user database, choose **Create master user** and specify a username and password. Whichever option you choose, the master user can access all indexes in the cluster and all OpenSearch APIs. For guidance on which option to choose, see [Key concepts](fgac.md#fgac-concepts). If you disable fine-grained access control, you can still control access to your domain by placing it within a VPC, applying a restrictive access policy, or both. You must enable node-to-node encryption and encryption at rest to use fine-grained access control. **Note** We *strongly* recommend enabling fine-grained access control to protect the data on your domain. Fine-grained access control provides security at the cluster, index, document, and field levels. 1. (Optional) If you want to use SAML authentication for OpenSearch Dashboards, choose **Enable SAML authentication** and configure SAML options for the domain. For instructions, see [SAML authentication for OpenSearch Dashboards](saml.md). 1. (Optional) If you want to use Amazon Cognito authentication for OpenSearch Dashboards, choose **Enable Amazon Cognito authentication**. Then choose the Amazon Cognito user pool and identity pool that you want to use for OpenSearch Dashboards authentication. For guidance on creating these resources, see [Configuring Amazon Cognito authentication for OpenSearch Dashboards](cognito-auth.md). 1. (Optional) If you want to use IAM Identity Center (IDC) authentication to connect your existing identity source and give your AWS applications a common view of your users, choose **Enable API access authenticated with IAM Identity Center **. For more information, see [Trusted identity propagation overview](https://docs.aws.amazon.com/singlesignon/latest/userguide/trustedidentitypropagation-overview.html?icmpid=docs_console_unmapped) in the *IAM Identity Center User Guide.* 1. (Optional) In the **Advanced features** section, leave **Enable Natural Language Query Generation and Amazon Q Developer features** selected, if you want to use these features. 1. Choose **Enable S3 Vectors as an engine option** for enhanced vector search options. For more information, see [Advanced search capabilities with an Amazon S3 vector engine](s3-vector-opensearch-integration-engine.md). 1. Choose **Enable GPU acceleration for enhanced vector search options. For more information, see [GPU-acceleration for vector indexing](gpu-acceleration-vector-index.md).** 1. For **Access policy**, choose an access policy or configure one of your own. If you choose to create a custom policy, you can configure it yourself or import one from another domain. For more information, see [Identity and Access Management in Amazon OpenSearch Service](ac.md). **Note** If you enabled VPC access, you can't use IP-based policies. Instead, you can use [security groups](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_SecurityGroups.html) to control which IP addresses can access the domain. For more information, see [About access policies on VPC domains](vpc.md#vpc-security). 1. (Optional) To require that all requests to the domain arrive over HTTPS, select **Require HTTPS for all traffic to the domain**. To enable node-to-node encryption, select **Node-to-node encryption**. For more information, see [Node-to-node encryption for Amazon OpenSearch Service](ntn.md). To enable encryption of data at rest, select **Enable encryption of data at rest**. These options are pre-selected if you chose the Multi-AZ with Standby deployment option. 1. (Optional) Select **Use AWS owned key** to have OpenSearch Service create an AWS KMS encryption key on your behalf (or use the one that it already created). Otherwise, choose your own KMS key. For more information, see [Encryption of data at rest for Amazon OpenSearch Service](encryption-at-rest.md). 1. For **Off-peak window**, select a start time to schedule service software updates and Auto-Tune optimizations that require a blue/green deployment. Off-peak updates help to minimize strain on a cluster's dedicated master nodes during high traffic periods. 1. For **Auto-Tune**, choose whether to allow OpenSearch Service to suggest memory-related configuration changes to your domain to improve speed and stability. For more information, see [Auto-Tune for Amazon OpenSearch Service](auto-tune.md). (Optional) Select **Off-peak window** to schedule a recurring window during which Auto-Tune updates the domain. 1. (Optional) Select **Automatic software update** to enable automatic software updates. 1. (Optional) Add tags to describe your domain so you can categorize and filter on that information. For more information, see [Tagging Amazon OpenSearch Service domains](managedomains-awsresourcetagging.md). 1. (Optional) Expand and configure **Advanced cluster settings**. For a summary of these options, see [Advanced cluster settings](#createdomain-configure-advanced-options). 1. Choose **Create**. ### Creating OpenSearch Service domains (AWS CLI) Instead of creating an OpenSearch Service domain by using the console, you can use the AWS CLI. For syntax, see Amazon OpenSearch Service in the [AWS CLI command reference](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/opensearch/index.html)a. #### Example commands This first example demonstrates the following OpenSearch Service domain configuration: + Creates an OpenSearch Service domain named *mylogs* with OpenSearch version 1.2 + Populates the domain with two instances of the `r6g.large.search` instance type + Uses a 100 GiB General Purpose (SSD) `gp3` EBS volume for storage for each data node + Allows anonymous access, but only from a single IP address: 192.0.2.0/32 ``` aws opensearch create-domain \ --domain-name mylogs \ --engine-version OpenSearch_1.2 \ --cluster-config InstanceType=r6g.large.search,InstanceCount=2 \ --ebs-options EBSEnabled=true,VolumeType=gp3,VolumeSize=100,Iops=3500,Throughput=125 \ --access-policies '{"Version": "2012-10-17", "Statement": [{"Action": "es:*", "Principal":"*","Effect": "Allow", "Condition": {"IpAddress":{"aws:SourceIp":["192.0.2.0/32"]}}}]}' ``` The next example demonstrates the following OpenSearch Service domain configuration: + Creates an OpenSearch Service domain named *mylogs* with Elasticsearch version 7.10 + Populates the domain with six instances of the `r6g.large.search` instance type + Uses a 100 GiB General Purpose (SSD) `gp2` EBS volume for storage for each data node + Restricts access to the service to a single user, identified by the user's AWS account ID: 555555555555 + Distributes instances across three Availability Zones ``` aws opensearch create-domain \ --domain-name mylogs \ --engine-version Elasticsearch_7.10 \ --cluster-config InstanceType=r6g.large.search,InstanceCount=6,ZoneAwarenessEnabled=true,ZoneAwarenessConfig={AvailabilityZoneCount=3} \ --ebs-options EBSEnabled=true,VolumeType=gp2,VolumeSize=100 \ --access-policies '{"Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": {"AWS": "arn:aws:iam::555555555555:root" }, "Action":"es:*", "Resource": "arn:aws:es:us-east-1:555555555555:domain/mylogs/*" } ] }' ``` The next example demonstrates the following OpenSearch Service domain configuration: + Creates an OpenSearch Service domain named *mylogs* with OpenSearch version 1.0 + Populates the domain with ten instances of the `r6g.xlarge.search` instance type + Populates the domain with three instances of the `r6g.large.search` instance type to serve as dedicated master nodes + Uses a 100 GiB Provisioned IOPS EBS volume for storage, configured with a baseline performance of 1000 IOPS for each data node + Restricts access to a single user and to a single subresource, the `_search` API ``` aws opensearch create-domain \ --domain-name mylogs \ --engine-version OpenSearch_1.0 \ --cluster-config InstanceType=r6g.xlarge.search,InstanceCount=10,DedicatedMasterEnabled=true,DedicatedMasterType=r6g.large.search,DedicatedMasterCount=3 \ --ebs-options EBSEnabled=true,VolumeType=io1,VolumeSize=100,Iops=1000 \ --access-policies '{"Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::555555555555:root" }, "Action": "es:*", "Resource": "arn:aws:es:us-east-1:555555555555:domain/mylogs/_search" } ] }' ``` **Note** If you attempt to create an OpenSearch Service domain and a domain with the same name already exists, the CLI does not report an error. Instead, it returns details for the existing domain. ### Creating OpenSearch Service domains (AWS SDKs) The AWS SDKs (except the Android and iOS SDKs) support all the actions defined in the [Amazon OpenSearch Service API Reference](https://docs.aws.amazon.com/opensearch-service/latest/APIReference/Welcome.html), including `CreateDomain`. For sample code, see [Using the AWS SDKs to interact with Amazon OpenSearch Service](configuration-samples.md). For more information about installing and using the AWS SDKs, see [AWS Software Development Kits](http://aws.amazon.com/code). ### Creating OpenSearch Service domains (AWS CloudFormation) OpenSearch Service is integrated with AWS CloudFormation, a service that helps you to model and set up your AWS resources so that you can spend less time creating and managing your resources and infrastructure. You create a template that describes the OpenSearch domain you want to create, and CloudFormation provisions and configures the domain for you. For more information, including examples of JSON and YAML templates for OpenSearch domains, see the [Amazon OpenSearch Service resource type reference](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-elasticsearch-domain.html) in the *AWS CloudFormation User Guide*. ## Configuring access policies Amazon OpenSearch Service offers several ways to configure access to your OpenSearch Service domains. For more information, see [Identity and Access Management in Amazon OpenSearch Service](ac.md) and [Fine-grained access control in Amazon OpenSearch Service](fgac.md). The console provides preconfigured access policies that you can customize for the specific needs of your domain. You also can import access policies from other OpenSearch Service domains. For information about how these access policies interact with VPC access, see [About access policies on VPC domains](vpc.md#vpc-security). **To configure access policies (console)** 1. Go to [https://aws.amazon.com](https://aws.amazon.com), and then choose **Sign In to the Console**. 1. Under **Analytics**, choose **Amazon OpenSearch Service**. 1. In the navigation pane, under **Domains**, choose the domain you want to update. 1. Choose **Actions** and **Edit security configuration**. 1. Edit the access policy JSON, or import a preconfigured option. 1. Choose **Save changes**. ## Migrating to OpenSearch Service with Migration Assistant Migration Assistant for Amazon OpenSearch Service is a comprehensive solution that simplifies the process of migrating from self-managed Elasticsearch or OpenSearch clusters to OpenSearch Service. This toolkit addresses the operational complexity of migrations while ensuring data integrity and validating performance post-migration. ### Overview Whether you're setting up a proof-of-concept in AWS, transitioning production workloads, or upgrading to the latest OpenSearch versions, Migration Assistant provides step-by-step guidance, best practices, and tools to leverage the full potential of OpenSearch migrations. The Migration Assistant offers the following key benefits: + **Metadata migration** - Migrates cluster metadata, including index settings, type mappings, index templates, and aliases + **Data migration** - Migrates existing data from legacy clusters to OpenSearch Service domains + **Live traffic handling** - Intercepts and redirects live traffic from self-managed clusters to OpenSearch Service domains with minimal latency + **Traffic replication** - Replicates production traffic on target clusters to validate accuracy and performance + **Performance testing** - Simulates real-world traffic by capturing and replaying request patterns to fine-tune system performance + **Global availability** - Deploys across the most common AWS Regions for global reach and scalability Migration Assistant supports migrating from Elasticsearch versions 6.x and 7.x, and OpenSearch 1.x and 2.x. For more information, see [Supported migration paths](https://docs.opensearch.org/latest/migration-assistant/is-migration-assistant-right-for-you/#supported-migration-paths). **Note** Note the following additional information about Migration Assistant. The tool supports multi-hop migrations (for example, migration from Elasticsearch 5.x to OpenSearch Service 3.x in one hop). You can roll back a migration. For some use cases, the tool requires little or no downtime. The tool provides high-performance backfill without impacting a source cluster. ### Migration scenarios The Migration Assistant is designed to handle the following migration scenarios: **Metadata migration** Migrates cluster metadata such as index settings, aliases, and templates from your source cluster to the target OpenSearch Service domain. **Backfill migration** Migrates existing or historical data from a source cluster to a target OpenSearch Service domain, ensuring that all your important data is preserved during the transition. **Live traffic migration** Replicates ongoing live traffic from your source cluster to the target OpenSearch Service domain, allowing you to maintain service availability during migration. **Important** Migration strategies aren't universally applicable. The Migration Assistant provides guidance based on engineering best practices, but you should evaluate your specific requirements and test thoroughly before migrating production workloads. ### Getting started with Migration Assistant Migration Assistant for Amazon OpenSearch Service is available as an AWS Solution with comprehensive documentation, deployment templates, and source code. To get started with Migration Assistant: **To access Migration Assistant resources** 1. Review the complete solution documentation at [Migration Assistant for Amazon OpenSearch Service solution overview](https://docs.aws.amazon.com/solutions/latest/migration-assistant-for-amazon-opensearch-service/solution-overview.html). 1. Understand the costs and requirements by reading the [deployment planning guide](https://docs.aws.amazon.com/solutions/latest/migration-assistant-for-amazon-opensearch-service/plan-your-deployment.html). 1. Deploy the solution using the [deployment instructions](https://docs.opensearch.org/latest/migration-assistant/migration-phases/) and follow the [usage guide](https://docs.aws.amazon.com/solutions/latest/migration-assistant-for-amazon-opensearch-service/use-the-solution.html) to perform your migration. For developers and advanced users, Migration Assistant source code and additional documentation are available in the [OpenSearch migrations GitHub repository](https://github.com/opensearch-project/opensearch-migrations). ### Solution architecture When deployed in AWS, Migration Assistant uses several AWS services to provide a comprehensive migration solution: + **AWS CloudFormation** - Provides Infrastructure as Code (IaC) templates to deploy and configure Migration Assistant + **OpenSearch Service** - The target service for your migrated search and analytics workloads + **Amazon Managed Streaming for Apache Kafka** - Provides stream processing capabilities for durable storage and reuse of HTTP traffic + **Amazon Elastic Container Service** - Runs the Migration Management Console and Traffic Replayer in secure, scalable containers + **Amazon Elastic File System** - Provides scalable persistent storage for request and response data from both source and target clusters + **Amazon Simple Storage Service** - Stores snapshots for historical backfill tasks and Infrastructure as Code content For detailed architecture information, see [Migration Assistant architecture details](https://docs.opensearch.org/latest/migration-assistant/architecture/). ## Advanced cluster settings Use advanced options to configure the following: **Indices in request bodies** Specifies whether explicit references to indexes are allowed inside the body of HTTP requests. Setting this property to `false` prevents users from bypassing access control for subresources. By default, the value is `true`. For more information, see [Advanced options and API considerations](ac.md#ac-advanced). **Fielddata cache allocation** Specifies the percentage of Java heap space that is allocated to field data. By default, this setting is 20% of the JVM heap. Many customers query rotating daily indices. We recommend that you begin benchmark testing with `indices.fielddata.cache.size` configured to 40% of the JVM heap for most of these use cases. For very large indices, you might need a large field data cache. **Max clause count** Specifies the maximum number of clauses allowed in a Lucene boolean query. The default is 1,024. Queries with more than the permitted number of clauses result in a `TooManyClauses` error. For more information, see [the Lucene documentation](https://lucene.apache.org/core/6_6_0/core/org/apache/lucene/search/BooleanQuery.html). # Automatic semantic enrichment for Amazon OpenSearch Service ## Introduction Amazon OpenSearch Service uses word-to-word matching (lexical search) to find results, similar to other traditional search engines. This approach works well for specific queries like product codes or model numbers, but struggles with abstract searches where understanding user intent becomes crucial. For example, when you search for "shoes for the beach," lexical search matches individual words "shoes," "beach," "for," and "the" in catalog items, potentially missing relevant products like "water-resistant sandals" or "surf footwear" that don't contain the exact search terms. Automatic Semantic Enrichment solves this limitation by considering both keyword matches and the contextual meaning behind searches. This feature understands search intent and improves search relevance by up to 20%. Enable this feature for text fields in your index to enhance search results. **Note** Automatic semantic enrichment is available for OpenSearch Service domains running version 2.19 or later. Additionally, domains with OpenSearch version 2.19 also need to be on the latest service software version update. Currently, feature is available for public domains, and VPC domains are not supported. ## Model details and performance benchmark While this feature handles the technical complexities behind the scenes without exposing the underlying model, we provide transparency through a brief model description and benchmark results to help you make informed decisions about feature adoption in your critical workloads. Automatic semantic enrichment uses a service-managed, pre-trained sparse model that works effectively without requiring custom fine-tuning. The model analyzes the fields you specify, expanding them into sparse vectors based on learned associations from diverse training data. The expanded terms and their significance weights are stored in native Lucene index format for efficient retrieval. We’ve optimized this process using [document-only mode,](https://docs.opensearch.org/docs/latest/vector-search/ai-search/neural-sparse-with-pipelines/#step-1a-choose-the-search-mode) where encoding happens only during data ingestion. Search queries are merely tokenized rather than processed through the sparse model, making the solution both cost-effective and performant. Our performance validation during feature development used the [MS MARCO](https://huggingface.co/datasets/BeIR/msmarco) passage retrieval dataset, featuring passages averaging 334 characters. For relevance scoring, we measured average Normalized Discounted Cumulative Gain (NDCG) for the first 10 search results (ndcg@10) on the [BEIR](https://github.com/beir-cellar/beir) benchmark for English content and average ndcg@10 on MIRACL for multilingual content. We assessed latency through client-side, 90th-percentile (p90) measurements and search response p90 [took values.](https://github.com/beir-cellar/beir) These benchmarks provide baseline performance indicators for both search relevance and response times. Here are the key benchmark numbers - + English language - Relevance improvement of 20% over lexical search. It also lowered P90 search latency by 7.7% over lexical search (BM25 is 26 ms, and automatic semantic enrichment is 24 ms). + Multi-lingual - Relevance improvement of 105% over lexical search, whereas P90 search latency increased by 38.4% over lexical search (BM25 is 26 ms, and automatic semantic enrichment is 36 ms). Given the unique nature of each workload, we encourage you to evaluate this feature in your development environment using your own benchmarking criteria before making implementation decisions. ## Languages Supported The feature supports English. In addition, the model also supports Arabic, Bengali, Chinese, Finnish, French, Hindi, Indonesian, Japanese, Korean, Persian, Russian, Spanish, Swahili, and Telugu. ## Set up an automatic semantic enrichment index for domains Setting up an index with automatic semantic enrichment enabled for your text fields is easy, and you can manage it through the console, APIs, and CloudFormation templates during new index creation. To enable it for an existing index, you need to recreate the index with automatic semantic enrichment enabled for text fields. Console experience - The AWS console allows you to easily create an index with automatic semantic enrichment fields. Once you select a domain, you will find the create index button at the top of the console. Once you click the create index button, you will find options to define automatic semantic enrichment fields. In one index, you can have combinations of automatic semantic enrichment for English and multilingual, as well as lexical fields. ![\[alt text not found\]](http://docs.aws.amazon.com/opensearch-service/latest/developerguide/images/ase-console-exp.png) API experience - To create an automatic semantic enrichment index using the AWS Command Line Interface (AWS CLI), use the create-index command: ``` aws opensearch create-index \ --domain-name [domain_name] \ --index-name [index_name] \ --index-schema [index_body] \ ``` In the following example index-schema, the *title\$1semantic *field has a field type set to *text* and has parameter *semantic\$1enrichment *set to status *ENABLED*. Setting the *semantic\$1enrichment* parameter enables automatic semantic enrichment on the *title\$1semantic* field. You can use the *language\$1options* field to specify either *english* or *multi-lingual*. ``` aws opensearch create-index \ --id XXXXXXXXX \ --index-name 'product-catalog' \ --index-schema '{ "mappings": { "properties": { "product_id": { "type": "keyword" }, "title_semantic": { "type": "text", "semantic_enrichment": { "status": "ENABLED", "language_options": "english" } }, "title_non_semantic": { "type": "text" } } } }' ``` To describe the created index, use the following command: ``` aws opensearch get-index \ --domain-name [domain_name] \ --index-name [index_name] \ ``` ## Data ingestion and search Once you've created an index with automatic semantic enrichment enabled, the feature works automatically during data ingestion process, no additional configuration required. Data ingestion: When you add documents to your index, the system automatically: + Analyzes the text fields you designated for semantic enrichment + Generates semantic encodings using OpenSearch Service managed sparse model + Stores these enriched representations alongside your original data This process uses OpenSearch's built-in ML connectors and ingest pipelines, which are created and managed automatically behind the scenes. Search: The semantic enrichment data is already indexed, so queries run efficiently without invoking the ML model again. This means you get improved search relevance with no additional search latency overhead. ## Configuring permissions for automatic semantic enrichment Before creating an index with automatic semantic enrichment, you need to configure the required permissions. This section explains the permissions needed for different index operations and how to set them up for both AWS Identity and Access Management (IAM) and fine-grained access control scenarios. ### IAM permissions The following IAM permissions are required for automatic semantic enrichment operations. These permissions vary depending on the specific index operation you want to perform. #### CreateIndex API permissions To create an index with automatic semantic enrichment, you need the following IAM permissions: + `es:CreateIndex` – Create an index with semantic enrichment capabilities. + `es:ESHttpHead` – Perform HEAD requests to check index existence. + `es:ESHttpPut` – Perform PUT requests for index creation. + `es:ESHttpPost` – Perform POST requests for index operations. #### UpdateIndex API permissions To update an existing index with automatic semantic enrichment, you need the following IAM permissions: + `es:UpdateIndex` – Update index settings and mappings. + `es:ESHttpPut` – Perform PUT requests for index updates. + `es:ESHttpGet` – Perform GET requests to retrieve index information. + `es:ESHttpPost` – Perform POST requests for index operations. #### GetIndex API permissions To retrieve information about an index with automatic semantic enrichment, you need the following IAM permissions: + `es:GetIndex` – Retrieve index information and settings. + `es:ESHttpGet` – Perform GET requests to retrieve index data. #### DeleteIndex API permissions To delete an index with automatic semantic enrichment, you need the following IAM permissions: + `es:DeleteIndex` – Delete an index and its semantic enrichment components. + `es:ESHttpDelete` – Perform DELETE requests for index removal. ### Sample IAM policy The following sample identity-based access policy provides the permissions necessary for a user to manage indexes with automatic semantic enrichment: ``` { "Version": "2012-10-17", "Statement": [ { "Sid": "AllowSemanticEnrichmentIndexOperations", "Effect": "Allow", "Action": [ "es:CreateIndex", "es:UpdateIndex", "es:GetIndex", "es:DeleteIndex", "es:ESHttpHead", "es:ESHttpGet", "es:ESHttpPut", "es:ESHttpPost", "es:ESHttpDelete" ], "Resource": "arn:aws:es:aws-region:111122223333:domain/domain-name" } ] } ``` Replace *aws-region*, *111122223333*, and *domain-name* with your specific values. You can further restrict access by specifying particular index patterns in the resource ARN. ### Fine-grained access control permissions If your Amazon OpenSearch Service domain has fine-grained access control enabled, you need additional permissions beyond the IAM permissions. The following permissions are required for each index operation. #### CreateIndex API permissions When fine-grained access control is enabled, the following additional permissions are required for creating an index with automatic semantic enrichment: + `indices:admin/create` – Create index operations. + `indices:admin/mapping/put` – Create and update index mappings. + `cluster:admin/opensearch/ml/create_connector` – Create machine learning connectors for semantic processing. + `cluster:admin/opensearch/ml/register_model` – Register machine learning models for semantic enrichment. + `cluster:admin/ingest/pipeline/put` – Create ingest pipelines for data processing. + `cluster:admin/search/pipeline/put` – Create search pipelines for query processing. #### UpdateIndex API permissions When fine-grained access control is enabled, the following additional permissions are required for updating an index with automatic semantic enrichment: + `indices:admin/get` – Retrieve index information. + `indices:admin/settings/update` – Update index settings. + `indices:admin/mapping/put` – Update index mappings. + `cluster:admin/opensearch/ml/create_connector` – Create machine learning connectors. + `cluster:admin/opensearch/ml/register_model` – Register machine learning models. + `cluster:admin/ingest/pipeline/put` – Create ingest pipelines. + `cluster:admin/search/pipeline/put` – Create search pipelines. + `cluster:admin/ingest/pipeline/get` – Retrieve ingest pipeline information. + `cluster:admin/search/pipeline/get` – Retrieve search pipeline information. #### GetIndex API permissions When fine-grained access control is enabled, the following additional permissions are required for retrieving information about an index with automatic semantic enrichment: + `indices:admin/get` – Retrieve index information. + `cluster:admin/ingest/pipeline/get` – Retrieve ingest pipeline information. + `cluster:admin/search/pipeline/get` – Retrieve search pipeline information. #### DeleteIndex API permissions When fine-grained access control is enabled, the following additional permission is required for deleting an index with automatic semantic enrichment: + `indices:admin/delete` – Delete index operations. ## Query Rewrites Automatic semantic enrichment automatically converts your existing “match” queries to semantic search queries without requiring query modifications. If a match query is part of a compound query, the system traverses your query structure, finds match queries, and replaces them with neural sparse queries. Currently, the feature only supports replacing “match” queries, whether it’s a standalone query or part of a compound query. “multi\$1match” is not supported. In addition, the feature supports all compound queries to replace their nested match queries. Compound queries include: bool, boosting, constant\$1score, dis\$1max, function\$1score, and hybrid. ## Limitations of automatic semantic enrichment Automatic semantic search is most effective when applied to small-to-medium sized fields containing natural language content, such as movie titles, product descriptions, reviews, and summaries. Although semantic search enhances relevance for most use cases, it might not be optimal for certain scenarios. Consider following limitations when deciding whether to implement automatic semantic enrichment for your specific use case. + Very long documents – The current sparse model processes only the first 8,192 tokens of each document for English. For multilingual documents, it’s 512 tokens. For lengthy articles, consider implementing document chunking to ensure complete content processing. + Log analysis workloads – Semantic enrichment significantly increases index size, which might be unnecessary for log analysis where exact matching typically suffices. The additional semantic context rarely improves log search effectiveness enough to justify the increased storage requirements. + Automatic semantic enrichment is not compatible with the Derived Source feature. + Throttling – Indexing inference requests are currently capped at 200 TPS for OpenSearch Service domains. This is a soft limit; reach out to AWS Support for higher limits. ## Pricing Amazon OpenSearch Service bills automatic semantic enrichment based on OpenSearch Compute Units (OCUs) consumed during sparse vector generation at indexing time. You’re charged only for actual usage during indexing. You can monitor this consumption using the Amazon CloudWatch metric SemanticSearchOCU. For specific details about model token limits, volume throughput per OCU, and example of sample calculation, visit [ OpenSearch Service Pricing](https://aws.amazon.com/opensearch-service/pricing/). # Making configuration changes in Amazon OpenSearch Service Amazon OpenSearch Service uses a *blue/green* deployment process when updating domains. A blue/green deployment creates an idle environment for domain updates that copies the production environment, and routes users to the new environment after those updates are complete. In a blue/green deployment, the blue environment is the current production environment. The green environment is the idle environment. Data is migrated from the blue environment to the green environment. When the new environment is ready, OpenSearch Service switches over the environments to promote the green environment to be the new production environment. The switchover happens with no data loss. This practice minimizes downtime and maintains the original environment in the event that deployment to the new environment is unsuccessful. **Topics** + [ ## Changes that usually cause blue/green deployments ](#bg) + [ ## Changes that usually don't cause blue/green deployments ](#nobg) + [ ## Blue/Green Deployment options ](#bg-deployment-options) + [ ## Determining whether a change will cause a blue/green deployment ](#dryrun) + [ ## Tracking a configuration change ](#initiating-tracking-configuration-changes) + [ ## Stages of a configuration change ](#managedomains-config-stages) + [ ## Performance impact of blue/green deployments ](#performance-impact-bluegreen) + [ ## Charges for configuration changes ](#managedomains-config-charges) + [ ## Troubleshooting validation errors ](#validation) ## Changes that usually cause blue/green deployments The following operations cause blue/green deployments: + Changing the instance type + Enabling fine-grained access control + Performing service software updates + Enabling or disabling dedicated master nodes + Enabling or disabling Multi-AZ without Standby + Changing the storage type, volume type, or volume size + Choosing different VPC subnets + Adding or removing VPC security groups + Adding or removing dedicated coordinator nodes + Enabling or disabling Amazon Cognito authentication for OpenSearch Dashboards + Choosing a different Amazon Cognito user pool or identity pool + Modifying advanced settings + Upgrading to a new OpenSearch version (OpenSearch Dashboards might be unavailable during some or all of the upgrade) + Enabling encryption of data at rest or node-to-node encryption + Enabling or disabling UltraWarm or cold storage + Disabling Auto-Tune and rolling back its changes + Associating an optional plugin to a domain and dissociating an optional plugin from a domain + Increasing the dedicated master node count for Multi-AZ domains with two dedicated master nodes + Decreasing the EBS volume size + Changing EBS volume size, IOPS, or throughput, if the the last change you made is in progress or occurred less than 6 hours ago + Enabling the publication of audit logs to CloudWatch. For Multi-AZ with Standby domains, you can only make one change request at a time. If a change is already in progress, the new request is rejected. You can check the status of the current change with the `DescribeDomainChangeProgress` API. ## Changes that usually don't cause blue/green deployments In *most* cases, the following operations do not cause blue/green deployments: + Modifying the access policy + Modifying the custom endpoint + Changing the Transport Layer Security (TLS) policy + Changing the automated snapshot hour + Enabling or disabling **Require HTTPS** + Enabling Auto-Tune or disabling it without rolling back its changes + If your domain has dedicated master nodes, changing the data node or UltraWarm node count + If your domain has dedicated master nodes, changing the dedicated master instance type or count (except for Multi-AZ domains with two dedicated master nodes) + Enabling or disabling the publication of error logs or slow logs to CloudWatch + Disabling the publication of audit logs to CloudWatch + Increasing the volume size, IOPS, or throughput for gp3 EBS type. **Note** Prior to March 10, 2026, in-place volume increases were supported only for volumes up to 3 TiB for gp3. On March 10, 2026, this limitation was removed, allowing in-place volume increases beyond 3 TiB. However, if your cluster had a volume size above 3 TiB before this limitation was removed, the first volume increase will require a blue/green deployment. All subsequent volume increases for that cluster will be performed as in-place updates and will not require a blue/green deployment. + Adding or removing tags **Note** There are some exceptions depending on your service software version. If you want to be sure that a change won't cause a blue/green deployment, [perform a dry run](#dryrun) before updating your domain, if this option is available. Some changes don't offer a dry run option. We generally recommend that you make changes to your cluster outside of peak traffic hours. ## Blue/Green Deployment options Select a deployment strategy to control how your cluster handles deployments when sufficient capacity is not available at the time of update. 1. **Full Swap Blue/Green** — The default deployment behavior. Requires full instance capacity upfront, ensuring the fastest deployment when capacity is available. Deployment will not proceed if sufficient capacity cannot be allocated. 1. **Capacity Optimized** — Recommended for clusters with 30\$1 data nodes. Attempts a full blue/green swap first, and if capacity is insufficient, proceeds with deploying in batches. Ensures deployments can complete even when capacity is limited. Completion time may increase, as deployment will be done in batches. ------ #### [ Console ] **For Edit Domain Flow:** 1. Open the Amazon OpenSearch Service console at [https://console.aws.amazon.com/aos/home](https://console.aws.amazon.com/aos/home). 1. In the navigation pane, under **Domains**, choose the domain name to open the Cluster Configuration. 1. Click on **Edit** button to right side in **Cluster Configuration** tab. 1. From **Deployment strategy** options choose required configurations for domain update: 1. **Full Swap Blue/Green** — The default deployment behavior. Requires full instance capacity upfront, ensuring the fastest deployment when capacity is available. Deployment will not proceed if sufficient capacity cannot be allocated. 1. **Capacity Optimized** — Recommended for clusters with 30\$1 data nodes. Attempts a full blue/green swap first, and if capacity is insufficient, proceeds with deploying in batches. Ensures deployments can complete even when capacity is limited. Completion time may increase, as deployment will be done in batches. 1. Choose **Save changes**. **For Create Domain Flow:** 1. Open the Amazon OpenSearch Service console at [https://console.aws.amazon.com/aos/home](https://console.aws.amazon.com/aos/home). 1. In the left navigation pane, choose **Domains**. 1. Click on **Create Domain** button. 1. Select all required configurations for domain. 1. From **Deployment strategy** options choose required configurations for domain update: 1. **Full Swap Blue/Green** — The default deployment behavior. Requires full instance capacity upfront, ensuring the fastest deployment when capacity is available. Deployment will not proceed if sufficient capacity cannot be allocated. 1. **Capacity Optimized** — Recommended for clusters with 30\$1 data nodes. Attempts a full blue/green swap first, and if capacity is insufficient, proceeds with deploying in batches. Ensures deployments can complete even when capacity is limited. Completion time may increase, as deployment will be done in batches. 1. Click on **Create** button on **Domain Summary** panel on right hand side. ------ #### [ API ] You can configure the deployment strategy using the [UpdateDomainConfig](https://docs.aws.amazon.com/opensearch-service/latest/APIReference/API_UpdateDomainConfig.html) API. **Capacity Optimized option** ``` POST https://es.us-east-1.amazonaws.com/2021-01-01/opensearch/domain/my-domain/config { "DeploymentStrategyOptions": { "DeploymentStrategy": "CapacityOptimized" } } ``` **Default option** ``` POST https://es.us-east-1.amazonaws.com/2021-01-01/opensearch/domain/my-domain/config { "DeploymentStrategyOptions": { "DeploymentStrategy": "Default" } } ``` ------ ## Determining whether a change will cause a blue/green deployment You can test some types of planned configuration changes to determine whether they will cause a blue/green deployment, without having to commit to those changes. Before you initiate a configuration change, use the console or an API to run a validation check to ensure that your domain is eligible for an update. ------ #### [ Console ] **To validate a configuration change** 1. Navigate to the Amazon OpenSearch Service console at [https://console.aws.amazon.com/aos/](https://console.aws.amazon.com/aos/). 1. In the left navigation pane, choose **Domains**. 1. Select the domain you want to make a configuration change for. This opens the domain details page. Select the **Actions** dropdown menu and then choose **Edit cluster configuration**. 1. Make changes to the domain, such as changing the instance type or the number of nodes. 1. Under **Dry run analysis**, choose **Run**. The dry run validates your configuration change for errors and determines whether it requires a blue/green deployment. 1. When the dry run is complete, the results appear at the bottom of the page, along with a dry run ID. The analysis indicate whether or not the configuration change requires a blue/green deployment. Each dry run overwrites the previous one. To retain the details of each run, save its dry run ID. Dry runs are available for 90 days or until you make a configuration update. 1. To proceed with your configuration update, choose **Save changes**. Otherwise, choose **Cancel**. Either option takes you back to the **Cluster configuration** tab. On this tab, you can choose **Dry run details** to see the details of your latest dry run. This page also includes a side-by-side comparison between the configuration before the dry run and the dry run configuration. ------ #### [ API ] You can perform a dry run validation through the configuration API. To test your changes with the API, set `DryRun` to `true`, and `DryRunMode` to `Verbose`. Verbose mode runs a validation check in addition to determining whether the change will initiate a blue/green deployment. For example, this [UpdateDomainConfig](https://docs.aws.amazon.com/opensearch-service/latest/APIReference/API_UpdateDomainConfig.html) request tests the deployment type that results from enabling UltraWarm: ``` POST https://es.us-east-1.amazonaws.com/2021-01-01/opensearch/domain/my-domain/config { "ClusterConfig": { "WarmCount": 3, "WarmEnabled": true, "WarmType": "ultrawarm1.large.search" }, "DryRun": true, "DryRunMode": "Verbose" } ``` The request runs a validation check and returns the type of deployment the change will cause but doesn't actually perform the update: ``` { "ClusterConfig": { ... }, "DryRunResults": { "DeploymentType": "Blue/Green", "Message": "This change will require a blue/green deployment." } } ``` Possible deployment types are: + `Blue/Green` – The change will cause a blue/green deployment. + `DynamicUpdate` – The change won't cause a blue/green deployment. + `Undetermined` – The domain is still in a processing state, so the deployment type can't be determined. + `None` – No configuration change. If the validation fails, it returns a list of [validation failures](#validation). ``` { "ClusterConfig":{ "..." }, "DryRunProgressStatus":{ "CreationDate":"2023-01-12T01:14:33.847Z", "DryRunId":"db00ca39-48b2-4774-bbd3-252cf094d205", "DryRunStatus":"failed", "UpdateDate":"2023-01-12T01:14:33.847Z", "ValidationFailures":[ { "Code":"Cluster.Index.WriteBlock", "Message":"Cluster has index write blocks." } ] } } ``` If the status is still `pending`, you can use the dry run ID in your UpdateDomainConfig response in subsequent [DescribeDryRunProgress](https://docs.aws.amazon.com/opensearch-service/latest/APIReference/API_DescribeDryRunProgress.html) calls to check the status of the validation. ``` GET https://es.us-east-1.amazonaws.com/2021-01-01/opensearch/domain/my-domain/dryRun?dryRunId=my-dry-run-id { "DryRunConfig": null, "DryRunProgressStatus": { "CreationDate": "2023-01-12T01:14:42.998Z", "DryRunId": "db00ca39-48b2-4774-bbd3-252cf094d205", "DryRunStatus": "succeeded", "UpdateDate": "2023-01-12T01:14:49.334Z", "ValidationFailures": null }, "DryRunResults": { "DeploymentType": "Blue/Green", "Message": "This change will require a blue/green deployment." } } ``` To run a dry run analysis without a validation check, set `DryRunMode` to `Basic` when you use the configuration API. ------ #### [ Python ] The following Python code uses the [UpdateDomainConfig](https://docs.aws.amazon.com/opensearch-service/latest/APIReference/API_UpdateDomainConfig.html) API to perform a dry run validation check and, if the check succeeds, calls the same API without a dry run to start the update. If the check fails, the script prints out the error and stops. ``` import time import boto3 client = boto3.client('opensearch') response = client.UpdateDomainConfig( ClusterConfig={ 'WarmCount': 3, 'WarmEnabled': True, 'WarmCount': 123, }, DomainName='test-domain', DryRun=True, DryRunMode='Verbose' ) dry_run_id = response.DryRunProgressStatus.DryRunId retry_count = 0 while True: if retry_count == 5: print('An error occured') break dry_run_progress_response = client.DescribeDryRunProgress('test-domain', dry_run_id) dry_run_status = dry_run_progress_response.DryRunProgressStatus.DryRunStatus if dry_run_status == 'succeeded': client.UpdateDomainConfig( ClusterConfig={ 'WarmCount': 3, 'WarmEnabled': True, 'WarmCount': 123, }) break elif dry_run_status == 'failed': validation_failures_list = dry_run_progress_response.DryRunProgressStatus.ValidationFailures for item in validation_failures_list: print(f"Code: {item['Code']}, Message: {item['Message']}") break retry_count += 1 time.sleep(30) ``` ------ ## Tracking a configuration change You can request one configuration change at a time, or group multiple changes in a single request. Use the **Domain processing status** and **Configuration change status** fields in the console to track configuration changes. Wait for the domain status to become `Active` before you request additional changes. A domain can have the following **processing statuses**: + `Active` – No configuration change is in progress. You can submit a new configuration change request. + `Creating` – Domain is being created. + `Modifying` – Configuration changes, such as the addition of new data nodes, EBS, gp3, IOPS provisioning, or setting up KMS keys, are in progress. + `Upgrading engine version` – An engine version upgrade is in progress. + `Updating service software` – A service software update is in progress. + `Deleting` – The domain is being deleted. + `Isolated` – The domain is suspended. A domain can have the following **configuration change statuses**: + `Pending` – A configuration change request has been submitted. + `Initializing` – The service is initializing a configuration change. + `Validating` – The service is validating the requested changes and resources required. + `Awaiting user inputs` – The service expects configuration changes, such as an instance type change, to proceed. You can edit the configuration changes. + `Applying changes` – Service is applying the requested configuration changes. + `Cancelled` – The configuration change is cancelled. Choose **Cancel** roll back all changes. + `Completed` – The requested configuration changes have been successfully completed. + `Validation failed` – The requested configuration changes failed to complete. No configuration changes were applied. **Note** Validation failures can be the result of red indexes present in your domain, unavailability of a chosen instance type, or low disk space. For a list of validation errors, see [Troubleshooting validation errors](#validation). During a validation failure event, you can cancel, retry, or edit configuration changes. When the configuration changes are complete, the domain status changes back to `Active`. You can review the cluster health and Amazon CloudWatch metrics and see that the number of nodes in the cluster temporarily increases—often doubling—while the domain update occurs. In the following illustration, you can see the number of nodes doubling from 11 to 22 during a configuration change and returning to 11 when the update is complete. ![\[Number of nodes doubling from 11 to 22 during a domain configuration change.\]](http://docs.aws.amazon.com/opensearch-service/latest/developerguide/images/NodesDoubled.png) This temporary increase can strain the cluster's [dedicated master nodes](managedomains-dedicatedmasternodes.md), which suddenly might have many more nodes to manage. It can also increase search and indexing latencies as OpenSearch Service copies data from the old cluster to the new one. It's important to maintain sufficient capacity on the cluster to handle the overhead that is associated with these blue/green deployments. **Important** You do *not* incur any additional charges during configuration changes and service maintenance. You're billed only for the number of nodes that you request for your cluster. For specifics, see [Charges for configuration changes](#managedomains-config-charges). To prevent overloading dedicated master nodes, you can [monitor usage with the Amazon CloudWatch metrics](managedomains-cloudwatchmetrics.md). For recommended maximum values, see [Recommended CloudWatch alarms for Amazon OpenSearch Service](cloudwatch-alarms.md). ## Stages of a configuration change After you initiate a configuration change, OpenSearch Service goes through a series of steps to update your domain. You can view the progress of the configuration change under **Configuration change status** in the console. The exact steps that an update goes through depends on the type of change you're making. You can also monitor a configuration change using the [DescribeDomainChangeProgress](https://docs.aws.amazon.com/opensearch-service/latest/APIReference/API_DescribeDomainChangeProgress.html) API operation. The following are possible stages an update can go through during a configuration change: | Stage name | Description | | --- | --- | | Validation | Validating that the domain is eligible for an update, and surfacing [validation issues](#validation) if necessary. | | Creating a new environment | Completing the necessary prerequisites and creating required resources to start the blue/green deployment. | | Provisioning new nodes | Creating a new set of instances in the new environment. | | Traffic routing on new nodes | Redirecting traffic to the newly created data nodes. | | Traffic routing on old nodes | Disabling traffic on the old data nodes. | | Preparing nodes for removal | Preparing to remove nodes. This step only happens when you're downscaling your domain (for example, from 8 nodes to 6 nodes). | | Copying shards to new nodes | Moving shards from the old nodes to the new nodes. | | Terminating nodes | Terminating and deleting old nodes after shards are removed. | | Deleting older resources | Deleting resources associated with the old environment (e.g. load balancer). | | Dynamic update | Displayed when the update does not require a blue/green deployment and can be dynamically applied. | | Applying dedicated master related changes | Displayed when the dedicated master instance type or count is changed. | | Applying volume related changes | Displayed when volume size, type, IOPS and throughput are changed. | ## Performance impact of blue/green deployments During blue/green deployment your Amazon OpenSearch Service cluster is available for incoming search and indexing requests. However, you might experience the following performance issues: + Temporary increase in usage on leader nodes as clusters have more nodes to manage. + Increased search and indexing latency as OpenSearch Service copies data from old nodes to new nodes. + Increased rejections for incoming requests as the cluster load increases during blue/green deployments. + To avoid latency issues and request rejections, you should run blue/green deployments when the cluster is healthy and there's low network traffic. ## Charges for configuration changes If you change the configuration for a domain, OpenSearch Service creates a new cluster as described in [Making configuration changes in Amazon OpenSearch Service](#managedomains-configuration-changes). During the migration of old to new, you incur the following charges: + If you change the instance type, you're charged for both clusters for the first hour. After the first hour, you're only charged for the new cluster. EBS volumes aren't charged twice because they're part of your cluster, so their billing follows instance billing. **Example**: You change the configuration from three `m3.xlarge` instances to four `m4.large` instances. For the first hour, you're charged for both clusters (3 \$1 `m3.xlarge` \$1 4 \$1 `m4.large`). After the first hour, you're charged only for the new cluster (4 \$1 `m4.large`). + If you don't change the instance type, you're charged only for the largest cluster for the first hour. After the first hour, you're charged only for the new cluster. **Example**: You change the configuration from six `m3.xlarge` instances to three `m3.xlarge` instances. For the first hour, you're charged for the largest cluster (6 \$1 `m3.xlarge`). After the first hour, you're charged only for the new cluster (3 \$1 `m3.xlarge`). ## Troubleshooting validation errors When you initiate a configuration change or perform an OpenSearch or Elasticsearch version upgrade, OpenSearch Service first performs a series of validation checks to ensure that your domain is eligible for an update. If any of these checks fail, you receive a notification in the console containing the specific issues that you must fix before updating your domain. OpenSearch Service isolates your domain if it remains in an unusable state for more than 60 days. We'll send you notification reminders to resolve these issues. If you don't fix the issues within this time frame, OpenSearch Service deletes your domain and its data. The following table lists the possible domain issues that OpenSearch Service might surface, and steps to resolve them. | Issue | Error code | Troubleshooting steps | | --- | --- | --- | | Security group not found | SecurityGroupNotFound | The security group associated with your OpenSearch Service domain does not exist. To resolve this issue, [create a security group](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_SecurityGroups.html#creating-security-groups) with the specified name. | | Subnet not found | SubnetNotFound | The subnet associated with your OpenSearch Service domain does not exist. To resolve this issue, [create a subnet](https://docs.aws.amazon.com/vpc/latest/userguide/working-with-subnets.html#create-subnets) in your VPC. | | Service-linked role not configured | SLRNotConfigured | The [service-linked role](slr.md) for OpenSearch Service is not configured. The service-linked role is predefined by OpenSearch Service and includes all the permissions the service requires to call other AWS services on your behalf. If the role doesn't exist, you might need to [create it manually](slr-aos.md#create-slr). | | Not enough IP addresses | InsufficientFreeIPsForSubnets | One or more of your VPC subnets don't have enough IP addresses to update your domain. To calculate how many IP addresses you need, see [Reserving IP addresses in a VPC subnet](vpc.md#reserving-ip-vpc-endpoints). | | Cognito user pool doesn't exist | CognitoUserPoolNotFound | OpenSearch Service can't find the Amazon Cognito user pool. Confirm that you created one and have the correct ID. To find the ID, you can use the Amazon Cognito console or the following AWS CLI command: 
aws cognito-idp list-user-pools --max-results 60 --region us-east-1
| | Cognito identity pool doesn't exist | CognitoIdentityPoolNotFound | OpenSearch Service can't find the Cognito identity pool. Confirm that you created one and have the correct ID. To find the ID, you can use the Amazon Cognito console or the following AWS CLI command: 
aws cognito-identity list-identity-pools --max-results 60 --region us-east-1
| | Cognito domain not found for user pool | CognitoDomainNotFound | The user pool does not have a domain name. You can configure one using the Amazon Cognito console or the following AWS CLI command: 
aws cognito-idp create-user-pool-domain --domain my-domain --user-pool-id id
| | Cognito role not configured | CognitoRoleNotConfigured | The IAM role that grants OpenSearch Service permission to configure the Amazon Cognito user and identity pools, and use them for authentication, is not configured. Configure the role with an appropriate permission set and trust relationship. You can use the console, which creates the default [CognitoAccessForAmazonOpenSearch](cognito-auth.md#cognito-auth-role) role for you, or you can manually configure a role using the AWS CLI or the AWS SDK. | | Unable to describe user pool | UserPoolNotDescribable | The specified Amazon Cognito role doesn't have permission to describe the user pool associated with your domain. Make sure the role permissions policy allows the cognito-identity:DescribeUserPool action. See [About the CognitoAccessForAmazonOpenSearch role](cognito-auth.md#cognito-auth-role) for the full permissions policy. | | Unable to describe identity pool | IdentityPoolNotDescribable | The specified Amazon Cognito role doesn't have permission to describe the identity pool associated with your domain. Make sure the role permissions policy allows the cognito-identity:DescribeIdentityPool action. See [About the CognitoAccessForAmazonOpenSearch role](cognito-auth.md#cognito-auth-role) for the full permissions policy. | | Unable to describe user and identity pool | CognitoPoolsNotDescribable | The specified Amazon Cognito role doesn't have permission to describe the user and identity pools associated with your domain. Make sure the role permissions policy allows the cognito-identity:DescribeIdentityPool and cognito-identity:DescribeUserPool actions. See [About the CognitoAccessForAmazonOpenSearch role](cognito-auth.md#cognito-auth-role) for the full permissions policy. | | KMS key not enabled | KMSKeyNotEnabled | The AWS Key Management Service (AWS KMS) key used to encrypt your domain is disabled. [Re-enable the key](https://docs.aws.amazon.com/kms/latest/developerguide/enabling-keys) immediately. | | Custom certificate not in ISSUED state | InvalidCertificate | If your domain uses a custom endpoint, you secure it by either generating an SSL certificate in AWS Certificate Manager (ACM) or importing one of your own. The certificate status must be **Issued**. If you receive this error, [check the status of your certificate](https://docs.aws.amazon.com/acm/latest/userguide/gs-acm-describe.html) in the ACM console. If the status is Expired, Failed, Inactive, or Pending validation, see the ACM [troubleshooting documentation](https://docs.aws.amazon.com/acm/latest/userguide/troubleshooting.html) to resolve the issue. | | Not enough capacity to launch chosen instance type | InsufficientInstanceCapacity | The requested instance type capacity is not available. For example, you might have requested five `i3.16xlarge.search` nodes, but OpenSearch Service doesn't have enough `i3.16xlarge.search` hosts available, so the request can't be fulfilled. Check the [supported instance types](supported-instance-types.md) in OpenSearch Service and choose a different instance type. | | Red indexes in cluster | RedCluster | One or more indexes in your cluster have a red status, leading to an overall red cluster status. To troubleshoot and remediate this issue, see [Red cluster status](handling-errors.md#handling-errors-red-cluster-status). | | Memory circuit breaker, too many requests | TooManyRequests | There are too many search and write requests to your domain, so OpenSearch Service can't update its configuration. You can reduce the number of requests, scale instances vertically up to 64 GiB of RAM, or scale horizontally by adding instances. | | New configuration can't hold data (low disk space) | InsufficientStorageCapacity | The configured storage size can't hold all of the data on your domain. To resolve this issue, [choose a larger volume](limits.md#ebsresource), [delete unused indexes](https://opensearch.org/docs/latest/opensearch/rest-api/index-apis/delete-index/), or increase the number of nodes in the cluster to immediately free up disk space. | | Shards pinned to specific nodes | ShardMovementBlocked | One or more indexes in your domain are attached to specific nodes and can't be reassigned. This most likely happened because you configured shard allocation filtering, which lets you specify which nodes are allowed to host the shards of a particular index. To resolve this issue, remove shard allocation filters from all affected indexes: 
PUT my-index/_settings
{  
  "settings": {    
    "index.routing.allocation.require._name": null  
  }
}
| | New configuration can't hold all shards (shard count) | TooManyShards | The shard count on your domain is too high, which prevents OpenSearch Service from moving them to the new configuration. To resolve this issue, scale your domain horizonally by adding nodes of the same configuration type as your current cluster nodes. Note that the [maximum EBS volume size](limits.md#ebsresource) depends on the node's instance type.To prevent this issue in the future, see [Choosing the number of shards](bp-sharding.md) and define a sharding strategy that is appropriate for your use case. | | The subnet associated with your domain does not support IPv4 addresses | `ResultCodeIPv4BlockNotExists` | To resolve this issue, [create a subnet or update the existing subnet](https://docs.aws.amazon.com//vpc/latest/userguide/configure-subnets.html#subnet-IP-address-range) in your VPC according to the configured IP address type of the domain. If your domain uses an **IPv4 only** address type, use an IPv4-only subnet. If your domain uses **Dual-stack mode**, use a dual-stack subnet. | | The subnet associated with your domain does not support IPv6 addresses | `ResultCodeIPv6BlockNotExists` | To resolve this issue, [create a subnet or update the existing subnet](https://docs.aws.amazon.com//vpc/latest/userguide/configure-subnets.html#subnet-IP-address-range) in your VPC according to the configured IP address type of the domain. If your domain uses an **IPv4 only** address type, use an IPv4-only subnet. If your domain uses **Dual-stack mode**, use a dual-stack subnet. | # Service software updates in Amazon OpenSearch Service **Note** For explanations of the changes and additions made in each *major* (non-patch) service software update, see the [release notes](release-notes.md). Amazon OpenSearch Service regularly releases service software updates that add features or otherwise improve your domains. The **Notifications** panel in the console is the easiest way to see if an update is available or to check the status of an update. Each notification includes details about the service software update. All service software updates use blue/green deployments to minimize downtime. Service software updates differ from OpenSearch Service *version* upgrades. For information about upgrading to a later version of OpenSearch Service, see [Upgrading Amazon OpenSearch Service domains](version-migration.md). OpenSearch Service requires you to apply required service software updates within 30 days of their availability. These updates are critical for maintaining security compliance. If you don't apply required updates within the 30-day period, you'll receive reminder notifications every 15 days for 30 days. After this period without compliance, your domain will be isolated with the following effects: + All network access to your domain is removed + Domain status changes to **isolated** + The domain remains unusable until you apply the required updates During isolation, you'll continue receiving reminder notifications every 15 days for 60 days. If you don't apply the required updates within this period, your OpenSearch Service domain and all associated data will be permanently deleted. For more information, see [Troubleshooting validation errors](managedomains-configuration-changes.md#validation). ## Optional versus required updates OpenSearch Service has two broad categories of service software updates: ### Optional updates Optional service software updates generally include enhancements and support for new features or functionality. Optional updates aren't enforced on your domains, and there's no hard deadline to install them. The availability of the update is communicated through email and a console notification. You can choose to apply the update immediately or reschedule it for a more appropriate date and time. You can also schedule it during the domain's [off-peak window](off-peak.md). The majority of software updates are optional. Regardless of whether or not you schedule an update, if you make a change on the domain that causes a [blue/green deployment](managedomains-configuration-changes.md), OpenSearch Service automatically updates your service software for you. You can configure your domain to automatically apply optional updates during [off-peak hours](off-peak.md). When this option is turned on, OpenSearch Service waits at least 13 days from when an optional update is available and then schedules the update after seven days. You receive a console notification when the update is scheduled and you can choose to reschedule it for a later date. To turn on automatic software updates, select **Enable automatic software update** when you create or update your domain. To configure the same setting using the AWS CLI, set `--software-update-options` to `true` when you create or update your domain. ### Required updates Required service software updates generally include critical security fixes or other mandatory updates to ensure the continued integrity and functionality of your domain. Examples of required updates are Log4j Common Vulnerabilities and Exposures (CVEs) and enforcement of Instance Metadata Service Version 2 (IMDSv2). The number of mandatory updates in a year is usually less than three. OpenSearch Service automatically schedules these updates and notifies you seven days before the scheduled update through email and a console notification. You can choose to apply the update immediately or reschedule it for a more appropriate date and time *within the allowed timeframe.* You can also schedule it during the domain's next [off-peak window](off-peak.md). If you take no action on a required update and you don't make any domain changes that cause a blue/green deployment, OpenSearch Service can initiate the update at any time beyond the specified deadline (typically 14 days from availability), within the domain's off-peak window. Regardless of when the update is scheduled for, if you make a change on the domain that causes a [blue/green deployment](managedomains-configuration-changes.md), OpenSearch Service automatically updates your domain for you. ## Patch updates Service software versions that end in "-P" and a number, such as R20211203-*P4*, are patch releases. Patches are likely to include performance improvements, minor bug fixes, and security fixes or posture improvements. Patch releases do not include new features or breaking changes, and they generally don't have a direct or noticeable impact on users. The service software notification tells you if a patch release is optional or mandatory. ## Considerations Consider the following when deciding whether to update your domain: + Manually updating your domain lets you take advantage of new features more quickly. When you choose **Update**, OpenSearch Service places the request in a queue and begins the update when it has time. + When you initiate a service software update, OpenSearch Service sends a notification when the update starts and when it completes. + Software updates use blue/green deployments to minimize downtime. Updates can temporarily strain a cluster's dedicated master nodes, so make sure to maintain sufficient capacity to handle the associated overhead. + Updates typically complete within minutes, but can also take several hours or even days if your system is experiencing heavy load. Consider updating your domain during the configured [off-peak window](off-peak.md) to avoid long update periods. ## Starting a service software update You can request a service software update through the OpenSearch Service console, the AWS CLI, or one of the SDKs. ### Console **To request a service software update** 1. Open the Amazon OpenSearch Service console at [https://console.aws.amazon.com/aos/home](https://console.aws.amazon.com/aos/home). 1. Select the domain name to open its configuration. 1. Choose **Actions**, **Update** and select one of the following options: + **Apply update now** - Immediately schedules the action to happen in the current hour *if there's capacity available*. If capacity isn't available, we provide other available time slots to choose from. + **Schedule it in off-peak window** – Only available if the off-peak window is enabled for the domain. Schedules the update to take place during the domain's configured off-peak window. There's no guarantee that the update will happen during the next immediate window. Depending on capacity, it might happen in subsequent days. For more information, see [Scheduling software updates during off-peak windows](#service-software-offpeak). + **Schedule for specific date and time** – Schedules the update to take place at a specific date and time. If the time that you specify is unavailable for capacity reasons, you can select a different time slot. If you schedule the update for a later date (within or outside the domain's off-peak window), you can reschedule it at any time. For instructions, see [Rescheduling actions](off-peak.md#off-peak-reschedule). 1. Choose **Confirm**. ### AWS CLI Send a [start-service-software-update](https://docs.aws.amazon.com/cli/latest/reference/opensearch/start-service-software-update.html) AWS CLI request to initiate a service software update. This example adds the update to the queue immediately: ``` aws opensearch start-service-software-update \ --domain-name my-domain \ --schedule-at "NOW" ``` **Response**: ``` { "ServiceSoftwareOptions": { "CurrentVersion": "R20220928-P1", "NewVersion": "R20220928-P2", "UpdateAvailable": true, "Cancellable": true, "UpdateStatus": "PENDING_UPDATE", "Description": "", "AutomatedUpdateDate": "1969-12-31T16:00:00-08:00", "OptionalDeployment": true } } ``` **Tip** After you request an update, you have a narrow window of time in which you can cancel it. The duration of this `PENDING_UPDATE` state can vary greatly and depends on your AWS Region and the number of concurrent updates that OpenSearch Service is performing. To cancel an update, use the console or `cancel-service-software-update` AWS CLI command. If the request fails with a `BaseException`, it means that the time you specified isn't available for capacity reasons, and you must specify a different time. OpenSearch Service provides alternate available slot suggestions in the response. ### AWS SDKs This sample Python script uses the [describe\$1domain](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/opensearch.html#OpenSearchService.Client.describe_domain) and [start\$1service\$1software\$1update](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/opensearch.html#OpenSearchService.Client.start_service_software_update) methods from the AWS SDK for Python (Boto3) to check whether a domain is eligible for a service software update and if so, starts the update. You must provide a value for `domain_name`. ``` import boto3 from botocore.config import Config import time # Build the client using the default credential configuration. # You can use the CLI and run 'aws configure' to set access key, secret # key, and default region. my_config = Config( # Optionally lets you specify a Region other than your default. region_name='us-east-1' ) domain_name = '' # The name of the domain to check and update client = boto3.client('opensearch', config=my_config) def getUpdateStatus(client): """Determines whether the domain is eligible for an update""" response = client.describe_domain( DomainName=domain_name ) sso = response['DomainStatus']['ServiceSoftwareOptions'] if sso['UpdateStatus'] == 'ELIGIBLE': print('Domain [' + domain_name + '] is eligible for a service software update from version ' + sso['CurrentVersion'] + ' to version ' + sso['NewVersion']) updateDomain(client) else: print('Domain is not eligible for an update at this time.') def updateDomain(client): """Starts a service software update for the eligible domain""" response = client.start_service_software_update( DomainName=domain_name ) print('Updating domain [' + domain_name + '] to version ' + response['ServiceSoftwareOptions']['NewVersion'] + '...') waitForUpdate(client) def waitForUpdate(client): """Waits for the domain to finish updating""" response = client.describe_domain( DomainName=domain_name ) status = response['DomainStatus']['ServiceSoftwareOptions']['UpdateStatus'] if status == 'PENDING_UPDATE' or status == 'IN_PROGRESS': time.sleep(30) waitForUpdate(client) elif status == 'COMPLETED': print('Domain [' + domain_name + '] successfully updated to the latest software version') else: print('Domain is not currently being updated.') def main(): getUpdateStatus(client) ``` ## Scheduling software updates during off-peak windows Each OpenSearch Service domain created after February 16, 2023 has a daily 10-hour window between 10:00 P.M. and 8:00 A.M. local time that we consider the [off-peak window](off-peak.md). OpenSearch Service uses this window to schedule service software updates for the domain. Off-peak updates help to minimize strain on a cluster's dedicated master nodes during higher traffic periods. OpenSearch Service can't initiate updates outside of this 10-hour window without your consent. + For *optional* updates, OpenSearch Service notifies you of the update's availability and prompts you to schedule the update during an upcoming off-peak window. + For *required* updates, OpenSearch Service automatically schedules the update during an upcoming off-peak window and notifies you three days ahead of time. You can reschedule the update (for within or outside the off-peak window), but only within the required timeframe for the update to be completed. For each domain, you can choose to override the default 10:00 P.M. start time with a custom time. For instructions, see [Configuring a custom off-peak window](off-peak.md#off-peak-custom). ### Console **To schedule an update during an upcoming off-peak window** 1. Open the Amazon OpenSearch Service console at [https://console.aws.amazon.com/aos/home](https://console.aws.amazon.com/aos/home ). 1. Select the domain name to open its configuration. 1. Choose **Actions**, **Update**. 1. Select **Schedule it in off-peak window**. 1. Choose **Confirm**. You can view the scheduled action on the **Off-peak window** tab and reschedule it at any time. See [Viewing scheduled actions](off-peak.md#off-peak-view). ### CLI To schedule an update during an upcoming off-peak window using the AWS CLI, send a [StartServiceSoftwareUpdate](https://docs.aws.amazon.com/opensearch-service/latest/APIReference/API_StartServiceSoftwareUpdate.html) request and specify `OFF_PEAK_WINDOW` for the `--schedule-at` parameter: ``` aws opensearch start-service-software-update \ --domain-name my-domain \ --schedule-at "OFF_PEAK_WINDOW" ``` ## Monitoring service software updates OpenSearch Service sends a [notification](managedomains-notifications.md) when a service software update is available, required, started, completed, or failed. You can view these notifications on the **Notifications** panel of the OpenSearch Service console. The notification severity is `Informational` if the update is optional and `High` if it's required. OpenSearch Service also sends service software events to Amazon EventBridge. You can use EventBridge to configure rules that send an email or perform a specific action when an event is received. For an example walk through, see [Tutorial: Sending Amazon SNS alerts for available software updates](sns-events.md). To see the format of each service software event sent to Amazon EventBridge, see [Service software update events](monitoring-events.md#monitoring-events-sso). ## When domains are ineligible for an update Your domain is ineligible for a service software update if it's in any of the following states: | State | Description | | --- | --- | | Domain in processing | The domain is in the middle of a configuration change. Check update eligibility after the operation completes. | | Red cluster status | One or more indexes in the cluster is red. For troubleshooting steps, see [Red cluster status](handling-errors.md#handling-errors-red-cluster-status). | | High error rate | The OpenSearch cluster is returning a large number of 5*xx* errors when attempting to process requests. This problem is usually the result of too many simultaneous read or write requests. Consider reducing traffic to the cluster or scaling your domain. | | Split brain | *Split brain* means your OpenSearch cluster has more than one master node and has split into two clusters that never will rejoin on their own. You can avoid split brain by using the recommended number of [dedicated master nodes](managedomains-dedicatedmasternodes.md). For help recovering from split brain, contact [Support](https://console.aws.amazon.com/support/home). | | Amazon Cognito integration issue | Your domain uses [authentication for OpenSearch Dashboards](cognito-auth.md), and OpenSearch Service can't find one or more Amazon Cognito resources. This problem usually occurs if the Amazon Cognito user pool is missing. To correct the issue, recreate the missing resource and configure the OpenSearch Service domain to use it. | | Other service issue | Issues with OpenSearch Service itself might cause your domain to display as ineligible for an update. If none of the previous conditions apply to your domain and the problem persists for more than a day, contact [Support](https://console.aws.amazon.com/support/home). | # Defining off-peak windows for Amazon OpenSearch Service When you create an Amazon OpenSearch Service domain, you define a daily 10-hour window that's considered *off-peak* hours. OpenSearch Service uses this window to schedule service software updates and Auto-Tune optimizations that require a [blue/green deployment](managedomains-configuration-changes.md) during comparatively lower traffic times, whenever possible. Blue/green refers to the process of creating a new environment for domain updates and routing users to the new environment after those updates are complete. Although blue/green deployments are non-disruptive, to minimize any potential [performance impact](managedomains-configuration-changes.md#initiating-tracking-configuration-changes) while resources are being consumed for a blue/green deployment, we recommend that you schedule these deployments during the domain's configured off-peak window. Updates such as node replacements, or those that need to be deployed to the domain immediately, don't use the off-peak window. You can modify the start time for the off-peak window, but you can't modify the length of the window. **Note** Off-peak windows were introduced on February 16, 2023. All domains created before this date have the off-peak window disabled by default. You must manually enable and configure the off-peak window for these domains. All domains created *after* this date will have the off-peak window enabled by default. You can't disable the off-peak window for a domain after it's enabled. ## Off-peak service software updates OpenSearch Service has two broad categories of service software updates—*optional* and *required*. Both types require blue/green deployments. Optional updates aren't enforced on your domains, while required updates are automatically installed if you take no action before the specified deadline (typically two weeks from availability). For more information, see [Optional versus required updates](service-software.md#service-software-optional-required). When you initiate an *optional* update, you have the choice to apply the update immediately, schedule it for a subsequent off-peak window, or specify a custom date and time to apply it. ![\[Software update dialog with options to apply now, schedule off-peak, or set specific time.\]](http://docs.aws.amazon.com/opensearch-service/latest/developerguide/images/off-peak-sso.png) For *required* updates, OpenSearch Service automatically schedules a date and time during off-peak hours to perform the update. You receive a notification three days before the scheduled update, and you can choose to reschedule it for a later date and time within the required deployment period. For instructions, see [Rescheduling actions](#off-peak-reschedule). ## Off-peak Auto-Tune optimizations Previously, Auto-Tune used [maintenance windows](auto-tune-schedule.md) to schedule changes that required a blue/green deployment. Domains that already had Auto-Tune and maintenance windows enabled prior to the introduction of off-peak windows will continue to use maintenance windows for these updates, unless you migrate them to use the off-peak window. We recommend that you migrate your domains to use the off-peak window, as it's used to schedule other activities on the domain such as service softwate updates. For instructions, see [Migrating from Auto-Tune maintenance windows](#off-peak-migrate). You can't revert back to using maintenance windows after you migrate your domain to the off-peak window. All domains created after February 16, 2023 will use the off-peak window, rather than legacy maintenance windows, to schedule blue/green deployments. You can't disable the off-peak window for a domain. For a list of Auto-Tune optimizations that require blue/green deployments, see [Types of changes](auto-tune.md#auto-tune-types). ## Enabling the off-peak window Any domains created before February 16, 2023 (when off-peak windows were introduced) have the feature disabled by default. You must manually enable it for these domains. You can't disable the off-peak window after it's enabled. ### Console **To enable the off-peak window for a domain** 1. Open the Amazon OpenSearch Service console at [https://console.aws.amazon.com/aos/home](https://console.aws.amazon.com/aos/home ). 1. Select the name of the domain to open its configuration. 1. Navigate to the **Off-peak window** tab and choose **Edit**. 1. Specify a custom start time in Coordinated Universal Time (UTC). For example, to configure a start time of 11:30 P.M. in the US West (Oregon) Region, specify **07:30**. 1. Choose **Save changes**. ### CLI To modify the off-peak window using the AWS CLI, send an [UpdateDomainConfig](https://docs.aws.amazon.com/opensearch-service/latest/APIReference/API_UpdateDomainConfig.html) request: ``` aws opensearch update-domain-config \ --domain-name my-domain \ --off-peak-window-options 'Enabled=true, OffPeakWindow={WindowStartTime={Hours=02,Minutes=00}}' ``` If you don't specify a custom window start time, it defaults to 00:00 UTC. ## Configuring a custom off-peak window You specify a custom off-peak window for your domain in Coordinated Universal Time (UTC). For example, if your want the off-peak window to start at 11:00 P.M. for a domain in the US East (N. Virginia) Region, you'd specify 04:00 UTC. ### Console **To modify the off-peak window for a domain** 1. Open the Amazon OpenSearch Service console at [https://console.aws.amazon.com/aos/home](https://console.aws.amazon.com/aos/home ). 1. Select the name of the domain to open its configuration. 1. Navigate to the **Off-peak window** tab. You can view the configured off-peak window and a list of upcoming scheduled actions for the domain. 1. Choose **Edit** and specify a new start time in UTC. For example, to configure a start time of 9:00 PM in the US East (N. Virginia) Region, specify **02:00 UCT**. 1. Choose **Save changes**. ### CLI To configure a custom off-peak window using the AWS CLI, send an [UpdateDomainConfig](https://docs.aws.amazon.com/opensearch-service/latest/APIReference/API_UpdateDomainConfig.html) request and specify the hour and minute in 24-hour time format. For example, the following request changes the window start time to 2:00 A.M. UTC: ``` aws opensearch update-domain-config \ --domain-name my-domain \ --off-peak-window-options 'OffPeakWindow={WindowStartTime={Hours=02,Minutes=00}}' ``` If you don't specify a window start time, it defaults to 10:00 P.M. local time for the AWS Region that the domain is created in. ## Viewing scheduled actions You can view all actions that are currently scheduled, in progress, or pending for each of your domains. Actions can have a severity of `HIGH`, `MEDIUM`, and `LOW`. Actions can have the following statuses: + `Pending update` – The action is in the queue to be processed. + `In progress` – The action is currently in progress. + `Failed` – The action failed to complete. + `Completed` – The action has completed successfully. + `Not eligible` – Only for service software updates. The update can't proceed because the cluster is in an unhealthy state. + `Eligible` – Only for service software updates. The domain is eligible for an update. ### Console The OpenSearch Service console displays all scheduled actions within the domain configuration, along with each action's severity and current status. **To view scheduled actions for a domain** 1. Open the Amazon OpenSearch Service console at [https://console.aws.amazon.com/aos/home](https://console.aws.amazon.com/aos/home ). 1. Select the name of the domain to open its configuration. 1. Navigate to the **Off-peak window** tab. 1. Under **Scheduled actions**, view all actions that are currently scheduled, in progress, or pending for the domain. ### CLI To view scheduled actions using the AWS CLI, send a [ListScheduledActions](https://docs.aws.amazon.com/opensearch-service/latest/APIReference/API_ListScheduledActions.html) request: ``` aws opensearch list-scheduled-actions \ --domain-name my-domain ``` **Response**: ``` { "ScheduledActions": [ { "Cancellable": true, "Description": "The Deployment type is : BLUE_GREEN.", "ID": "R20220721-P13", "Mandatory": false, "Severity": "HIGH", "ScheduledBy": "CUSTOMER", "ScheduledTime": 1.673871601E9, "Status": "PENDING_UPDATE", "Type": "SERVICE_SOFTWARE_UPDATE", }, { "Cancellable": true, "Description": "Amazon Opensearch will adjust the young generation JVM arguments on your domain to improve performance", "ID": "Auto-Tune", "Mandatory": true, "Severity": "MEDIUM", "ScheduledBy": "SYSTEM", "ScheduledTime": 1.673871601E9, "Status": "PENDING_UPDATE", "Type": "JVM_HEAP_SIZE_TUNING", } ] } ``` ## Rescheduling actions OpenSearch Service notifies you of scheduled service software updates and Auto-Tune optimizations. You can choose to apply the change immediately, or reschedule it for a later date and time. **Note** OpenSearch Service can schedule the action within an hour of the time you select. For exmple, if you choose to apply an update at 5 P.M., it can be applied between 5 and 6 P.M. ### Console **To reschedule an action** 1. Open the Amazon OpenSearch Service console at [https://console.aws.amazon.com/aos/home](https://console.aws.amazon.com/aos/home ). 1. Select the name of the domain to open its configuration. 1. Navigate to the **Off-peak window** tab. 1. Under **Scheduled actions**, select the action and choose **Reschedule**. 1. Choose one of the following options: + **Apply update now** - Immediately schedules the action to happen in the current hour *if there's capacity available*. If capacity isn't available, we provide other available time slots to choose from. + **Schedule it in off-peak window** - Marks the action to be picked up during an upcoming off-peak window. There's no guarantee that the change will be implemented during the immediate next window. Depending on capacity, it might happen in subsequent days. + **Reschedule this update** - Lets you specify a custom date and time to apply the change. If the time that you specify is unavailable for capacity reasons, you can select a different time slot. + **Cancel scheduled update** - Cancels the update. This option is only available for optional service software updates. It's not available for Auto-Tune actions or mandatory software updates. 1. Choose **Save changes**. ### CLI To reschedule an action using the AWS CLI, send an [UpdateScheduledAction](https://docs.aws.amazon.com/opensearch-service/latest/APIReference/API_UpdateScheduledAction.html) request. To retrieve the action ID, send a `ListScheduledActions` request. The following request reschedules a service software update for a specific date and time: ``` aws opensearch update-scheduled-action \ --domain-name my-domain \ --action-id R20220721-P13 \ --action-type "SERVICE_SOFTWARE_UPDATE" \ --desired-start-time 1677348395000 \ --schedule-at TIMESTAMP ``` **Response**: ``` { "ScheduledAction": { "Cancellable": true, "Description": "Cluster status is updated.", "Id": "R20220721-P13", "Mandatory": false, "ScheduledBy": "CUSTOMER", "ScheduledTime": 1677348395000, "Severity": "HIGH", "Status": "PENDING_UPDATE", "Type": "SERVICE_SOFTWARE_UPDATE" } } ``` If the request fails with a `SlotNotAvailableException`, it means that the time you specified isn't available for capacity reasons, and you must specify a different time. OpenSearch Service provides alternate available slot suggestions in the response. ## Migrating from Auto-Tune maintenance windows If a domain was created before February 16, 2023, it could use [maintenance windows](auto-tune-schedule.md) to schedule Auto-Tune optimizations that require a blue/green deployment. You can migrate your existing Auto-Tune domains to use the off-peak window instead. **Note** You can't revert back to using maintenance windows after you migrate your domain to use off-peak windows. ### Console **To migrate a domain to use the off-peak window** 1. Within the Amazon OpenSearch Service console, select the name of the domain to open its configuration. 1. Go to the **Auto-Tune** tab and choose **Edit**. 1. Select **Migrate to off-peak window**. 1. For **Start time (UTC)**, provide a daily start time for the off-peak window in Universal Coordinated Time (UTC). 1. Choose **Save changes**. ### CLI To migrate from a Auto-Tune maintenance window to the off-peak window using the AWS CLI, send an [UpdateDomainConfig](https://docs.aws.amazon.com/opensearch-service/latest/APIReference/API_UpdateDomainConfig.html) request: ``` aws opensearch update-domain-config \ --domain-name my-domain \ --auto-tune-options DesiredState=ENABLED,UseOffPeakWindow=true,MaintenanceSchedules=[] ``` The off-peak window must be turned on in order for you to migrate a domain from the Auto-Tune maintenance window to the off-peak window. You can enable the off-peak window in a separate request or in the same request. For instructions, see [Enabling the off-peak window](#off-peak-enable). # Notifications in Amazon OpenSearch Service Notifications in Amazon OpenSearch Service contain important information about the performance and health of your domains. OpenSearch Service notifies you about service software updates, Auto-Tune enhancements, cluster health events, and domain errors. Notifications are available for all versions of OpenSearch and Elasticsearch OSS. You can view notifications in the **Notifications** panel of the OpenSearch Service console. All notifications for OpenSearch Service are also surfaced in [Amazon EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-what-is.html). For a full list of notifications and sample events, see [Monitoring OpenSearch Service events with Amazon EventBridge](monitoring-events.md). ## Getting started with notifications Notifications are enabled automatically when you create a domain. Go to the **Notifications** panel of the OpenSearch Service console to monitor and acknowledge notifications. Each notification includes information such as the time it was posted, the domain it relates to, a severity and status level, and a brief explanation. You can view historical notifications for up to 90 days in the console. After accessing the **Notifications** panel or acknowledging a notification, you might receive an error message about not having permissions to perform `es:ListNotificationsV2` or `es:UpdateNotificationStatus`. To resolve this problem, give your user or role the following permissions in IAM: ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [{ "Effect": "Allow", "Action": [ "es:DescribeDomain", "es:ListDomainNames" ], "Resource": "arn:aws:es:*:111122223333:domain/*" }] } ``` ------ The IAM console throws an error ("IAM does not recognize one or more actions.") that you can safely ignore. You can also restrict the `es:UpdateNotificationStatus` action to certain domains. To learn more, see [Policy element reference](ac.md#ac-reference). ## Notification severities Notifications in OpenSearch Service can be *informational*, which relate to any action you've already taken or the operations of your domain, or *actionable*, which require you to take specific actions such as applying a mandatory security patch. Each notification has a severity associated with it, which can be `Informational`, `Low`, `Medium`, `High`, or `Critical`. The following table summarizes each severity: | Severity | Description | Examples | | --- | --- | --- | | Informational | Information related to the operation of your domain. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/opensearch-service/latest/developerguide/managedomains-notifications.html) | | Low | A recommended action, but has no adverse impact on domain availability or performance if no action is taken. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/opensearch-service/latest/developerguide/managedomains-notifications.html) | | Medium | There might be an impact if the recommended action is not taken, but comes with an extended time window for the action to be taken. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/opensearch-service/latest/developerguide/managedomains-notifications.html) | | High | Urgent action is required to avoid adverse impact. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/opensearch-service/latest/developerguide/managedomains-notifications.html) | | Critical | Immediate action is required to avoid adverse impact, or to recover from it. | None currently available | ## Sample EventBridge event The following example shows an OpenSearch Service notification event sent to Amazon EventBridge. The notification has a severity of `Informational` because the update is optional: ``` { "version": "0", "id": "01234567-0123-0123-0123-012345678901", "detail-type": "Amazon OpenSearch Service Software Update Notification", "source": "aws.es", "account": "123456789012", "time": "2016-11-01T13:12:22Z", "region": "us-east-1", "resources": ["arn:aws:es:us-east-1:123456789012:domain/test-domain"], "detail": { "event": "Service Software Update", "status": "Available", "severity": "Informational", "description": "Service software update [R20200330-p1] available." } } ``` # Configuring a multi-AZ domain in Amazon OpenSearch Service To prevent data loss and minimize Amazon OpenSearch Service cluster downtime in the event of a service disruption, you can distribute nodes across two or three *Availability Zones* in the same Region, a configuration known as Multi-AZ. Availability Zones are isolated locations within each AWS Region. For domains that run production workloads, we recommend the Multi-AZ with Standby deployment option, which creates the following configuration: + The domain deployed across three zones. + Current-generation instance types for dedicated master nodes and data nodes. + Three dedicated master nodes and three (or a multiple of three) data nodes. + At least two replicas for each index in your domain, or a multiple of three copies of data (including both primary nodes and replicas). The rest of this section provides explanations for and context around these configurations. ## Multi-AZ with Standby Multi-AZ with Standby is a deployment option for Amazon OpenSearch Service domains that offers 99.99% availability, consistent performance for production workloads, and simplified domain configuration and management. When you use Multi-AZ with Standby, domains are resilient to infrastructure failures, with no impact to performance or availability. This deployment option achieves this standard by mandating a number of best practices, such as a specified data node count, master node count, instance type, replica count, software update settings, and Auto-Tune turned on. **Note** Multi-AZ with Standby is available for OpenSearch version 1.3 and above, and requires regions with at least three Availability Zones. When you use Multi-AZ with Standby, OpenSearch Service creates a domain across three Availability Zones, with each zone containing a complete copy of data and with the data equally distributed in each of the zones. Your domain reserves nodes in one of these zones as standby, which means that they don't serve search requests. When OpenSearch Service detects a failure in the underlying infrastructure, it automatically activates the standby nodes in less than a minute. The domain continues to serve indexing and search requests, and any impact is limited to the time it takes to perform the failover. There is no redistribution of data or resources, which results in unaffected cluster performance and no risk of degraded availability. Multi-AZ with Standby is available at no extra cost. You have two options to create a domain with standby on the AWS Management Console. First, you can create a domain with the **Easy create** creation method, and OpenSearch Service will automatically use a predetermined configuration, which includes the following: + Three Availability Zones, with one acting as a standby + Three dedicated master node and data nodes + Auto-Tune enabled on the domain + GP3 storage for the data nodes You can also choose the **Standard create** creation method and select **Domain with standby** as your deployment option. This allows you to customize your domain while still mandating key features of standby, such as three zones and three master nodes. We recommend choosing a data node count that's a multiple of three (the number of Availability Zones). Once you've created your domain, you can navigate to the domain details pages and, in the **Cluster configuration** tab, confirm that *3-AZ with standby* appears under Availability Zone(s). If you have problems migrating an existing domain to Multi-AZ with Standby, see [Error migrating to Multi-AZ with Standby](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/handling-errors.html#troubleshooting-multi-az-standby) in the troubleshooting guide. ### Limitations When you set up a domain with Multi-AZ with Standby, consider the following limitations: + The total number of shards on a node can't exceed 1000, the total number of shards on a cluster can't exceed 75000, and the size of a single shard can't exceed 65 GB. + Multi-AZ with Standby only works with the `m5`, `c5`, `r5`, `r6g`, `c6g`, `m6g`, `r6gd`, `r7g`, `c7g`, `m7g`, `r7gd`, and `i3` instance types. For more information on supported instances, see [Supported instance types](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/supported-instance-types.html). + You can only use Provisioned IOPs SSD, General Purpose SSD (GP3), or instance-backed storage with standby. + If you enable [UltraWarm](ultrawarm.md) on a Multi-AZ with Standby domain, the number of warm nodes must be a multiple of the number of Availability Zones being used. ## Multi-AZ without Standby OpenSearch Service still supports multi-AZ without Standby, which offers 99.9% availability. Nodes are distributed across Availability Zone(s), and availability depends on the number of Availability Zones and copies of data. Whereas with standby you have to configure your domain with best practices, without standby you can choose your own number of Availability Zones, nodes, and replicas. We don't recommend this option unless you have existing workflows that would be disrupted by creating domains with standby. If you choose this option, we still recommend that you select three Availability Zones in order to remain resilient to node, disk, and single-AZ failures. When a failure occurs, the cluster redistributes data across the remaining resources to maintain availability and redundancy. This data movement increases resource usage on the cluster, and can have an impact on the performance. If the cluster isn't sized properly, it can experience degraded availability, which largely defeats the purpose of multi-AZ. The only way to configure a domain without standby on the AWS Management Console is to choose the **Standard create** creation method, and select **Domain without standby** as your deployment option. ### Shard distribution If you enable multi-AZ without Standby, you should create at least one replica for each index in your cluster. Without replicas, OpenSearch Service can't distribute copies of your data to other Availability Zones. Fortunately, the default configuration for any index is a replica count of 1. As the following diagram shows, OpenSearch Service makes a best effort to distribute primary shards and their corresponding replica shards to different zones. ![\[Diagram showing three Availability Zones, each with primary and replica shards distributed across them.\]](http://docs.aws.amazon.com/opensearch-service/latest/developerguide/images/za-3-az.png) In addition to distributing shards by Availability Zone, OpenSearch Service distributes them by node. Still, certain domain configurations can result in imbalanced shard counts. Consider the following domain: + 5 data nodes + 5 primary shards + 2 replicas + 3 Availability Zones In this situation, OpenSearch Service has to overload one node in order to distribute the primary and replica shards across the zones, as shown in the following diagram. ![\[Diagram showing distribution of primary and replica shards across three availability zones.\]](http://docs.aws.amazon.com/opensearch-service/latest/developerguide/images/za-3-az-imbal.png) To avoid these kinds of situations, which can strain individual nodes and hurt performance, we recommend that you choose multi-AZ with Standby, or choose an instance count that is a multiple of three when you plan to have two or more replicas per index. ### Dedicated master node distribution Even if you select two Availability Zones when configuring your domain, OpenSearch Service automatically distributes [dedicated master nodes](managedomains-dedicatedmasternodes.md) across three Availability Zones. This distribution helps prevent cluster downtime if a zone experiences a service disruption. If you use the recommended three dedicated master nodes and one Availability Zone goes down, your cluster still has a quorum (2) of dedicated master nodes and can elect a new master. The following diagram demonstrates this configuration. ![\[Diagram showing distribution of primary and replica nodes across three Availability Zones.\]](http://docs.aws.amazon.com/opensearch-service/latest/developerguide/images/za-2-az.png) If you choose an older-generation instance type that is not available in three Availability Zones, the following scenarios apply: + If you chose three Availability Zones for the domain, OpenSearch Service throws an error. Choose a different instance type, and try again. + If you chose two Availability Zones for the domain, OpenSearch Service distributes the dedicated master nodes across two zones. ## Availability zone disruptions Availability Zone disruptions are rare, but do occur. The following table lists different Multi-AZ configurations and behaviors during a disruption. The last row in the table applies to Multi-AZ with Standby, while all other rows have configurations that only apply to Multi-AZ without Standby. | Number of Availability Zones in a region | Number of Availability Zones you chose | Number of dedicated master nodes | Behavior if one Availability Zone experiences a disruption | | --- | --- | --- | --- | | 2 or more | 2 | 0 | Downtime. Your cluster loses half of its data nodes and must replace at least one in the remaining Availability Zone before it can elect a master. | | 2 | 2 | 3 | 50/50 chance of downtime. OpenSearch Service distributes two dedicated master nodes into one Availability Zone and one into the other: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/opensearch-service/latest/developerguide/managedomains-multiaz.html) | | 3 or more | 2 | 3 | No downtime. OpenSearch Service automatically distributes the dedicated master nodes across three Availability Zones, so the remaining two dedicated master nodes can elect a master. | | 3 or more | 3 | 0 | No downtime. Roughly two-thirds of your data nodes are still available to elect a master. | | 3 or more | 3 | 3 | No downtime. The remaining two dedicated master nodes can elect a master. | In *all* configurations, regardless of the cause, node failures can cause the cluster's remaining data nodes to experience a period of increased load while OpenSearch Service automatically configures new nodes to replace the now-missing ones. For example, in the event of an Availability Zone disruption in a three-zone configuration, two-thirds as many data nodes have to process just as many requests to the cluster. As they process these requests, the remaining nodes are also replicating shards onto new nodes as they come online, which can further impact performance. If availability is critical to your workload, consider adding resources to your cluster to alleviate this concern. **Note** OpenSearch Service manages Multi-AZ domains transparently, so you can't manually simulate Availability Zone disruptions. # Launching your Amazon OpenSearch Service domains within a VPC You can launch AWS resources, such as Amazon OpenSearch Service domains, into a *virtual private cloud* (VPC). A VPC is a virtual network that's dedicated to your AWS account. It's logically isolated from other virtual networks in the AWS Cloud. Placing an OpenSearch Service domain within a VPC enables secure communication between OpenSearch Service and other services within the VPC without the need for an internet gateway, NAT device, or VPN connection. All traffic remains securely within the AWS Cloud. **Note** If you place your OpenSearch Service domain within a VPC, your computer must be able to connect to the VPC. This connection often takes the form of a VPN, transit gateway, managed network, or proxy server. You can't directly access your domains from outside the VPC. ## VPC versus public domains The following are some of the ways VPC domains differ from public domains. Each difference is described later in more detail. + Because of their logical isolation, domains that reside within a VPC have an extra layer of security compared to domains that use public endpoints. + While public domains are accessible from any internet-connected device, VPC domains require some form of VPN or proxy. + Compared to public domains, VPC domains display less information in the console. Specifically, the **Cluster health** tab does not include shard information, and the **Indices** tab isn't present. + The domain endpoints take different forms (`https://search-domain-name` vs. `https://vpc-domain-name`). + You can't apply IP-based access policies to domains that reside within a VPC because security groups already enforce IP-based access policies. ## Limitations Operating an OpenSearch Service domain within a VPC has the following limitations: + If you launch a new domain within a VPC, you can't later switch it to use a public endpoint. The reverse is also true: If you create a domain with a public endpoint, you can't later place it within a VPC. Instead, you must create a new domain and migrate your data. + You can either launch your domain within a VPC or use a public endpoint, but you can't do both. You must choose one or the other when you create your domain. + You can't launch your domain within a VPC that uses dedicated tenancy. You must use a VPC with tenancy set to **Default**. + After you place a domain within a VPC, you can't move it to a different VPC, but you can change the subnets and security group settings. + To access the default installation of OpenSearch Dashboards for a domain that resides within a VPC, users must have access to the VPC. This process varies by network configuration, but likely involves connecting to a VPN or managed network or using a proxy server or transit gateway. To learn more, see [About access policies on VPC domains](#vpc-security), the [Amazon VPC User Guide](https://docs.aws.amazon.com/vpc/latest/userguide/), and [Controlling access to Dashboards](dashboards.md#dashboards-access). ## Architecture To support VPCs, OpenSearch Service places an endpoint into one, two, or three subnets of your VPC. If you enable [multiple Availability Zones](managedomains-multiaz.md) for your domain, each subnet must be in a different Availability Zone in the same region. If you only use one Availability Zone, OpenSearch Service places an endpoint into only one subnet. The following illustration shows the VPC architecture for one Availability Zone: ![\[VPC architecture showing subnet with security group connecting to OpenSearch Service data nodes.\]](http://docs.aws.amazon.com/opensearch-service/latest/developerguide/images/VPCNoZoneAwareness.png) The following illustration shows the VPC architecture for two Availability Zones: ![\[VPC architecture with two Availability Zones, showing security groups, data nodes, and master nodes.\]](http://docs.aws.amazon.com/opensearch-service/latest/developerguide/images/VPCZoneAwareness.png) OpenSearch Service also places an *elastic network interface* (ENI) in the VPC for each of your data nodes. OpenSearch Service assigns each ENI a private IP address from the IPv4 address range of your subnet. The service also assigns a public DNS hostname (which is the domain endpoint) for the IP addresses. You must use a public DNS service to resolve the endpoint (which is a DNS hostname) to the appropriate IP addresses for the data nodes: + If your VPC uses the Amazon-provided DNS server by setting the `enableDnsSupport` option to `true` (the default value), resolution for the OpenSearch Service endpoint will succeed. + If your VPC uses a private DNS server and the server can reach the public authoritative DNS servers to resolve DNS hostnames, resolution for the OpenSearch Service endpoint will also succeed. Because the IP addresses might change, you should resolve the domain endpoint periodically so that you can always access the correct data nodes. We recommend that you set the DNS resolution interval to one minute. If you’re using a client, you should also ensure that the DNS cache in the client is cleared. ### Migrating from public access to VPC access When you create a domain, you specify whether it should have a public endpoint or reside within a VPC. Once created, you cannot switch from one to the other. Instead, you must create a new domain and either manually reindex or migrate your data. Snapshots offer a convenient means of migrating data. For information about taking and restoring snapshots, see [Creating index snapshots in Amazon OpenSearch Service](managedomains-snapshots.md). ### About access policies on VPC domains Placing your OpenSearch Service domain within a VPC provides an inherent, strong layer of security. When you create a domain with public access, the endpoint takes the following form: ``` https://search-domain-name-identifier.region.es.amazonaws.com ``` As the "public" label suggests, this endpoint is accessible from any internet-connected device, though you can (and should) [control access to it](ac.md). If you access the endpoint in a web browser, you might receive a `Not Authorized` message, but the request reaches the domain. When you create a domain with VPC access, the endpoint *looks* similar to a public endpoint: ``` https://vpc-domain-name-identifier.region.es.amazonaws.com ``` If you try to access the endpoint in a web browser, however, you might find that the request times out. To perform even basic `GET` requests, your computer must be able to connect to the VPC. This connection often takes the form of a VPN, transit gateway, managed network, or proxy server. For details on the various forms it can take, see [Examples for VPC](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Scenarios.html) in the *Amazon VPC User Guide*. For a development-focused example, see [Testing VPC domains](#vpc-test). In addition to this connectivity requirement, VPCs let you manage access to the domain through [security groups](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_SecurityGroups.html). For many use cases, this combination of security features is sufficient, and you might feel comfortable applying an open access policy to the domain. Operating with an open access policy does *not* mean that anyone on the internet can access the OpenSearch Service domain. Rather, it means that if a request reaches the OpenSearch Service domain and the associated security groups permit it, the domain accepts the request. The only exception is if you're using fine-grained access control or an access policy that specifies IAM roles. In these situations, for the domain to accept a request, the security groups must permit it *and* it must be signed with valid credentials. **Note** Because security groups already enforce IP-based access policies, you can't apply IP-based access policies to OpenSearch Service domains that reside within a VPC. If you use public access, IP-based policies are still available. ### Before you begin: prerequisites for VPC access Before you can enable a connection between a VPC and your new OpenSearch Service domain, you must do the following: + **Create a VPC** To create your VPC, you can use the Amazon VPC console, the AWS CLI, or one of the AWS SDKs. For more information, see [Working with VPCs](https://docs.aws.amazon.com/vpc/latest/userguide/working-with-vpcs.html) in the *Amazon VPC User Guide*. If you already have a VPC, you can skip this step. + **Reserve IP addresses ** OpenSearch Service enables the connection of a VPC to a domain by placing network interfaces in a subnet of the VPC. Each network interface is associated with an IP address. You must reserve a sufficient number of IP addresses in the subnet for the network interfaces. For more information, see [Reserving IP addresses in a VPC subnet](#reserving-ip-vpc-endpoints). ### Testing VPC domains The enhanced security of a VPC can make connecting to your domain and running basic tests a challenge. If you already have an OpenSearch Service VPC domain and would rather not create a VPN server, try the following process: 1. For your domain's access policy, choose **Only use fine-grained access control**. You can always update this setting after you finish testing. 1. Create an Amazon Linux Amazon EC2 instance in the same VPC, subnet, and security group as your OpenSearch Service domain. Because this instance is for testing purposes and needs to do very little work, choose an inexpensive instance type like `t2.micro`. Assign the instance a public IP address and either create a new key pair or choose an existing one. If you create a new key, download it to your `~/.ssh` directory. To learn more about creating instances, see [Getting started with Amazon EC2 Linux instances](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EC2_GetStarted.html). 1. Add an [internet gateway](https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Internet_Gateway.html) to your VPC. 1. In the [route table](https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Route_Tables.html) for your VPC, add a new route. For **Destination**, specify a [CIDR block](https://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing#IPv4_CIDR_blocks) that contains your computer's public IP address. For **Target**, specify the internet gateway you just created. For example, you might specify `123.123.123.123/32` for just your computer or `123.123.123.0/24` for a range of computers. 1. For the security group, specify two inbound rules: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/opensearch-service/latest/developerguide/vpc.html) The first rule lets you SSH into your EC2 instance. The second allows the EC2 instance to communicate with the OpenSearch Service domain over HTTPS. 1. From the terminal, run the following command: ``` ssh -i ~/.ssh/your-key.pem ec2-user@your-ec2-instance-public-ip -N -L 9200:vpc-domain-name-identifier.region.es.amazonaws.com:443 ``` This command creates an SSH tunnel that forwards requests to [https://localhost:9200](https://localhost:9200) to your OpenSearch Service domain through the EC2 instance. Specifying port 9200 in the command simulates a local OpenSearch install, but use whichever port you'd like. OpenSearch Service only accepts connections over port 80 (HTTP) or 443 (HTTPS). The command provides no feedback and runs indefinitely. To stop it, press `Ctrl + C`. 1. Navigate to [https://localhost:9200/\$1dashboards/](https://localhost:9200/_plugin/kibana/) in your web browser. You might need to acknowledge a security exception. Alternately, you can send requests to [https://localhost:9200](https://localhost:9200) using [curl](https://curl.haxx.se/), [Postman](https://www.getpostman.com/), or your favorite programming language. **Tip** If you encounter curl errors due to a certificate mismatch, try the `--insecure` flag. ### Reserving IP addresses in a VPC subnet OpenSearch Service connects a domain to a VPC by placing network interfaces in a subnet of the VPC (or multiple subnets of the VPC if you enable [multiple Availability Zones](managedomains-multiaz.md)). Each network interface is associated with an IP address. Before you create your OpenSearch Service domain, you must have a sufficient number of IP addresses available in each subnet to accommodate the network interfaces. Here's the basic formula: The number of IP addresses that OpenSearch Service reserves in each subnet is three times the number of data nodes, divided by the number of Availability Zones. **Examples** + If a domain has nine data nodes across three Availability Zones, the IP count per subnet is 9 \$1 3 / 3 = 9. + If a domain has eight data nodes across two Availability Zones, the IP count per subnet is 8 \$1 3 / 2 = 12. + If a domain has six data nodes in one Availability Zone, the IP count per subnet is 6 \$1 3 / 1 = 18. When you create the domain, OpenSearch Service reserves the IP addresses, uses some for the domain, and reserves the rest for [blue/green deployments](managedomains-configuration-changes.md). You can see the network interfaces and their associated IP addresses in the **Network Interfaces** section of the Amazon EC2 console. The **Description** column shows which OpenSearch Service domain the network interface is associated with. **Tip** We recommend that you create dedicated subnets for the OpenSearch Service reserved IP addresses. By using dedicated subnets, you avoid overlap with other applications and services and ensure that you can reserve additional IP addresses if you need to scale your cluster in the future. To learn more, see [Creating a subnet in your VPC](https://docs.aws.amazon.com/vpc/latest/userguide/working-with-vpcs.html#AddaSubnet). You may also consider provisioning dedicated coordinator nodes to reduce the number of private IP address reservations required for your VPC domain. OpenSearch attaches an elastic network interface (ENI) to your dedicated coordinator nodes instead of your data nodes. Dedicated coordinator nodes usually represent around 10% of total data nodes. As a result, a smaller number of private IP addresses will be reserved for VPC domains. ### Service-linked role for VPC access A [service-linked role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_terms-and-concepts.html#iam-term-service-linked-role) is a unique type of IAM role that delegates permissions to a service so that it can create and manage resources on your behalf. OpenSearch Service requires a service-linked role to access your VPC, create the domain endpoint, and place network interfaces in a subnet of your VPC. OpenSearch Service automatically creates the role when you use the OpenSearch Service console to create a domain within a VPC. For this automatic creation to succeed, you must have permissions for the `iam:CreateServiceLinkedRole` action. To learn more, see [Service-linked role permissions](https://docs.aws.amazon.com/IAM/latest/UserGuide/using-service-linked-roles.html#service-linked-role-permissions) in the *IAM User Guide*. After OpenSearch Service creates the role, you can view it (`AWSServiceRoleForAmazonOpenSearchService`) using the IAM console. For full information on this role's permissions and how to delete it, see [Using service-linked roles for Amazon OpenSearch Service](slr.md). # Creating index snapshots in Amazon OpenSearch Service Snapshots in Amazon OpenSearch Service are backups of a cluster's indexes and state. *State* includes cluster settings, node information, index settings, and shard allocation. OpenSearch Service snapshots come in the following forms: + **Automated snapshots** are only for cluster recovery. You can use them to restore your domain in the event of red cluster status or data loss. For more information, see [Restoring snapshots](managedomains-snapshot-restore.md) below. OpenSearch Service stores automated snapshots in a preconfigured Amazon S3 bucket at no additional charge. + **Manual snapshots** are for cluster recovery *or* for moving data from one cluster to another. You have to initiate manual snapshots. These snapshots are stored in your own Amazon S3 bucket and standard S3 charges apply. If you have a snapshot from a self-managed OpenSearch cluster, you can use that snapshot to migrate to an OpenSearch Service domain. For more information, see [Migrating to Amazon OpenSearch Service](migration.md). All OpenSearch Service domains take automated snapshots, but the frequency differs in the following ways: + For domains running OpenSearch or Elasticsearch 5.3 and later, OpenSearch Service takes hourly automated snapshots and retains up to 336 of them for 14 days. Hourly snapshots are less disruptive because of their incremental nature. They also provide a more recent recovery point in case of domain problems. + For domains running Elasticsearch 5.1 and earlier, OpenSearch Service takes daily automated snapshots during the hour you specify, retains up to 14 of them, and doesn't retain any snapshot data for more than 30 days. If your cluster enters red status, all automated snapshots fail while the cluster status persists. If you don't correct the problem within two weeks, you can permanently lose the data in your cluster. For troubleshooting steps, see [Red cluster status](handling-errors.md#handling-errors-red-cluster-status). ## Prerequisites To create snapshots manually, you need to work with IAM and Amazon S3. Make sure you meet the following prerequisites before you attempt to take a snapshot: **** | Prerequisite | Description | | --- | --- | | S3 bucket | Create an S3 bucket to store manual snapshots for your OpenSearch Service domain. For instructions, see [Creating a general purpose bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html) in the *Amazon Simple Storage Service User Guide*. Remember the name of the bucket to use it in the following places:[\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/opensearch-service/latest/developerguide/managedomains-snapshots.html) Do not apply an Amazon Glacier lifecycle rule to this bucket. Manual snapshots don't support the Amazon Glacier storage class. | | IAM role | Create an IAM role to delegate permissions to OpenSearch Service. For instructions, see [Creating an IAM role (console)](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user.html#roles-creatingrole-user-console) in the *IAM User Guide*. The rest of this chapter refers to this role as `TheSnapshotRole`. **Attach an IAM policy** Attach the following policy to `TheSnapshotRole` to allow access to the S3 bucket: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/opensearch-service/latest/developerguide/managedomains-snapshots.html) For instructions to attach a policy to a role, see [Adding IAM identity permissions (console)](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_manage-attach-detach.html#add-policies-console) in the *IAM User Guide*. **Edit the trust relationship** Edit the trust relationship of `TheSnapshotRole` to specify OpenSearch Service in the `Principal` statement as shown in the following example: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/opensearch-service/latest/developerguide/managedomains-snapshots.html) For instructions to edit the trust relationship, see [Update a role trust policy](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_update-role-trust-policy.html) in the *IAM User Guide*. | | Permissions | In order to register the snapshot repository, you need to be able to pass `TheSnapshotRole` to OpenSearch Service. You also need access to the `es:ESHttpPut` action. To grant both of these permissions, attach the following policy to the IAM role whose credentials are being used to sign the request: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/opensearch-service/latest/developerguide/managedomains-snapshots.html) If your user or role doesn't have `iam:PassRole` permissions to pass `TheSnapshotRole`, you might encounter the following common error when you try to register a repository in the next step: 
$ python register-repo.py
{"Message":"User: arn:aws:iam::123456789012:user/MyUserAccount
is not authorized to perform: iam:PassRole on resource:
arn:aws:iam::123456789012:role/TheSnapshotRole"}
| # Registering a manual snapshot repository You need to register a snapshot repository with OpenSearch Service before you can take manual index snapshots. This one-time operation requires that you sign your AWS request with credentials that are allowed to access `TheSnapshotRole`, as described in [Prerequisites](managedomains-snapshots.md#managedomains-snapshot-prerequisites). ## Step 1: Map the snapshot role in OpenSearch Dashboards (if using fine-grained access control) Fine-grained access control introduces an additional step when registering a repository. Even if you use HTTP basic authentication for all other purposes, you need to map the `manage_snapshots` role to your IAM role that has `iam:PassRole` permissions to pass `TheSnapshotRole`. 1. Navigate to the OpenSearch Dashboards plugin for your OpenSearch Service domain. You can find the Dashboards endpoint on your domain dashboard on the OpenSearch Service console. 1. From the main menu choose **Security**, **Roles**, and select the **manage\$1snapshots** role. 1. Choose **Mapped users**, **Manage mapping**. 1. Add the ARN of the role that has permissions to pass `TheSnapshotRole`. Put role ARNs under **Backend roles**. ``` arn:aws:iam::123456789123:role/role-name ``` 1. Select **Map** and confirm the user or role shows up under **Mapped users**. ## Step 2: Register a repository The following **Snapshots** tab demonstrates how to register a snapshot directory. For options specific to encrypting a manual snapshot and registering a snapshot after migrating to a new domain, see the relevant tabs. ------ #### [ Snapshots ] To register a snapshot repository, send a PUT request to the OpenSearch Service domain endpoint. You can use [curl](https://curl.se/docs/manpage.html#--aws-sigv4), the [sample Python client](#managedomains-snapshot-client-python), [Postman](https://www.getpostman.com/), or some other method to send a signed request to register the snapshot repository. Note that you can't use a PUT request in the OpenSearch Dashboards console to register the repository. The request takes the following format: ``` PUT domain-endpoint/_snapshot/my-snapshot-repo-name { "type": "s3", "settings": { "bucket": "amzn-s3-demo-bucket", "base_path": "my/snapshot/directory", "region": "region", "role_arn": "arn:aws:iam::123456789012:role/TheSnapshotRole" } } ``` **Note** Repository names cannot start with "cs-". Additionally, you shouldn't write to the same repository from multiple domains. Only one domain should have write access to the repository. If your domain resides within a virtual private cloud (VPC), your computer must be connected to the VPC for the request to successfully register the snapshot repository. Accessing a VPC varies by network configuration, but likely involves connecting to a VPN or corporate network. To check that you can reach the OpenSearch Service domain, navigate to `https://your-vpc-domain.region.es.amazonaws.com` in a web browser and verify that you receive the default JSON response. When your Amazon S3 bucket is in another AWS Region than your OpenSearch domain, add the parameter `"endpoint": "s3.amazonaws.com"` to the request. ------ #### [ Encrypted snapshots ] You currently can't use AWS Key Management Service (KMS) keys to encrypt manual snapshots, but you can protect them using server-side encryption (SSE). To turn on SSE with S3-managed keys for the bucket you use as a snapshot repository, add `"server_side_encryption": true` to the `"settings"` block of the PUT request. For more information, see [Using server-side encryption with Amazon S3 managed keys (SSE-S3)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingServerSideEncryption.html) in the *Amazon Simple Storage Service User Guide*. Alternatively, you can use AWS KMS keys for server-side encryption on the S3 bucket that you use as a snapshot repository. If you use this approach, make sure to provide `TheSnapshotRole` permission to the AWS KMS key used to encrypt the S3 bucket. For more information, see [Key policies in AWS KMS](https://docs.aws.amazon.com/kms/latest/developerguide/key-policies.html). ------ #### [ Domain migration ] Registering a snapshot repository is a one-time operation. However, to migrate from one domain to another, you have to register the same snapshot repository on the old domain and the new domain. The repository name is arbitrary. Consider the following guidelines when migrating to a new domain or registering the same repository with multiple domains: + When registering the repository on the new domain, add `"readonly": true` to the `"settings"` block of the PUT request. This setting prevents you from accidentally overwriting data from the old domain. Only one domain should have write access to the repository. + If you're migrating data to a domain in a different AWS Region, (for example, from an old domain and bucket located in us-east-2 to a new domain in us-west-2), replace `"region": "region"` with `"endpoint": "s3.amazonaws.com"` in the PUT statement and retry the request. ------ ### Using the sample Python client The Python client is easier to automate than a simple HTTP request and has better reusability. If you choose to use this method to register a snapshot repository, save the following sample Python code as a Python file, such as `register-repo.py`. The client requires the [AWS SDK for Python (Boto3)](https://aws.amazon.com/sdk-for-python/), [requests](http://docs.python-requests.org/) and [requests-aws4auth](https://pypi.python.org/pypi/requests-aws4auth) packages. The client contains commented-out examples for other snapshot operations. Update the following variables in the sample code: `host`, `region`, `path`, and `payload`. ``` import boto3 import requests from requests_aws4auth import AWS4Auth host = '' # domain endpoint region = '' # e.g. us-west-1 service = 'es' credentials = boto3.Session().get_credentials() awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service, session_token=credentials.token) # Register repository path = '/_snapshot/my-snapshot-repo-name' # the OpenSearch API endpoint url = host + path payload = { "type": "s3", "settings": { "bucket": "amzn-s3-demo-bucket", "base_path": "my/snapshot/directory", "region": "us-west-1", "role_arn": "arn:aws:iam::123456789012:role/snapshot-role" } } headers = {"Content-Type": "application/json"} r = requests.put(url, auth=awsauth, json=payload, headers=headers) print(r.status_code) print(r.text) # # Take snapshot # # path = '/_snapshot/my-snapshot-repo-name/my-snapshot' # url = host + path # # r = requests.put(url, auth=awsauth) # # print(r.text) # # # Delete index # # path = 'my-index' # url = host + path # # r = requests.delete(url, auth=awsauth) # # print(r.text) # # # Restore snapshot (all indexes except Dashboards and fine-grained access control) # # path = '/_snapshot/my-snapshot-repo-name/my-snapshot/_restore' # url = host + path # # payload = { # "indices": "-.kibana*,-.opendistro_security,-.opendistro-*", # "include_global_state": False # } # # headers = {"Content-Type": "application/json"} # # r = requests.post(url, auth=awsauth, json=payload, headers=headers) # # print(r.text) # # # Restore snapshot (one index) # # path = '/_snapshot/my-snapshot-repo-name/my-snapshot/_restore' # url = host + path # # payload = {"indices": "my-index"} # # headers = {"Content-Type": "application/json"} # # r = requests.post(url, auth=awsauth, json=payload, headers=headers) # # print(r.text) ``` # Taking manual snapshots Snapshots are not instantaneous. They take time to complete and don't represent perfect point-in-time views of the cluster. While a snapshot is in progress, you can still index documents and make other requests to the cluster, but new documents and updates to existing documents generally aren't included in the snapshot. The snapshot includes primary shards as they existed when OpenSearch initiated the snapshot. Depending on the size of your snapshot thread pool, different shards might be included in the snapshot at slightly different times. For snapshot best practices, see [Improve snapshot performance](bp.md#bp-stability-snapshots). **Warning** Manual snapshots do not include data stored in UltraWarm or cold storage tiers. If your domain uses UltraWarm or cold storage, migrate those indexes to hot storage before taking a manual snapshot if you need to retain that data. ## Snapshot storage and performance OpenSearch snapshots are incremental, meaning they only store data that changed since the last successful snapshot. This incremental nature means the difference in disk usage between frequent and infrequent snapshots is often minimal. In other words, taking hourly snapshots for a week (for a total of 168 snapshots) might not use much more disk space than taking a single snapshot at the end of the week. Also, the more frequently you take snapshots, the less time they take to complete. For example, daily snapshots can take 20-30 minutes to complete, whereas hourly snapshots might complete within a few minutes. Some OpenSearch users take snapshots as often as every half hour. ## Take a snapshot You specify the following information when you create a snapshot: + The name of your snapshot repository + A name for the snapshot The examples in this chapter use [curl](https://curl.haxx.se/), a common HTTP client, for convenience and brevity. To pass a username and password to your curl request, see the [Getting started tutorial](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/gsg.html). If your access policies specify users or roles, you must sign your snapshot requests. For curl, you can use the [`--aws-sigv4` option](https://curl.se/docs/manpage.html#--aws-sigv4) with version 7.75.0 or later. You can also use the commented-out examples in the [sample Python client](managedomains-snapshot-registerdirectory.md#managedomains-snapshot-client-python) to make signed HTTP requests to the same endpoints that the curl commands use. To take a manual snapshot, perform the following steps: 1. You can't take a snapshot if one is currently in progress. To check, run the following command: ``` curl -XGET 'domain-endpoint/_snapshot/_status' ``` 1. Run the following command to take a manual snapshot: ``` curl -XPUT 'domain-endpoint/_snapshot/repository-name/snapshot-name' ``` To include or exclude certain indexes and specify other settings, add a request body. For the request structure, see [Take snapshots](https://opensearch.org/docs/1.1/opensearch/snapshot-restore/#take-snapshots) in the OpenSearch documentation. **Note** The time required to take a snapshot increases with the size of the OpenSearch Service domain. Long-running snapshot operations sometimes encounter the following error: `504 GATEWAY_TIMEOUT`. You can typically ignore these errors and wait for the operation to complete successfully. Run the following command to verify the state of all snapshots of your domain: ``` curl -XGET 'domain-endpoint/_snapshot/repository-name/_all?pretty' ``` # Restoring snapshots Before you restore a snapshot, make sure that the destination domain does not use [Multi-AZ with Standby](managedomains-multiaz.md#managedomains-za-standby). Having standby enabled causes the restore operation to fail. **Warning** If you use index aliases, you should either cease write requests to an alias or switch the alias to another index prior to deleting its index. Halting write requests helps avoid the following scenario: You delete an index, which also deletes its alias. An errant write request to the now-deleted alias creates a new index with the same name as the alias. You can no longer use the alias due to a naming conflict with the new index. If you switched the alias to another index, specify `"include_aliases": false` when you restore from a snapshot. To restore a snapshot 1. Identify the snapshot you want to restore. Ensure that all settings for this index, such as custom analyzer packages or allocation requirement settings, are compatible with the domain. To see all snapshot repositories, run the following command: ``` curl -XGET 'domain-endpoint/_snapshot?pretty' ``` After you identify the repository, run the following command to see all snapshots: ``` curl -XGET 'domain-endpoint/_snapshot/repository-name/_all?pretty' ``` **Note** Most automated snapshots are stored in the `cs-automated` repository. If your domain encrypts data at rest, they're stored in the `cs-automated-enc` repository. If you don't see the manual snapshot repository you're looking for, make sure you [registered it](managedomains-snapshot-registerdirectory.md) to the domain. 1. (Optional) Delete or rename one or more indexes in the OpenSearch Service domain if you have naming conflicts between indexes on the cluster and indexes in the snapshot. You can't restore a snapshot of your indexes to an OpenSearch cluster that already contains indexes with the same names. You have the following options if you have index naming conflicts: + Delete the indexes on the existing OpenSearch Service domain and then restore the snapshot. + Rename the indexes as you restore them from the snapshot and reindex them later. To learn how to rename indexes, see [this example request](https://opensearch.org/docs/latest/api-reference/snapshots/restore-snapshot/#example-request) in the OpenSearch documentation. + Restore the snapshot to a different OpenSearch Service domain (only possible with manual snapshots). The following command deletes all existing indexes in a domain: ``` curl -XDELETE 'domain-endpoint/_all' ``` However, if you don't plan to restore all indexes, you can just delete one: ``` curl -XDELETE 'domain-endpoint/index-name' ``` 1. To restore a snapshot, run the following command: ``` curl -XPOST 'domain-endpoint/_snapshot/repository-name/snapshot-name/_restore' ``` Due to special permissions on the OpenSearch Dashboards and fine-grained access control indexes, attempts to restore all indexes might fail, especially if you try to restore from an automated snapshot. The following example restores just one index, `my-index`, from `2020-snapshot` in the `cs-automated` snapshot repository: ``` curl -XPOST 'domain-endpoint/_snapshot/cs-automated/2020-snapshot/_restore' \ -d '{"indices": "my-index"}' \ -H 'Content-Type: application/json' ``` Alternately, you might want to restore all indexes *except* the Dashboards and fine-grained access control indexes: ``` curl -XPOST 'domain-endpoint/_snapshot/cs-automated/2020-snapshot/_restore' \ -d '{"indices": "-.kibana*,-.opendistro*"}' \ -H 'Content-Type: application/json' ``` **Note** Depending on your OpenSearch version, additional system indexes may also need to be excluded, such as `-.opensearch-observability*` and `-.plugins-ml-config*`. The restore operation may fail if these indexes already exist on the target domain. To exclude them, add them to the `indices` exclusion list. For example: `"indices": "-.kibana*,-.opendistro*,-.opensearch-observability*,-.plugins-ml-config*"`. You can restore a snapshot without deleting its data by using the `rename_pattern` and `rename_replacement` parameters. For more information on these parameters, see the Restore Snapshot API [request fields](https://opensearch.org/docs/latest/api-reference/snapshots/restore-snapshot/#request-fields) and [example request](https://opensearch.org/docs/latest/api-reference/snapshots/restore-snapshot/#example-request) in the OpenSearch documentation. **Note** If not all primary shards were available for the indexes involved, a snapshot might have a `state` of `PARTIAL`. This value indicates that data from at least one shard wasn't stored successfully. You can still restore from a partial snapshot, but you might need to use older snapshots to restore any missing indexes. ## Deleting manual snapshots To delete a manual snapshot, run the following command: ``` DELETE _snapshot/repository-name/snapshot-name ``` # Automating snapshots with Snapshot Management You can set up a Snapshot Management (SM) policy in OpenSearch Dashboards to automate periodic snapshot creation and deletion. SM can snapshot of a group of indices, whereas [Index State Management](managedomains-snapshots.md#managedomains-snapshot-ism) can only take one snapshot per index. To use SM in OpenSearch Service, you need to register your own Amazon S3 repository. For instructions to register your repository, see [Registering a manual snapshot repository](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/managedomains-snapshots.html#managedomains-snapshot-registerdirectory). **Note** Snapshot Management is supported starting with OpenSearch version 2.5. Domains running earlier versions do not support this feature. Prior to SM, OpenSearch Service offered a free, automated snapshot feature that's still turned on by default. This feature sends snapshots into the service-maintained `cs-*` repository. To deactivate the feature, reach out to Support. For more information on the SM feature, see [Snapshot management](https://opensearch.org/docs/latest/dashboards/sm-dashboards/) in the OpenSearch documentation. SM doesn't currently support snapshot creation on multiple index types. For example, if you try to create snapshot on multiple indices with `*` and some indices are in the [warm tier](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/ultrawarm.html#ultrawarm-manual-snapshot), the snapshot creation will fail. If you need your snapshot to contain multiple index types, use the [ISM snapshot action](https://opensearch.org/docs/latest/im-plugin/ism/policies/#snapshot) until SM supports this option. # Configure permissions If you're upgrading to 2.5 from a previous OpenSearch Service domain version, the snapshot management security permissions might not be defined on the domain. Non-admin users must be mapped to this role in order to use snapshot management on domains using fine-grained access control. To manually create the snapshot management role, perform the following steps: 1. In OpenSearch Dashboards, go to **Security** and choose **Permissions**. 1. Choose **Create action group** and configure the following groups: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/opensearch-service/latest/developerguide/sm-security.html) 1. Choose **Roles** and **Create role**. 1. Name the role **snapshot\$1management\$1role**. 1. For **Cluster permissions**, select `snapshot_management_full_access` or `snapshot_management_read_access`. 1. Choose **Create**. 1. After you create the role, [map it](fgac.md#fgac-mapping) to any user or backend role that will manage snapshots. ## Considerations Consider the following when you configure snapshot management: + One policy is allowed per repository. + Up to 400 snapshots are allowed for one policy. + This feature won't run if your domain has a red status, is under high JVM pressure (85% or above), or has a stuck snapshot function. When the overall indexing and searching performance of your cluster is impacted, SM may also be impacted. + A snapshot operation only starts after the previous operation finishes, so that no concurrent snapshot operations are activated by one policy. + Multiple policies with the same schedule can cause a resource spike. If the policies' snapshotted indices overlap, the shard-level snapshot operations can only run sequentially, which can cause a cascaded performance problem. If the policies share a repository, there will be spike of write operations to that repository. + We recommend that you schedule your snapshot operations automation to no more than once per hour, unless you have a special use case. ## Automating snapshots with Index State Management You can use the Index State Management (ISM) [snapshot](https://opendistro.github.io/for-elasticsearch-docs/docs/im/ism/policies/#snapshot) operation to automatically trigger snapshots of indexes based on changes in their age, size, or number of documents. ISM is best when you need one snapshot per index. If you need to snapshot of a group of indices, see [Automating snapshots with Snapshot Management](managedomains-snapshot-mgmt.md). To use SM in OpenSearch Service, you need to register your own Amazon S3 repository. For an example ISM policy using the `snapshot` operation, see [Sample Policies](ism.md#ism-example). ## Using Curator for snapshots If ISM doesn't work for index and snapshot management, you can use Curator instead. It offers advanced filtering functionality that can help simplify management tasks on complex clusters. Use [pip](https://pip.pypa.io/en/stable/installing/) to install Curator: ``` pip install elasticsearch-curator ``` You can use Curator as a command line interface (CLI) or Python API. If you use the Python API, you must use version 7.13.4 or earlier of the legacy [elasticsearch-py](https://elasticsearch-py.readthedocs.io/) client. It doesn't support the opensearch-py client. If you use the CLI, export your credentials at the command line and configure `curator.yml` as follows: ``` client: hosts: search-my-domain.us-west-1.es.amazonaws.com port: 443 use_ssl: True aws_region: us-west-1 aws_sign_request: True ssl_no_validate: False timeout: 60 logging: loglevel: INFO ``` # Upgrading Amazon OpenSearch Service domains **Note** OpenSearch and Elasticsearch version upgrades differ from service software updates. For information on updating the service software for your OpenSearch Service domain, see [Service software updates in Amazon OpenSearch Service](service-software.md). Amazon OpenSearch Service offers in-place upgrades for domains that run OpenSearch 1.0 or later, or Elasticsearch 5.1 or later. If you use services like Amazon Data Firehose or Amazon CloudWatch Logs to stream data to OpenSearch Service, check that these services support the newer version of OpenSearch before migrating. ## Supported upgrade paths Currently, OpenSearch Service supports the following upgrade paths: | From version | To version | | --- | --- | | OpenSearch 1.3, 2.x, or 3.x | OpenSearch 3.*x* If you're running OpenSearch 1.3 or 2.x, you must first upgrade to OpenSearch 2.19 *before* upgrading to OpenSearch 3.x. When upgrading from OpenSearch 2.19 to OpenSearch 3.x, you may need to address the following incompatible index settings. These were deprecated earlier in 2.x and will lead to an upgrade check validation failure if not removed. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/opensearch-service/latest/developerguide/version-migration.html) For more information about deprecated index settings and steps to remediate them, see [Index settings](https://docs.opensearch.org/2.6/search-plugins/knn/knn-index/#index-settings). For a complete list of breaking changes for OpenSearch 3.x, see [3.0.0](https://docs.opensearch.org/latest/breaking-changes/#300). If your domain contains indexes created in OpenSearch 1.3, Elasticsearch 7.10, or earlier versions, you must reindex them before upgrading to OpenSearch 3.x. OpenSearch 3.x doesn't support these older indexes, regardless of whether they're in hot, UltraWarm, or cold storage. To reindex incompatible UltraWarm or cold indexes: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/opensearch-service/latest/developerguide/version-migration.html) Alternatively, delete the indexes if you no longer need them. | | OpenSearch 1.3 or 2.x | OpenSearch 2.*x* OpenSearch 2.17 will enable concurrent segment search by default with auto mode if the domain satisfies the following conditions: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/opensearch-service/latest/developerguide/version-migration.html) Version 2.3 has the following breaking changes: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/opensearch-service/latest/developerguide/version-migration.html) | | OpenSearch 1.x | OpenSearch 1.x | | Elasticsearch 7.x | Elasticsearch 7.*x* or OpenSearch 1.*x* OpenSearch 1.*x* introduces numerous breaking changes. For details, see [Amazon OpenSearch Service rename - Summary of changes](rename.md). | | Elasticsearch 6.8 | Elasticsearch 7.*x* or OpenSearch 1.*x* Elasticsearch 7.0 and OpenSearch 1.0 include numerous breaking changes. Before initiating an in-place upgrade, we recommend [taking a manual snapshot](managedomains-snapshots.md) of the 6.*x* domain, restoring it on a test 7.*x* or OpenSearch 1.*x* domain, and using that test domain to identify potential upgrade issues. For breaking changes in OpenSearch 1.0, see [Amazon OpenSearch Service rename - Summary of changes](rename.md). Like Elasticsearch 6.*x*, indexes can only contain one mapping type, but that type must now be named `_doc`. As a result, certain APIs no longer require a mapping type in the request body (such as the `_bulk` API). For new indexes, self-hosted Elasticsearch 7.*x* and OpenSearch 1.*x* have a default shard count of one. OpenSearch Service domains on Elasticsearch 7.*x* and later retain the previous default of five. | | Elasticsearch 6.*x* | Elasticsearch 6.*x* | | Elasticsearch 5.6 | Elasticsearch 6.*x* Indexes created in version 6.*x* no longer support multiple mapping types. Indexes created in version 5.*x* still support multiple mapping types when restored into a 6.*x* cluster. Check that your client code creates only a single mapping type per index. To minimize downtime during the upgrade from Elasticsearch 5.6 to 6.*x*, OpenSearch Service reindexes the `.kibana` index to `.kibana-6`, deletes `.kibana`, creates an alias named `.kibana`, and maps the new index to the new alias. | | Elasticsearch 5.x | Elasticsearch 5.x | The upgrade process consists of three steps: 1. **Pre-upgrade checks** – OpenSearch Service checks for issues that can block an upgrade and doesn't proceed to the next step unless these checks succeed. 1. **Snapshot** – OpenSearch Service takes a snapshot of the OpenSearch or Elasticsearch cluster and doesn't proceed to the next step unless the snapshot succeeds. If the upgrade fails, OpenSearch Service uses this snapshot to restore the cluster to its original state. For more information see [Can't downgrade after upgrade](handling-errors.md#troubleshooting-upgrade-snapshot). 1. **Upgrade** – OpenSearch Service starts the upgrade, which can take from 15 minutes to several hours to complete. OpenSearch Dashboards might be unavailable during some or all of the upgrade. # Upgrading a domain (console) The upgrade process is irreversible and can't be paused or cancelled. During an upgrade, you can't make configuration changes to the domain. Before starting an upgrade, double-check that you want to proceed. You can use these same steps to perform the pre-upgrade check without actually starting an upgrade. If the cluster has dedicated master nodes, OpenSearch upgrades complete without downtime. Otherwise, the cluster might be unresponsive for several seconds post-upgrade while it elects a master node. **To upgrade a domain to a later version of OpenSearch or Elasticsearch** 1. [Take a manual snapshot](managedomains-snapshots.md) of your domain. This snapshot serves as a backup that you can [restore on a new domain](managedomains-snapshot-restore.md) if you want to return to using the prior OpenSearch version. 1. Go to [https://aws.amazon.com](https://console.aws.amazon.com/) and choose **Sign In to the Console**. 1. Under **Analytics**, choose **Amazon OpenSearch Service**. 1. In the navigation pane, under **Domains**, choose the domain that you want to upgrade. 1. Choose **Actions** and **Upgrade**. 1. Select the version to upgrade to. If you're upgrading to an OpenSearch version, the **Enable compatibility mode** option appears. If you enable this setting, OpenSearch reports its version as 7.10 to allow Elasticsearch OSS clients and plugins like Logstash to continue working with Amazon OpenSearch Service. You can disable this setting later 1. Choose **Upgrade**. 1. Check the **Status** on the domain dashboard to monitor the status of the upgrade. # Upgrading a domain (CLI) You can use the following operations to identify the correct version of OpenSearch or Elasticsearch for your domain, start an in-place upgrade, perform the pre-upgrade check, and view progress: + `get-compatible-versions` (`GetCompatibleVersions`) + `upgrade-domain` (`UpgradeDomain`) + `get-upgrade-status` (`GetUpgradeStatus`) + `get-upgrade-history` (`GetUpgradeHistory`) For more information, see the [AWS CLI command reference](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/opensearch/index.html) and [Amazon OpenSearch Service API Reference](https://docs.aws.amazon.com/opensearch-service/latest/APIReference/Welcome.html). # Upgrading a domain (SDK) This sample uses the [OpenSearchService](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/opensearch.html) low-level Python client from the AWS SDK for Python (Boto) to check if a domain is eligible for upgrade to a specific version, upgrades it, and continuously checks the upgrade status. ``` import boto3 from botocore.config import Config import time # Build the client using the default credential configuration. # You can use the CLI and run 'aws configure' to set access key, secret # key, and default Region. DOMAIN_NAME = '' # The name of the domain to upgrade TARGET_VERSION = '' # The version you want to upgrade the domain to. For example, OpenSearch_1.1 my_config = Config( # Optionally lets you specify a Region other than your default. region_name='us-east-1' ) client = boto3.client('opensearch', config=my_config) def check_versions(): """Determine whether domain is eligible for upgrade""" response = client.get_compatible_versions( DomainName=DOMAIN_NAME ) compatible_versions = response['CompatibleVersions'] for i in range(len(compatible_versions)): if TARGET_VERSION in compatible_versions[i]["TargetVersions"]: print('Domain is eligible for upgrade to ' + TARGET_VERSION) upgrade_domain() print(response) else: print('Domain not eligible for upgrade to ' + TARGET_VERSION) def upgrade_domain(): """Upgrades the domain""" response = client.upgrade_domain( DomainName=DOMAIN_NAME, TargetVersion=TARGET_VERSION ) print('Upgrading domain to ' + TARGET_VERSION + '...' + response) time.sleep(5) wait_for_upgrade() def wait_for_upgrade(): """Get the status of the upgrade""" response = client.get_upgrade_status( DomainName=DOMAIN_NAME ) if (response['UpgradeStep']) == 'UPGRADE' and (response['StepStatus']) == 'SUCCEEDED': print('Domain successfully upgraded to ' + TARGET_VERSION) elif (response['StepStatus']) == 'FAILED': print('Upgrade failed. Please try again.') elif (response['StepStatus']) == 'SUCCEEDED_WITH_ISSUES': print('Upgrade succeeded with issues') elif (response['StepStatus']) == 'IN_PROGRESS': time.sleep(30) wait_for_upgrade() def main(): check_versions() if __name__ == "__main__": main() ``` ## Troubleshooting validation failures When you initiate an OpenSearch or Elasticsearch version upgrade, OpenSearch Service first performs a series of validation checks to ensure that your domain is eligible for an upgrade. If any of these checks fail, you receive a notification containing the specific issues that you must fix before upgrading your domain. For a list of potential issues and steps to resolve them, see [Troubleshooting validation errors](managedomains-configuration-changes.md#validation). ## Troubleshooting an upgrade In-place upgrades require healthy domains. Your domain might be ineligible for an upgrade or fail to upgrade for a wide variety of reasons. The following table shows the most common issues. | Issue | Description | | --- | --- | | Optional plugin not supported | When you upgrade a domain with optional plugins, OpenSearch Service automatically upgrades the plugins as well. Therefore, the target version for your domain must also support these optional plugins. If the domain has an optional plugin installed that is not available for the target version, the upgrade request fails. | | Too many shards per node | OpenSearch, as well as 7.x versions of Elasticsearch, have a default setting of no more than 1,000 shards per node. If a node in your current cluster exceeds this setting, OpenSearch Service doesn't allow you to upgrade. See [Exceeded maximum shard limit](handling-errors.md#troubleshooting-shard-limit) for troubleshooting options. | | Domain in processing | The domain is in the middle of a configuration change. Check upgrade eligibility after the operation completes. | | Red cluster status | One or more indexes in the cluster is red. For troubleshooting steps, see [Red cluster status](handling-errors.md#handling-errors-red-cluster-status). | | High error rate | The cluster is returning a large number of 5xx errors when attempting to process requests. This problem is usually the result of too many simultaneous read or write requests. Consider reducing traffic to the cluster or scaling your domain. | | Split brain | Split brain means that your cluster has more than one master node and has split into two clusters that never will rejoin on their own. You can avoid split brain by using the recommended number of [dedicated master nodes](managedomains-dedicatedmasternodes.md). For help recovering from split brain, contact [Support](https://console.aws.amazon.com/support/home). | | Master node not found | OpenSearch Service can't find the cluster's master node. If your domain uses [multi-AZ](managedomains-multiaz.md), an Availability Zone failure might have caused the cluster to lose quorum and be unable to elect a new [master node](managedomains-dedicatedmasternodes.md). If the issue does not self-resolve, contact [Support](https://console.aws.amazon.com/support/home). | | Too many pending tasks | The master node is under heavy load and has many pending tasks. Consider reducing traffic to the cluster or scaling your domain. | | Impaired storage volume | The disk volume of one or more nodes isn't functioning properly. This issue often occurs alongside other issues, like a high error rate or too many pending tasks. If it occurs in isolation and doesn't self-resolve, contact [Support](https://console.aws.amazon.com/support/home). | | KMS key issue | The KMS key that is used to encrypt the domain is either inaccessible or missing. For more information, see [Monitoring domains that encrypt data at rest](encryption-at-rest.md#monitoring-ear). | | Snapshot in progress | The domain is currently taking a snapshot. Check upgrade eligibility after the snapshot finishes. Also check that you can list manual snapshot repositories, list snapshots within those repositories, and take manual snapshots. If OpenSearch Service is unable to check whether a snapshot is in progress, upgrades can fail. | | Snapshot timeout or failure | The pre-upgrade snapshot took too long to complete or failed. Check cluster health, and try again. If the problem persists, contact [Support](https://console.aws.amazon.com/support/home). | | Incompatible indexes | One or more indexes is incompatible with the target version. This problem can occur if you migrated the indexes from an older version of OpenSearch or Elasticsearch. Reindex the indexes and try again. | | High disk usage | Disk usage for the cluster is above 90%. Delete data or scale the domain, and try again. | | High JVM usage | JVM memory pressure is above 75%. Reduce traffic to the cluster or scale the domain, and try again. | | OpenSearch Dashboards alias problem | .dashboards is already configured as an alias and maps to an incompatible index, likely one from an earlier version of OpenSearch Dashboards. Reindex and try again. | | Red Dashboards status | OpenSearch Dashboards status is red. Try using Dashboards when the upgrade completes. If the red status persists, resolve it manually, and try again. | | Cross-cluster compatibility | You can only upgrade if cross-cluster compatibility is maintained between the source and destination domains after the upgrade. During the upgrade process, any incompatible connections are identified. To proceed, either upgrade the remote domain or delete the incompatible connections. Note that if replication is active on the domain, you can't resume it once you delete the connection. | | Other OpenSearch Service service issue | Issues with OpenSearch Service itself might cause your domain to display as ineligible for an upgrade. If none of the preceding conditions apply to your domain and the problem persists for more than a day, contact [Support](https://console.aws.amazon.com/support/home). | # Using a snapshot to migrate data In-place upgrades are the easier, faster, and more reliable way to upgrade a domain to a later OpenSearch or Elasticsearch version. Snapshots are a good option if you need to migrate from a pre-5.1 version of Elasticsearch or want to migrate to an entirely new cluster. The following table shows how to use snapshots to migrate data to a domain that uses a different OpenSearch or Elasticsearch version. For more information about taking and restoring snapshots, see [Creating index snapshots in Amazon OpenSearch Service](managedomains-snapshots.md). | From version | To version | Migration process | | --- | --- | --- | | OpenSearch 1.3 or 2.x | OpenSearch 2.x | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/opensearch-service/latest/developerguide/snapshot-based-migration.html) | | OpenSearch 1.x | OpenSearch 1.x | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/opensearch-service/latest/developerguide/snapshot-based-migration.html) | | Elasticsearch 6.x or 7.x | OpenSearch 1.x | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/opensearch-service/latest/developerguide/snapshot-based-migration.html) | | Elasticsearch 6.x | Elasticsearch 7.x | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/opensearch-service/latest/developerguide/snapshot-based-migration.html) | | Elasticsearch 6.x | Elasticsearch 6.8 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/opensearch-service/latest/developerguide/snapshot-based-migration.html) | | Elasticsearch 5.x | Elasticsearch 6.x | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/opensearch-service/latest/developerguide/snapshot-based-migration.html) | | Elasticsearch 5.x | Elasticsearch 5.6 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/opensearch-service/latest/developerguide/snapshot-based-migration.html) | | Elasticsearch 2.3 | Elasticsearch 6.x | Elasticsearch 2.3 snapshots are not compatible with 6.*x*. To migrate your data directly from 2.3 to 6.*x*, you must manually recreate your indexes in the new domain. Alternately, you can follow the 2.3 to 5.*x* steps in this table, perform `_reindex` operations in the new 5.*x* domain to convert your 2.3 indexes to 5.*x* indexes, and then follow the 5.*x* to 6.*x* steps. | | Elasticsearch 2.3 | Elasticsearch 5.x | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/opensearch-service/latest/developerguide/snapshot-based-migration.html) | | Elasticsearch 1.5 | Elasticsearch 5.x | Elasticsearch 1.5 snapshots are not compatible with 5.*x*. To migrate your data from 1.5 to 5.*x*, you must manually recreate your indexes in the new domain. 1.5 snapshots *are* compatible with 2.3, but OpenSearch Service 2.3 domains do not support the `_reindex` operation. Because you cannot reindex them, indexes that originated in a 1.5 domain still fail to restore from 2.3 snapshots to 5.*x* domains. | | Elasticsearch 1.5 | Elasticsearch 2.3 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/opensearch-service/latest/developerguide/snapshot-based-migration.html) | # Creating a custom endpoint for Amazon OpenSearch Service Creating a custom endpoint for your Amazon OpenSearch Service domain makes it easier for you to refer to your OpenSearch and OpenSearch Dashboards URLs. You can include your company's branding or just use a shorter, easier-to-remember endpoint than the standard one. If you ever need to switch to a new domain, just update your DNS to point to the new URL and continue using the same endpoint as before. You secure custom endpoints by either generating a certificate in AWS Certificate Manager (ACM) or importing one of your own. ## Custom endpoints for new domains You can enable a custom endpoint for a new OpenSearch Service domain using the OpenSearch Service console, AWS CLI, or configuration API. **To customize your endpoint (console)** 1. From the OpenSearch Service console, choose **Create domain** and provide a name for the domain. 1. Under **Custom endpoint**, select **Enable custom endpoint**. 1. For **Custom hostname**, enter your preferred custom endpoint hostname. The hostname should be a fully qualified domain name (FQDN), such as www.yourdomain.com or example.yourdomain.com. **Note** If you don't have a [wildcard certificate](https://en.wikipedia.org/wiki/Wildcard_certificate) you must obtain a new certificate for your custom endpoint's subdomains. 1. For **AWS certificate**, choose the SSL certificate to use for your domain. If no certificates are available, you can import one into ACM or use ACM to provision one. For more information, see [Issuing and Managing Certificates](https://docs.aws.amazon.com/acm/latest/userguide/gs.html) in the *AWS Certificate Manager User Guide*. **Note** The certificate must have the custom endpoint name and be in the same account as your OpenSearch Service domain. The certificate status should be ISSUED. + Follow the rest of the steps to create your domain and choose **Create**. + Select the domain when it's finished processing to view your custom endpoint. To use the CLI or configuration API, use the `CreateDomain` and `UpdateDomainConfig` operations. For more information, see the [AWS CLI Command Reference](https://docs.aws.amazon.com/cli/latest/reference/) and [Amazon OpenSearch Service API Reference](https://docs.aws.amazon.com/opensearch-service/latest/APIReference/Welcome.html). ## Custom endpoints for existing domains To add a custom endpoint to an existing OpenSearch Service domain, choose **Edit** and perform steps 2–4 above. ## CNAME mapping After you enable a custom endpoint for your OpenSearch Service domain, you can create a CNAME mapping in Amazon Route 53 (or your preferred DNS service provider). Creating a CNAME mapping will enable you to route traffic to your custom endpoint and its subdomains. Without this mapping, you won't be able to route traffic to your custom endpoint. For steps to create this mapping in Route 53, see [Configuring DNS routing for a new domain ](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/dns-configuring-new-domain.html) and [Creating a new hosted zone for a subdomain](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/dns-routing-traffic-for-subdomains.html#dns-routing-traffic-for-subdomains-creating-hosted-zone). For other providers, consult their documentation. Create a CNAME record that points the custom endpoint to the automatically generated domain endpoint. If your domain is dual stack, you can point your CNAME record to either of the two service generated endpoints. The dual stack capabilty of your custom endpoint depends on the service generated endpoint that you point the CNAME record to. The custom endpoint hostname is the *name* of the CNAME record, and the domain endpoint hostname is the *value* of the CNAME record. If you use [SAML authentication for OpenSearch Dashboards](saml.md), you must update your IdP with the new SSO URL. You can use Amazon Route 53 to create an alias record type to point your domain's custom endpoint to a dual stack search endpoint. To create an alias record type, you must configure your domain to use the dual stack IP address type. You can do this using the Route 53 API. To create an alias record type using the Route 53 API, specify the alias target of your domain. You can find the alias target of your domain in the **Hosted Zone (dual stack)** field in the custom endpoint section of the OpenSearch Service console or by using the `DescribeDomain` API and copying the value of the `DomainEndpointV2HostedZoneId`. # Auto-Tune for Amazon OpenSearch Service Auto-Tune in Amazon OpenSearch Service uses performance and usage metrics from your OpenSearch cluster to suggest memory-related configuration changes, including queue and cache sizes and Java virtual machine (JVM) settings on your nodes. These optional changes improve cluster speed and stability. Some changes deploy immediately, while others are scheduled during your domain's off-peak window. You can revert to the default OpenSearch Service settings at any time. As Auto-Tune gathers and analyzes performance metrics for your domain, you can view its recommendations in the OpenSearch Service console on the **Notifications** page. Auto-Tune is available in commercial AWS Regions on domains running any OpenSearch version, or Elasticsearch 6.7 or later, with a [supported instance type](supported-instance-types.md). ## Types of changes Auto-Tune has two broad categories of changes: + Nondisruptive changes that it applies as the cluster runs. + Changes that require a [blue/green deployment](managedomains-configuration-changes.md), which it applies during the domain's off-peak window. Based on your domain's performance metrics, Auto-Tune can suggest adjustments to the following settings: | Change type | Category | Description | | --- | --- | --- | | JVM heap size | Blue/green | By default, OpenSearch Service uses 50% of an instance's RAM for the JVM heap, up to a heap size of 32 GiB. Increasing this percentage gives OpenSearch more memory, but leaves less for the operating system and other processes. Larger values can decrease the number of garbage collection pauses, but increase the length of those pauses. | | JVM young generation settings | Blue/green | JVM "young generation" settings affect the frequency of minor garbage collections. More frequent minor collections can decrease the number of major collections and pauses. | | Queue size | Nondisruptive | By default, the search queue size is `1000` and the write queue size is `10000`. Auto-Tune automatically scales the search and write queues if additional heap is available to handle requests. | | Cache size | Nondisruptive | The *field cache* monitors on-heap data structures, so it's important to monitor the cache's use. Auto-Tune scales the field data cache size to avoid out of memory and circuit breaker issues. The *shard request cache* is managed at the node level and has a default maximum size of 1% of the heap. Auto-Tune scales the shard request cache size to accept more search and index requests than what the configured cluster can handle. | | Request size | Nondisruptive | By default, when the aggregated size of in-flight requests surpasses 10% of total JVM (2% for `t2` instance types and 1% for `t3.small`), OpenSearch throttles all new `_search` and `_bulk` requests until the existing requests complete. Auto-Tune automatically tunes this threshold, typically between 5-15%, based on the amount of JVM that is currently occupied on the system. For example, if JVM memory pressure is high, Auto-Tune might reduce the threshold to 5%, at which point you might see more rejections until the cluster stabilizes and the threshold increases. | # Enabling or disabling Auto-Tune OpenSearch Service enables Auto-Tune by default on new domains. To enable or disable Auto-Tune on existing domains, we recommend using the console, which simplifies the process. Enabling Auto-Tune doesn't cause a blue/green deployment. You currently can't enable or disable Auto-Tune using AWS CloudFormation. ## Console **To enable Auto-Tune on an existing domain** 1. Open the Amazon OpenSearch Service console at [https://console.aws.amazon.com/aos/home](https://console.aws.amazon.com/aos/home ). 1. In the navigation pane, under **Domains**, choose the domain name to open the cluster configuration. 1. Choose **Turn on** if Auto-Tune isn't already enabled. 1. Optionally, select **Off-peak window** to schedule optimizations that require a blue/green deployment during the domain's configured off-peak window. For more information, see [Scheduling Auto-Tune enhancements](auto-tune-schedule.md). 1. Choose **Save changes**. ## CLI To enable Auto-Tune using the AWS CLI, send an [UpdateDomainConfig](https://docs.aws.amazon.com/opensearch-service/latest/APIReference/API_UpdateDomainConfig.html) request: ``` aws opensearch update-domain-config \ --domain-name my-domain \ --auto-tune-options DesiredState=ENABLED ``` # Scheduling Auto-Tune enhancements Prior to February 16, 2023, Auto-Tune used *maintenance windows* to schedule changes that required a blue/green deployment. Maintenance windows are now deprecated in favor of the [off-peak window](off-peak.md), which is a daily 10-hour time block during which your domain typically experiences low traffic. You can modify the default start time for the off-peak window, but you can't modify the length. Any domains that had Auto-Tune maintenance windows enabled before the introduction of off-peak windows on February 16, 2023 can continue to use legacy maintenance windows with no interruption. However, we recommend that you migrate your existing domains to use the off-peak window for domain maintenance instead. For instructions, see [Migrating from Auto-Tune maintenance windows](off-peak.md#off-peak-migrate). ## Console **To schedule Auto-Tune actions the off-peak window** 1. Open the Amazon OpenSearch Service console at [https://console.aws.amazon.com/aos/home](https://console.aws.amazon.com/aos/home ). 1. In the navigation pane, under **Domains**, choose the domain name to open the cluster configuration. 1. Go to the **Auto-Tune** tab and choose **Edit**. 1. Choose **Turn on** if Auto-Tune isn't already enabled. 1. Under **Schedule optimizations during off-peak window**, select **Off-peak window**. 1. Choose **Save changes**. ## CLI To configure your domain to schedule Auto-Tune actions during the configured off-peak window, include `UseOffPeakWindow` in the [UpdateDomainConfig](https://docs.aws.amazon.com/opensearch-service/latest/APIReference/API_UpdateDomainConfig.html) request: ``` aws opensearch update-domain-config \ --domain-name my-domain \ --auto-tune-options DesiredState=ENABLED,UseOffPeakWindow=true,MaintenanceSchedules=null ``` ## Monitoring Auto-Tune changes You can monitor Auto-Tune statistics in Amazon CloudWatch. For a full list of metrics, see [Auto-Tune metrics](managedomains-cloudwatchmetrics.md#managedomains-cloudwatchmetrics-autotune-metrics). OpenSearch Service sends Auto-Tune events to Amazon EventBridge. You can use EventBridge to configure rules that send an email or perform a specific action when an event is received. To see the format of each Auto-Tune event sent to EventBridge, see [Auto-Tune events](monitoring-events.md#monitoring-events-autotune). # Tagging Amazon OpenSearch Service domains Tags let you assign arbitrary information to an Amazon OpenSearch Service domain so you can categorize and filter on that information. A tag is a key-value pair that you define and associate with an OpenSearch Service domain. You can use these tags to track costs by grouping expenses for similarly tagged resources. AWS doesn't apply any semantic meaning to your tags. Tags are interpreted strictly as character strings. All tags have the following elements: | Tag Element | Description | Required | | --- | --- | --- | | Tag key | The tag key is the name of the tag. Key must be unique to the OpenSearch Service domain to which they're attached. For a list of basic restrictions on tag keys and values, see [User-Defined Tag Restrictions](https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/allocation-tag-restrictions.html). | Yes | | Tag value | The tag value is the string value of the tag. Tag values can be `null` and don't have to be unique in a tag set. For example, you can have a key-value pair in a tag set of project/Trinity and cost-center/Trinity. For a list of basic restrictions on tag keys and values, see [User-Defined Tag Restrictions](https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/allocation-tag-restrictions.html). | No | Each OpenSearch Service domain has a tag set, which contains all the tags assigned to that OpenSearch Service domain. AWS doesn't automatically assign any tags to OpenSearch Service domains. A tag set can contain between 0 and 50 tags. If you add a tag to a domain with the same key as an existing tag, the new value overwrites the old value. ## Tagging examples You can use a key to define a category, and the value could be an item in that category. For example, you could define a tag key of `project` and a tag value of `Salix`, indicating that the OpenSearch Service domain is assigned to the Salix project. You could also use tags to designate OpenSearch Service domains as being used for test or production by using a key such as `environment=test` or `environment=production`. Try to use a consistent set of tag keys to make it easier to track metadata that is associated with OpenSearch Service domains. You also can use tags to organize your AWS bill to reflect your own cost structure. To do this, sign up to get your AWS account bill with tag key values included. Then, organize your billing information according to resources with the same tag key values to see the cost of combined resources. For example, you can tag several OpenSearch Service domains with key-value pairs, and then organize your billing information to see the total cost for each domain across several services. For more information, see [Using Cost Allocation Tags](https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/cost-alloc-tags.html) in the *AWS Billing and Cost Management* documentation. **Note** Tags are cached for authorization purposes. Because of this, additions and updates to tags on OpenSearch Service domains might take several minutes before they're available. # Tagging domains (console) The console is the simplest way to tag a domain. ****To create a tag (console)**** 1. Go to [https://aws.amazon.com](https://aws.amazon.com/ ), and then choose **Sign In to the Console**. 1. Under **Analytics**, choose **Amazon OpenSearch Service**. 1. Select the domain you want to add tags to and go to the **Tags** tab. 1. Choose **Manage** and **Add new tag**. 1. Enter a tag key and an optional value. 1. Choose **Save**. To delete a tag, follow the same steps and choose **Remove** on the **Manage tags** page. For more information about using the console to work with tags, see [Tag Editor](https://docs.aws.amazon.com/awsconsolehelpdocs/latest/gsg/tag-editor.html) in the *AWS Management Console Getting Started Guide*. # Tagging domains (AWS CLI) You can create resource tags using the AWS CLI with the **--add-tags** command. **Syntax** `add-tags --arn= --tag-list Key=,Value=` **** | Parameter | Description | | --- | --- | | --arn | Amazon resource name for the OpenSearch Service domain to which the tag is attached. | | --tag-list | Set of space-separated key-value pairs in the following format: Key=,Value= | **Example** The following example creates two tags for the *logs* domain: ``` aws opensearch add-tags --arn arn:aws:es:us-east-1:379931976431:domain/logs --tag-list Key=service,Value=OpenSearch Key=instances,Value=m3.2xlarge ``` You can remove tags from an OpenSearch Service domain using the **--remove-tags** command. ** Syntax ** `remove-tags --arn= --tag-keys Key=,Value=` **** | Parameter | Description | | --- | --- | | --arn | Amazon Resource Name (ARN) for the OpenSearch Service domain to which the tag is attached. | | --tag-keys | Set of space-separated key-value pairs that you want to remove from the OpenSearch Service domain. | **Example** The following example removes two tags from the *logs* domain that were created in the preceding example: ``` aws opensearch remove-tags --arn arn:aws:es:us-east-1:379931976431:domain/logs --tag-keys service instances ``` You can view the existing tags for an OpenSearch Service domain with the **--list-tags** command: **Syntax** `list-tags --arn=` **** | Parameter | Description | | --- | --- | | --arn | Amazon Resource Name (ARN) for the OpenSearch Service domain to which the tags are attached. | **Example** The following example lists all resource tags for the *logs* domain: ``` aws opensearch list-tags --arn arn:aws:es:us-east-1:379931976431:domain/logs ``` # Tagging domains (AWS SDKs) The AWS SDKs (except the Android and iOS SDKs) support all the actions defined in the [Amazon OpenSearch Service API Reference](https://docs.aws.amazon.com/opensearch-service/latest/APIReference/Welcome.html), including the `AddTags`, `ListTags`, and `RemoveTags` operations. For more information about installing and using the AWS SDKs, see [AWS Software Development Kits](https://aws.amazon.com/code). ## **Python** This example uses the [OpenSearchService](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/opensearch.html) low-level Python client from the AWS SDK for Python (Boto) to add a tag to a domain, list the tag attached to the domain, and remove a tag from the domain. You must provide values for `DOMAIN_ARN`, `TAG_KEY`, and `TAG_VALUE`. ``` import boto3 from botocore.config import Config # import configuration DOMAIN_ARN = '' # ARN for the domain. i.e "arn:aws:es:us-east-1:123456789012:domain/my-domain TAG_KEY = '' # The name of the tag key. i.e 'Smileyface' TAG_VALUE = '' # The value assigned to the tag. i.e 'Practicetag' # defines the configurations parameters such as region my_config = Config(region_name='us-east-1') client = boto3.client('opensearch', config=my_config) # defines the client variable def addTags(): """Adds tags to the domain""" response = client.add_tags(ARN=DOMAIN_ARN, TagList=[{'Key': TAG_KEY, 'Value': TAG_VALUE}]) print(response) def listTags(): """List tags that have been added to the domain""" response = client.list_tags(ARN=DOMAIN_ARN) print(response) def removeTags(): """Remove tags that have been added to the domain""" response = client.remove_tags(ARN=DOMAIN_ARN, TagKeys=[TAG_KEY]) print('Tag removed') return response ``` # Performing administrative actions on Amazon OpenSearch Service domains Amazon OpenSearch Service offers several administrative options that provide granular control if you need to troubleshoot issues with your domain. These options include the ability to restart the OpenSearch process on a data node and the ability to restart a data node. OpenSearch Service monitors node health parameters and, when there are anomalies, takes corrective actions to keep domains stable. With the administrative options to restart the OpenSearch process on a node, and restart a node itself, you have control over some of these mitigation actions. You can use the AWS Management Console, AWS CLI, or the AWS SDK to perform these actions. The following sections cover how to perform these actions with the console. # Restarting the OpenSearch process on a node in Amazon OpenSearch Service **To restart the OpenSearch process on a node** 1. Navigate to the OpenSearch Service console at [https://console.aws.amazon.com/aos/](https://console.aws.amazon.com/aos/). 1. In the left navigation pane, choose **Domains**. Choose the name of the domain that you want to work with. 1. After the domain details page opens, navigate to the **Instance health** tab. 1. Under **Data nodes**, select the button next to the node that you want to restart the process on. 1. Select the **Actions** dropdown and choose **Restart OpenSearch/Elasticsearch process**. 1. Choose **Confirm** on the modal. 1. To see the status of the action that you initiated, select the name of the node. After the node details page opens, choose the **Events** tab under the name of the node to see a list of events associated with that node. # Rebooting a data node in Amazon OpenSearch Service **To reboot a data node** 1. Navigate to the OpenSearch Service console at [https://console.aws.amazon.com/aos/](https://console.aws.amazon.com/aos/). 1. In the left navigation pane, choose **Domains**. Choose the name of the domain that you want to work with. 1. After the domain details page opens, navigate to the **Instance health** tab. 1. Under **Data nodes**, select the button next to the node that you want to restart the process on. 1. Select the **Actions** dropdown and choose **Reboot node**. 1. Choose **Confirm** on the modal. 1. To see the status of the action that you initiated, select the name of the node. After the node details page opens, choose the **Events** tab under the name of the node to see a list of events associated with that node. # Restarting the OpenSearch Dashboards process on a node in Amazon OpenSearch Service You can restart the OpenSearch Dashboards (previously Kibana) process to recover from issues such as a frozen interface, loading failures, or unresponsive visualizations. The option to restart OpenSearch Dashboards is only available for the node that is actively running the Dashboards process. In most OpenSearch Service domains, this process runs on dedicated coordinator nodes, not data nodes. As a result, when you open the **Actions** dropdown in the console, the option typically appears only for coordinator nodes. For more information, see [Dedicated coordinator nodes in Amazon OpenSearch Service](Dedicated-coordinator-nodes.md). This behavior depends on how your domain is configured. + **Domains with dedicated coordinator nodes** – The Dashboards process runs exclusively on those nodes, and only they show the restart option. + **Domains without dedicated coordinator nodes** – In simpler or older setups, Dashboards might run on a data node instead, and the restart option appears there. + **Master nodes** – These nodes are solely for managing cluster state and elections. They don't run Dashboards and never show the restart option. To determine which node is running the Dashboards process, navigate to the **Cluster configuration** section of your domain and review the node roles. The option to restart is only available for the node hosting the Dashboards process. **To restart the Dashboards process on a node** 1. Navigate to the OpenSearch Service console at [https://console.aws.amazon.com/aos/](https://console.aws.amazon.com/aos/). 1. In the left navigation pane, choose **Domains**. Choose the name of the domain that you want to work with. 1. After the domain details page opens, navigate to the **Instance health** tab. 1. In the section for the nodes that are running the Dashboards process, select the button next to the node that you want to restart the process on. 1. Select the **Actions** dropdown and choose **Restart Dashboard/Kibana process**. 1. Choose **Confirm** on the modal. 1. To see the status of the action that you initiated, select the name of the node. After the node details page opens, choose the **Events** tab under the name of the node to see a list of events associated with that node. ## Limitations Administrative options have the following limitations: + Administrative options are supported on Elasticsearch versions 7.x and higher. + Administrative options don't support domains with Multi-AZ with Standby enabled. + The OpenSearch and Elasticsearch process restart and the data node reboot are supported on domains with three or more data nodes. + The Dashboards and Kibana process support is supported on domains with two or more data nodes. + To restart the OpenSearch process on a node or reboot a node, the domain must not be in red state and all indexes must have replicas configured.