# Index State Management in Amazon OpenSearch Service
Index State Management (ISM) in Amazon OpenSearch Service lets you define custom management policies that automate routine tasks, and apply them to indexes and index patterns. You no longer need to set up and manage external processes to run your index operations.
A policy contains a default state and a list of states for the index to transition between. Within each state, you can define a list of actions to perform and conditions that trigger these transitions. A typical use case is to periodically delete old indexes after a certain period of time. For example, you can define a policy that moves your index into a `read_only` state after 30 days and then ultimately deletes it after 90 days.
After you attach a policy to an index, ISM creates a job that runs every 5 to 8 minutes (or 30 to 48 minutes for pre-1.3 clusters) to perform policy actions, check conditions, and transition the index into different states. The base time for this job to run is every 5 minutes, plus a random 0-60% jitter is added to it to make sure you do not see a surge of activity from all your indexes at the same time. ISM doesn't run jobs if the cluster state is red.
ISM requires OpenSearch or Elasticsearch 6.8 or later.
**Note**
This documentation provides a brief overview of ISM and several sample policies. It also explains how ISM for Amazon OpenSearch Service domains differs from ISM on self-managed OpenSearch clusters. For full documentation of ISM, including a comprehensive parameter reference, descriptions of each setting, and an API reference, see [Index State Management](https://docs.opensearch.org/latest/im-plugin/ism/index/) in the OpenSearch documentation.
**Important**
You can no longer use index templates to apply ISM policies to newly created indexes. You can continue to automatically manage newly created indexes with the [ISM template field](https://opensearch.org/docs/latest/im-plugin/ism/policies/#sample-policy-with-ism-template-for-auto-rollover). This update introduces a breaking change that affects existing CloudFormation templates using this setting.
## Create an ISM policy
**To get started with Index State Management**
1. Open the Amazon OpenSearch Service console at [https://console.aws.amazon.com/aos/home](https://console.aws.amazon.com/aos/home).
1. Select the domain that you want to create an ISM policy for.
1. From the domain's dashboard, navigate to the OpenSearch Dashboards URL and sign in with your master username and password. The URL follows this format:
```
domain-endpoint/_dashboards/
```
1. Open the left navigation panel within OpenSearch Dashboards and choose **Index Management**, then **Create policy**.
1. Use the [visual editor](https://opensearch.org/docs/latest/im-plugin/ism/index/#visual-editor) or [JSON editor](https://opensearch.org/docs/latest/im-plugin/ism/index/#json-editor) to create policies. We recommend using the visual editor as it offers a more structured way of defining policies. For help creating policies, see the [sample policies](#ism-example) below.
1. After you create a policy, attach it to one or more indexes:
```
POST _plugins/_ism/add/my-index
{
"policy_id": "my-policy-id"
}
```
**Note**
If your domain is running a legacy Elasticsearch version, use `_opendistro` instead of `_plugins`.
Alternatively, select the index in OpenSearch Dashboards and choose **Apply policy**.
## Sample policies
The following sample policies demonstrate how to automate common ISM use cases.
### Hot to warm to cold storage
This sample policy moves an index from hot storage to [UltraWarm](ultrawarm.md), and eventually to
[cold storage](cold-storage.md). Then, it deletes the index.
The index is initially in the `hot` state. After ten days, ISM moves it
to the `warm` state. 80 days later, after the index is 90 days old, ISM moves the index to the
`cold` state. After a year, the service sends a notification to an Amazon Chime room that the index is being deleted and then permanently deletes it.
Note that cold indexes require the `cold_delete` operation rather than the normal `delete` operation. Also note that an explicit `timestamp_field` is required in your data in order to manage cold indexes with ISM.
```
{
"policy": {
"description": "Demonstrate a hot-warm-cold-delete workflow.",
"default_state": "hot",
"schema_version": 1,
"states": [{
"name": "hot",
"actions": [],
"transitions": [{
"state_name": "warm",
"conditions": {
"min_index_age": "10d"
}
}]
},
{
"name": "warm",
"actions": [{
"warm_migration": {},
"retry": {
"count": 5,
"delay": "1h"
}
}],
"transitions": [{
"state_name": "cold",
"conditions": {
"min_index_age": "90d"
}
}]
},
{
"name": "cold",
"actions": [{
"cold_migration": {
"timestamp_field": ""
}
}
],
"transitions": [{
"state_name": "delete",
"conditions": {
"min_index_age": "365d"
}
}]
},
{
"name": "delete",
"actions": [{
"notification": {
"destination": {
"chime": {
"url": ""
}
},
"message_template": {
"source": "The index {{ctx.index}} is being deleted."
}
}
},
{
"cold_delete": {}
}]
}
]
}
}
```
### Reduce replica count
This sample policy reduces replica count to zero after seven days to conserve disk space and then deletes the index after 21 days. This policy assumes your index is non-critical and no longer receiving write requests; having zero replicas carries some risk of data loss.
```
{
"policy": {
"description": "Changes replica count and deletes.",
"schema_version": 1,
"default_state": "current",
"states": [{
"name": "current",
"actions": [],
"transitions": [{
"state_name": "old",
"conditions": {
"min_index_age": "7d"
}
}]
},
{
"name": "old",
"actions": [{
"replica_count": {
"number_of_replicas": 0
}
}],
"transitions": [{
"state_name": "delete",
"conditions": {
"min_index_age": "21d"
}
}]
},
{
"name": "delete",
"actions": [{
"delete": {}
}],
"transitions": []
}
]
}
}
```
### Take an index snapshot
This sample policy uses the `[snapshot](https://docs.opensearch.org/latest/im-plugin/ism/policies/#snapshot)` operation to take a snapshot of an index as soon as it contains at least one document. `repository` is the name of the manual snapshot repository you registered in Amazon S3. `snapshot` is the name of the snapshot. For snapshot prerequisites and steps to register a repository, see [Creating index snapshots in Amazon OpenSearch Service](managedomains-snapshots.md).
```
{
"policy": {
"description": "Takes an index snapshot.",
"schema_version": 1,
"default_state": "empty",
"states": [{
"name": "empty",
"actions": [],
"transitions": [{
"state_name": "occupied",
"conditions": {
"min_doc_count": 1
}
}]
},
{
"name": "occupied",
"actions": [{
"snapshot": {
"repository": "",
"snapshot": ""
}
}],
"transitions": []
}
]
}
}
```
## ISM templates
You can set up an `ism_template` field in a policy so when you create an index that matches the template pattern, the policy is automatically attached to that index. In this example, any index you create with a name that begins with "log" is automatically matched to the ISM policy `my-policy-id`:
```
PUT _plugins/_ism/policies/my-policy-id
{
"policy": {
"description": "Example policy.",
"default_state": "...",
"states": [...],
"ism_template": {
"index_patterns": ["log*"],
"priority": 100
}
}
}
```
For a more detailed example, see [Sample policy with ISM template for auto rollover](https://opensearch.org/docs/latest/im-plugin/ism/policies/#sample-policy-with-ism-template-for-auto-rollover).
## Differences
Compared to OpenSearch and Elasticsearch, ISM for Amazon OpenSearch Service has several differences.
### ISM operations
+ OpenSearch Service supports three unique ISM operations, `warm_migration`, `cold_migration`, and `cold_delete`:
+ If your domain has [UltraWarm](ultrawarm.md) enabled, the `warm_migration` action transitions the index to warm storage.
+ If your domain has [cold storage](cold-storage.md) enabled, the `cold_migration` action transitions the index to cold storage, and the `cold_delete` action deletes the index from cold storage.
Even if one of these actions doesn’t complete within the [set timeout period](https://docs.opensearch.org/latest/im-plugin/ism/policies/#actions), the migration or deletion of indexes still continues. Setting an [error\$1notification](https://opensearch.org/docs/latest/im-plugin/ism/policies/#error-notifications) for one of the above actions will notify you that the action failed if it didn’t complete within the timeout period, but the notification is only for your own reference. The actual operation has no inherent timeout and continues to run until it eventually succeeds or fails.
+ If your domain runs OpenSearch or Elasticsearch 7.4 or later, OpenSearch Service supports the ISM `open` and `close` operations.
+ If your domain runs OpenSearch or Elasticsearch 7.7 or later, OpenSearch Service supports the ISM `snapshot` operation.
### Cold storage ISM operations
For cold indexes, you must specify a `?type=_cold` parameter when you use the following ISM APIs:
+ [Add policy](https://opensearch.org/docs/latest/im-plugin/ism/api/#add-policy)
+ [Remove policy](https://opensearch.org/docs/latest/im-plugin/ism/api/#remove-policy-from-index)
+ [Update policy](https://opensearch.org/docs/latest/im-plugin/ism/api/#update-policy)
+ [Retry failed index](https://opensearch.org/docs/latest/im-plugin/ism/api/#retry-failed-index)
+ [Explain index](https://opensearch.org/docs/latest/im-plugin/ism/api/#explain-index)
These APIs for cold indexes have the following additional differences:
+ Wildcard operators are not supported except when you use it at the end. For example, `_plugins/_ism//logstash-*` is supported but `_plugins/_ism//iad-*-prod` isn’t supported.
+ Multiple index names and patterns are not supported. For example, `_plugins/_ism//app-logs` is supported but `_plugins/_ism//app-logs,sample-data` isn’t supported.
### ISM settings
OpenSearch and Elasticsearch let you change all available ISM settings using the `_cluster/settings` API. On Amazon OpenSearch Service, you can only change the following [ISM settings](https://opensearch.org/docs/latest/im-plugin/ism/settings/):
+ **Cluster-level settings:**
+ `plugins.index_state_management.enabled`
+ `plugins.index_state_management.history.enabled`
+ **Index-level settings:**
+ `plugins.index_state_management.rollover_alias`
# Tutorial: Automating Index State Management processes
This tutorial demonstrates how to implement an ISM policy that automates routine index management tasks and apply them to indexes and index patterns.
[Index State Management (ISM)](ism.md) in Amazon OpenSearch Service lets you automate recurring index management activities, so you can avoid using additional tools to manage index lifecycles. You can create a policy that automates these operations based on index age, size, and other conditions, all from within your Amazon OpenSearch Service domain.
OpenSearch Service supports three storage tiers: the default "hot" state for active writing and low-latency analytics, UltraWarm for read-only data up to three petabytes, and cold storage for unlimited long-term archival.
This tutorial presents a sample use case of handling time-series data in daily indexes. In this tutorial, you set up a policy that takes an automated snapshot of each attached index after 24 hours. It then migrates the index from the default hot state to UltraWarm storage after two days, cold storage after 30 days, and finally deletes the index after 60 days.
## Prerequisites
+ Your OpenSearch Service domain must be running Elasticsearch version 6.8 or later.
+ Your domain must have [UltraWarm](ultrawarm.md) and [cold storage](cold-storage.md) enabled.
+ You must [register a manual snapshot repository](managedomains-snapshot-registerdirectory.md) for your domain.
+ Your user role needs sufficient permissions to access the OpenSearch Service console. If necessary, validate and [configure access to your domain](ac.md).
## Step 1: Configure the ISM policy
First, configure an ISM policy in OpenSearch Dashboards.
1. From your domain dashboard in the OpenSearch Service console, navigate to the OpenSearch Dashboards URL and sign in with your master username and password. The URL follows this format: `domain-endpoint/_dashboards/`.
1. In OpenSearch Dashboards, choose **Add sample data** and add one or more of the sample indexes to your domain.
1. Open the left navigation panel and choose **Index Management**, then choose **Create policy**.
1. Name the policy `ism-policy-example`.
1. Replace the default policy with the following policy:
```
{
"policy": {
"description": "Move indexes between storage tiers",
"default_state": "hot",
"states": [
{
"name": "hot",
"actions": [],
"transitions": [
{
"state_name": "snapshot",
"conditions": {
"min_index_age": "24h"
}
}
]
},
{
"name": "snapshot",
"actions": [
{
"retry": {
"count": 5,
"backoff": "exponential",
"delay": "30m"
},
"snapshot": {
"repository": "snapshot-repo",
"snapshot": "ism-snapshot"
}
}
],
"transitions": [
{
"state_name": "warm",
"conditions": {
"min_index_age": "2d"
}
}
]
},
{
"name": "warm",
"actions": [
{
"retry": {
"count": 5,
"backoff": "exponential",
"delay": "1h"
},
"warm_migration": {}
}
],
"transitions": [
{
"state_name": "cold",
"conditions": {
"min_index_age": "30d"
}
}
]
},
{
"name": "cold",
"actions": [
{
"retry": {
"count": 5,
"backoff": "exponential",
"delay": "1h"
},
"cold_migration": {
"start_time": null,
"end_time": null,
"timestamp_field": "@timestamp",
"ignore": "none"
}
}
],
"transitions": [
{
"state_name": "delete",
"conditions": {
"min_index_age": "60d"
}
}
]
},
{
"name": "delete",
"actions": [
{
"cold_delete": {}
}
],
"transitions": []
}
],
"ism_template": [
{
"index_patterns": [
"index-*"
],
"priority": 100
}
]
}
}
```
**Note**
The `ism_template` field automatically attaches the policy to any newly created index that matches one of the specified `index_patterns`. In this case, all indexes that start with `index-`. You can modify this field to match an index format in your environment. For more information, see [ISM templates](ism.md#ism-template).
1. In the `snapshot` section of the policy, replace `snapshot-repo` with the name of the [snapshot repository](managedomains-snapshot-registerdirectory.md) that you registered for your domain. You can also optionally replace `ism-snapshot`, which will be the name of snapshot when it's created.
1. Choose **Create**. The policy is now visible on the **State management policies** page.
## Step 2: Attach the policy to one or more indexes
Now that you created your policy, attach it to one or more indexes in your cluster.
1. Go to the **Hot indicies** tab and search for `opensearch_dashboards_sample`, which lists all of the sample indexes that you added in step 1.
1. Select all of the indexes and choose **Apply policy**, then choose the **ism-policy-example** policy that you just created.
1. Choose **Apply**.
You can monitor the indexes as they move through the various states on the **Policy managed indices** page.