Creating and Managing Index Policies for an Elasticsearch Cluster

Context

Index State Management (ISM) of Elasticsearch is a plugin that allows you to automate periodic, administrative operations on indexes by triggering them based on changes in the index age, index size, or number of documents. With ISM, you can define custom policies to automate index rollovers and deletion, thus optimizing cluster search performance or cutting storage costs. The procedure for using ISM is as follows:

1. Creating an Index Lifecycle Policy: Create an index lifecycle policy on OpenSearch Dashboards.
2. Associating Indexes with a Lifecycle Policy: Associate indexes with a lifecycle policy.
3. Managing Index Policies: Modify, retry, and change index lifecycle policies.

For more information about ISM, see Index State Management.

Common use cases for index lifecycle policies:

• Automating Index Rollover in an Elasticsearch Cluster Through Index Lifecycle Management
• Decoupling Index Storage and Compute in an Elasticsearch Cluster Through Index Lifecycle Management

Constraints

• ISM is available only in Elasticsearch 7.6.2 clusters or later or OpenSearch clusters.
• You can customize policy names in Kibana. Chinese characters are not supported.
• The Kibana GUI varies depending on the Kibana version. Kibana 7.6.2 is used as an example here.

Creating an Index Lifecycle Policy

1. Log in to the CSS management console.
2. On the Clusters page, locate the target cluster, and click Access Kibana in the Operation column to log in to the Kibana page.
3. Choose IM or Index Management from the menu bar to go to the index management page.
4. Click Create policy to configure an index lifecycle policy.
   • Policy ID: use a custom policy name.
   • Define policy: define a custom policy based on the reference example provided.
   Figure 1 Configuring a policy
   
5. Click Create.

Associating Indexes with a Lifecycle Policy

You can attach a policy to one or more indexes, or associate a policy with an index template, so that the policy will be applied to all indexes created using this index template.

Use either of the following methods to associate a lifecycle policy with an index.

• Method 1: Kibana commands

  On the Dev Tools page of Kibana, run the following command to associate a policy ID with an index template:

  PUT _template/<template_name> 
  {
      "index_patterns": ["index_name-*"],
      "settings": {
          "opendistro.index_state_management.policy_id": "policy_id"
      }
  }

  • <template_name>: Replace it with the name of a created index template.
  • policy_id: Replace it with a custom policy ID.

  For details about how to create an index template, see Index Template.

• Method 2: Kibana console
  1. On the Kibana console, choose IM or Index Management from the menu bar to go to the index management page.
  2. Select Indices on the left.
     Figure 2 Choosing Indices
     
  3. In the Indices list, select the target index to which you want to bind a policy.
  4. Click Apply policy in the upper right corner. In the displayed dialog box, select a policy ID.
     Figure 3 Adding an index policy
     
     Figure 4 Selecting a policy
     
  5. Click Apply.

     After you bind a policy to an index, ISM creates a job that runs every 5 minutes by default to execute the policy, check criteria, and change index states.

Managing Index Policies

1. Log in to the CSS management console.
2. On the Clusters page, locate the target cluster, and click Access Kibana in the Operation column to log in to the Kibana page.
3. Choose IM or Index Management from the menu bar to go to the index management page.
4. Choose Managed Indices on the left.

   The displayed page shows index policies configured for the current cluster.

5. Manage policies in the index policy list.
   • For a policy whose status is abnormal, click Retry policy.
   • To update a policy for an index, select the associated index, and click Change policy. After the policy is updated, click Change to apply the new policy.
   • To remove a policy from an index, select the index, and click Remove policy. In the displayed dialog box, click Remove.

For details, see Index State Management.