Creating and Managing Index Policies for an Elasticsearch Cluster

Context

Index State Management (ISM) of Elasticsearch is a plugin that allows you to automate periodic, administrative operations on indexes by triggering them based on changes in the index age, index size, or number of documents. With ISM, you can define custom policies to automate index rollovers and deletion, thus optimizing cluster search performance or cutting storage costs. The procedure for using ISM is as follows:

1. Creating an Index Lifecycle Policy: Create an index lifecycle policy on OpenSearch Dashboards.
2. Associating Indexes with a Lifecycle Policy: Associate indexes with a lifecycle policy.
3. Managing Index Policies: Modify, retry, and change index lifecycle policies.

For more information about ISM, see Index State Management.

Constraints

• ISM is available only in Elasticsearch 7.6.2 clusters or later.
• You can customize policy names in Kibana. Chinese characters are not supported.
• The Kibana GUI varies depending on the Kibana version. Kibana 7.6.2 is used as an example here.

Creating an Index Lifecycle Policy

1. Log in to the CSS management console.
2. In the navigation pane on the left, choose Clusters > Elasticsearch.
3. In the cluster list, find the target cluster, and click Kibana in the Operation column to log in to the Kibana console.
4. On the Kibana console, expand the menu in the upper-left corner, and choose IM or Index Management to go to the index management page.
5. Click Create policy to configure an index lifecycle policy.
   • Policy ID: use a custom policy name.
   • Define policy: define a custom policy based on the reference example provided.
   Figure 1 Configuring a policy
   
6. Click Create.

Associating Indexes with a Lifecycle Policy

You can attach a policy to one or more indexes, or associate a policy with an index template, so that the policy will be applied to all indexes created using this index template.

Use either of the following methods to associate a lifecycle policy with an index.

• Method 1: Kibana commands

  On the Dev Tools page of Kibana, run the following command to associate a policy ID with an index template:

  PUT _template/<template_name> 
  {
      "index_patterns": ["index_name-*"],
      "settings": {
          "opendistro.index_state_management.policy_id": "policy_id"
      }
  }

  • <template_name>: Replace it with the name of a created index template.
  • policy_id: Replace it with a custom policy ID.

  For details about how to create an index template, see Index Template.

• Method 2: Kibana console
  1. On the Kibana console, choose IM or Index Management from the menu bar to go to the index management page.
  2. Select Indices on the left.
     Figure 2 Choosing Indices
     
  3. In the Indices list, select the target index to which you want to bind a policy.
  4. Click Apply policy in the upper right corner. In the displayed dialog box, select a policy ID.
     Figure 3 Adding an index policy
     
     Figure 4 Selecting a policy
     
  5. Click Apply.

     After you bind a policy to an index, ISM creates a job that runs every 5 minutes by default to execute the policy, check criteria, and change index states.

Managing Index Policies

1. On the Index Management page of the Kibana console, choose Managed Indices.

   The displayed page shows index policies configured for the current cluster.

2. Manage policies in the index policy list.
   • For a policy whose status is abnormal, click Retry policy.
   • To update a policy for an index, select the associated index, and click Change policy. After the policy is updated, click Change to apply the new policy.
   • To remove a policy from an index, select the index, and click Remove policy. In the displayed dialog box, click Remove.

For details, see Index State Management.

Disabling Auto Rollover for Empty Indexes

A lifecycle policy can be configured to enable automatic index rollover, that is, automatically creating a new index when an existing index meets certain conditions (for example, when it reaches a predefined age in terms of days). The new index then receive subsequent writes. By default, even if an existing index contains no documents, new indexes are still created. Over time, this can result in a large number of empty indexes. To prevent this, CSS allows you to disable automatic rollover for empty indexes. This is achieved by configuring a lifecycle policy to allow automatic rollover only for indexes that contain documents. By doing so, you can prevent excessive empty indexes from being created, thus optimizing storage utilization and enhancing overall system efficiency.

You can disable automatic rollover for empty indexes at either the index level or the cluster level. Index-level settings take precedence over cluster-level settings. You must configure this manually. The commands are as follows:

• Disable empty index automatic rollover for a specified index:

  PUT {index_name}/_settings
  {
    "index.plugins.index_state_management.rollover.only_if_has_documents": true
  }

• Disable empty index automatic rollover at the cluster level:

  PUT _cluster/settings
  {
    "persistent": {
      "plugins.index_state_management.rollover.only_if_has_documents": true
    }
  }

Configuring Auto Retry for ISM Tasks

ISM tasks (such as converting hot data to cold data and deleting expired indexes) can fail due to transient cluster issues like resource constraints, node restarts, or network interruptions. To address this, CSS automatically retries (reactivates) failed ISM tasks at configured intervals until they succeed. This ensures the continuity and reliability of ISM tasks.

Automatic retry of ISM tasks is enabled by default for Elasticsearch clusters that meet the conditions above.

For Elasticsearch clusters that do not meet these conditions, upgrade them to enable this feature. However, ISM tasks that have already failed before the upgrade will not be retried after the cluster is upgraded. You will need to manually retry them before the automatic retry can occur. The command for retrying failed ISM tasks is as follows:

POST _opendistro/_ism/retry/{index_name}

PUT _cluster/settings
{
  "persistent": {
    "plugins.index_state_management.coordinator.css.reactivate": true,
    "plugins.index_state_management.coordinator.css.reactivate_period": "5m",
    "plugins.index_state_management.coordinator.css.reactivate_duration": "30m",
    "plugins.index_state_management.coordinator.css.reactivate_max_duration": "12h",
    "plugins.index_state_management.coordinator.css.max_inflight_reactivate_tasks": 1000
  }
}

Idempotence considerations for ISM task retry

• Idempotent operations: All steps are retried, starting from the beginning, to ensure data consistency.
• Non-idempotent operations: The retry begins from where the failure occurred. If the original failure cause persists, retries may fail repeatedly, in which case, manual intervention is required.

  Examples of non-idempotent operations in Elasticsearch's index lifecycle management tasks: ForceMerge, Notification, and Snapshot.

In the ISM task details API, CSS has added fields that record ISM task execution details. You can run the following command to check ISM task records, including the total number of failures, failure occurrence time, activation time, and failure causes:

GET /_opendistro/_ism/explain/{index_name}

Example response:

{
  "rollover-0000001" : {
    "index.opendistro.index_state_management.policy_id" : "logs-policy-rollover",
    "index" : "rollover-0000001",
    ...
    "reactivate_info" : {
      "count": 2, //Total number of failures.
      "latest_failed_time": 1764301006792, //Latest failure time.
      "latest_reactivate_time": 1764301486800, //Latest activation time.
      "failed_infos": [  //Historical execution failure records. Only the latest 10 records are retained.
        {
          "start_time": 0, //Task execution time. The earliest execution time is 0.
          "failed_time": 1764299686814, //Task failure time.
          "info": {  //Task failure cause.
            "message" : "Missing rollover_alias index setting [index=rollover-0000001]"
          }
        },
        {
          "start_time" : 1764300286841,
          "failed_time" : 1764301006792,
          "info" : {
            "message" : "Missing rollover_alias index setting [index=rollover-0000001]"
          }
        }
      ]
    },
    ...
  }
}