Creating and Managing OpenSearch Index Policies

Context

Index State Management (ISM) of OpenSearch is a plugin that allows you to automate periodic, administrative operations on indexes by triggering them based on changes in the index age, index size, or number of documents. With ISM, you can define custom policies to automate index rollovers and deletion, thus optimizing cluster search performance or cutting storage costs. The procedure for using ISM is as follows:

1. Creating an Index Lifecycle Policy: Create an index lifecycle policy on OpenSearch Dashboards.
2. Associating Indexes with a Lifecycle Policy: Associate indexes with a lifecycle policy.
3. Managing Index Policies: Modify, retry, and change index lifecycle policies.

For more information about ISM, see Index State Management.

The OpenSearch Dashboards GUI may vary depending on the software version. OpenSearch Dashboards 1.3.6 is used as an example here.

Creating an Index Lifecycle Policy

1. Log in to the CSS management console.
2. In the navigation pane on the left, choose Clusters > OpenSearch.
3. In the cluster list, find the target cluster, and click Dashboards in the Operation column to log in to OpenSearch Dashboards.
4. Expand the OpenSearch Dashboards menu in the upper-left corner, and choose Index Management.
5. Click Create policy to create an index policy. In the Configuration method dialog box, select JSON editor and click Continue. The page for creating an index policy is displayed.
   • Policy ID: use a custom policy name.
   • Define policy: define a custom policy based on the reference example provided.
     Figure 1 Configuring a policy
     

     In the following example, a policy named policy1 is created. The ism_template part specifies the indexes to apply the policy to. After the policy is created, indexes whose names start with test are automatically associated with this policy.

     PUT _plugins/_ism/policies/policy1
     { 
       "policy": { 
         "description": "an example policy", 
         "default_state": "hot", 
         "states": [ 
             // Ignore the states part.
         ], 
         "ism_template": { 
           "index_patterns": [ 
             "test*" 
           ], 
           "priority": 100 
         } 
       } 
     }

6. Click Create.

Associating Indexes with a Lifecycle Policy

Managing Index Policies

1. On the Index Management page of the OpenSearch Dashboards, choose Managed Indices.

   The displayed page shows index policies configured for the current cluster.

2. Manage policies in the index policy list.
   • For a policy whose status is abnormal, click Retry policy.
   • To update a policy for an index, select the associated index, and click Change policy. After the policy is updated, click Change to apply the new policy.
   • To remove a policy from an index, select the index, and click Remove policy. In the displayed dialog box, click Remove.

For details, see Index State Management.

Disabling Auto Rollover for Empty Indexes

A lifecycle policy can be configured to enable automatic index rollover, that is, automatically creating a new index when an existing index meets certain conditions (for example, when it reaches a predefined age in terms of days). The new index then receive subsequent writes. By default, even if an existing index contains no documents, new indexes are still created. Over time, this can result in a large number of empty indexes. To prevent this, CSS allows you to disable automatic rollover for empty indexes. This is achieved by configuring a lifecycle policy to allow automatic rollover only for indexes that contain documents. By doing so, you can prevent excessive empty indexes from being created, thus optimizing storage utilization and enhancing overall system efficiency.

You can disable automatic rollover for empty indexes at either the index level or the cluster level. Index-level settings take precedence over cluster-level settings. You must configure this manually. The commands are as follows:

• Disable empty index automatic rollover for a specified index:

  PUT {index_name}/_settings
  {
    "index.plugins.index_state_management.rollover.only_if_has_documents": true
  }

• Disable empty index automatic rollover at the cluster level:

  PUT _cluster/settings
  {
    "persistent": {
      "plugins.index_state_management.rollover.only_if_has_documents": true
    }
  }

Configuring Auto Retry for ISM Tasks

ISM tasks (such as converting hot data to cold data and deleting expired indexes) can fail due to transient cluster issues like resource constraints, node restarts, or network interruptions. To address this, CSS automatically retries (reactivates) failed ISM tasks at configured intervals until they succeed. This ensures the continuity and reliability of ISM tasks.

Automatic retry of ISM tasks is enabled by default for OpenSearch clusters that meet the conditions above.

For OpenSearch clusters that do not meet these conditions, upgrade them to enable this feature. However, ISM tasks that have already failed before the upgrade will not be retried after the cluster is upgraded. You will need to manually retry them before the automatic retry can occur. The command for retrying failed ISM tasks is as follows:

POST _opendistro/_ism/retry/{index_name}

PUT _cluster/settings
{
  "persistent": {
    "plugins.index_state_management.coordinator.css.reactivate": true,
    "plugins.index_state_management.coordinator.css.reactivate_period": "5m",
    "plugins.index_state_management.coordinator.css.reactivate_duration": "30m",
    "plugins.index_state_management.coordinator.css.reactivate_max_duration": "12h",
    "plugins.index_state_management.coordinator.css.max_inflight_reactivate_tasks": 1000
  }
}

Idempotence considerations for ISM task retry

• Idempotent operations: All steps are retried, starting from the beginning, to ensure data consistency.
• Non-idempotent operations: The retry begins from where the failure occurred. If the original failure cause persists, retries may fail repeatedly, in which case, manual intervention is required.

  Examples of non-idempotent operations in OpenSearch's ISM tasks: ForceMerge, Notification, Snapshot, and Shrink (only for OpenSearch 2.19.0).

In the ISM task details API, CSS has added fields that record ISM task execution details. You can run the following command to check ISM task records, including the total number of failures, failure occurrence time, activation time, and failure causes:

GET /_opendistro/_ism/explain/{index_name}

Example response:

{
  "rollover-0000001" : {
    "index.opendistro.index_state_management.policy_id" : "logs-policy-rollover",
    "index" : "rollover-0000001",
    ...
    "reactivate_info" : {
      "count": 2, //Total number of failures.
      "latest_failed_time": 1764301006792, //Latest failure time.
      "latest_reactivate_time": 1764301486800, //Latest activation time.
      "failed_infos": [  //Historical execution failure records. Only the latest 10 records are retained.
        {
          "start_time": 1764299686801, //Task execution time. The earliest execution time is the index creation time.
          "failed_time": 1764299686814, //Task failure time.
          "info": {  //Task failure cause.
            "message" : "Missing rollover_alias index setting [index=rollover-0000001]"
          }
        },
        {
          "start_time" : 1764300286841,
          "failed_time" : 1764301006792,
          "info" : {
            "message" : "Missing rollover_alias index setting [index=rollover-0000001]"
          }
        }
      ]
    },
    ...
  }
}