Index Lifecycle Management

Scenario

Based on the time management, you can periodically perform some operations on the old index (for example, reducing the number of replicas or deleting the index) to ensure that there are sufficient resources in the cluster. The index life cycle management plug-in allows users to periodically trigger management operations based on the index existence time, size, and document quantity to implement automatic management.

Common Policy Commands

List of Supported Actions

For details about the ISM function of the open-source open distro plug-in, see https://opendistro.github.io/for-elasticsearch-docs/docs/ism/.

Common Condition Parameters

Example

1. Create a policy.

   PUT  _opendistro/_ism/policies/default_policy
   {
       "policy":{
           "description":"this policy is for all action.",
           "default_state":"hot",
           "states":[
               {
                   "name":"hot",
                   "actions":[
                       {
                           "index_priority":{
                               "priority":10
                           }
                       },
                       {
                           "rollover":{
                               "min_size":"1tb",
                               "min_doc_count":"1000000000",
                               "min_index_age":"1d"
                           }
                       }
                   ],
                   "transitions":[
                       {
                           "state_name":"warm",
                           "conditions":{
                               "min_index_age":"15d"
                           }
                       }
                   ]
               },
               {
                   "name":"warm",
                   "actions":[
                       {
                           "index_priority":{
                               "priority":5
                           }
                       },
                       {
                           "read_only":{
   
                           }
                       },
                       {
                           "allocation":{
                               "require":{
                                   "temperature":"warm"
                               }
                           }
                       },
                       {
                           "force_merge":{
                               "max_num_segments":1
                           }
                       }
                   ],
                   "transitions":[
                       {
                           "state_name":"cold",
                           "conditions":{
                               "min_index_age":"60d"
                           }
                       }
                   ]
               },
               {
                   "name":"cold",
                   "actions":[
                       {
                           "index_priority":{
                               "priority":1
                           }
                       },
                       {
                           "freeze_index":{
   
                           }
                       },
                       {
                           "shrink":{
                               "number_of_shards":10
                           }
                       }
                   ],
                   "transitions":[
                       {
                           "state_name":"close",
                           "conditions":{
                               "min_index_age":"90d"
                           }
                       }
                   ]
               },
               {
                   "name":"close",
                   "actions":[
                       {
                           "close":{
   
                           }
                       }
                   ]
               }
           ],
           "ism_template":{
               "index_patterns":[
                   "indextest*"
               ],
               "priority":100
           }
       }
   }

   Bind the policy to the index.

   After creating a policy, you need to apply the policy to an index or a group of indexes. You can also add the policy to the index template so that the policy can be applied to new indexes.

   Specify the policy and alias when creating the first index.

   PUT indextest-000001
   {
     "settings": {
       "opendistro.index_state_management.policy_id": "policy_id",
       "opendistro.index_state_management.rollover_alias": "indextest"
     },
     "aliases": {
       "indextest": {
         "is_write_index": true
       }
     }
   }

   

   • If shrink is configured in the policy, set cluster.routing.allocation.same_shard.host to false to ensure that all shards can be migrated to one instance. Otherwise, the shrink operation cannot be performed.

     You can run the following command to change the value of cluster.routing.allocation.same_shard.host:

     PUT _cluster/settings{ "persistent": { "cluster.routing.allocation.same_shard.host":"false" }}

   • If a rolling index is used, the index name must meet the regular expression of ^.*-\d+$.

   • You need to create the first index that complies with the naming rules. The index alias can be used for subsequent write and query operations.

   Add the policy to the index template.

   PUT _template/<template_name>
   {
     "index_patterns": ["indextest*"],                 
     "settings": {      
       "index.opendistro.index_state_management.policy_id": "policy_id",
       "index.opendistro.index_state_management.rollover_alias":"indextest" 
     }
   }

   

   • Set index_patterns based on the service scenario. Do not set index_patterns to *.

   • The value of policy_id cannot contain special characters such as the number sign (#).

   Run the following command to add the policy to the existing index in the cluster:

   PUT /indextest-000001/_settings?pretty
   {
     "opendistro.index_state_management.policy_id": "policy_id",
     "opendistro.index_state_management.rollover_alias": "indextest"
   }
   POST /_aliases
   {
     "actions": [
       {
         "add": {
           "index": "indextest-000001",
           "alias": "indextest",
           "is_write_index": true
         }
       }
     ]
   }

   

   • The new policy can be scheduled only when the cluster is in the yellow or green state. If the cluster is in the red state, restore the indexes in the red state first.
   • If a rolling index is used, the index name must meet the regular expression of ^.*-\d+$.

Sample of Cold and Hot Separation Configuration

1. Create a warm instance group.

   Log in to FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > Elasticsearch > Instance Groups, click under EsNodeX, set Group Name to warm, and click OK to create a warm instance group.

2. Add instances to the instance group.

   On the Basic tab page of EsNodeX, select the instance to be set to warm, click Move, set Target Instance Group to warm, and click OK to move the instance to the new warm instance group.

3. Add the warm tag to the warm instances.

   In the newly created warm instance group, choose Configuration > All Configurations > Hot-cold mode and set the value of node.attr.temperature to warm.

4. Restart the warm instances.

   After the instance of the warm group is restarted, the warm configuration takes effect.