Automating Index Rollover in an OpenSearch Cluster Through Index Lifecycle Management

Overview

Time series data is continuously written and increases index size. You can configure the lifecycle to periodically roll over data to new indexes and delete old indexes.

In this section, a lifecycle policy is configured. If the size of an index reaches 1 TB or the index has been created for more than one day, a new index will be automatically generated. Seven days after the index is created, the data copy will be disabled. Thirty days after the index is created, the index will be deleted.

Assume that an index generates approximately 2.4 TB of data each day. The index alias is log-alias. The following figure illustrates the data format in the cluster. During reads, it points to all indexes whose name starts with test. During writes, it points to the latest index.

Figure 1 log-alias format

The one day in the rollover time refers to 24 hours following the index creation time, not a calendar day.

Prerequisites

There are available CSS clusters.
There are available OpenSearch clusters.

Automating Index Rollover Through Index Lifecycle Management

Log in to the CSS management console.
In the navigation tree on the left, choose Clusters > OpenSearch. The cluster list is displayed.
Click Access Kibana in the Operation column of a cluster, and log in to OpenSearch Dashboards.
In the navigation tree on the left of OpenSearch Dashboards, choose Dev Tools. The command execution page is displayed.

Create a rollover lifecycle policy named rollover_workflow.

Policy description: When the size of an index reaches 1 TB or its age exceeds one day, index rollover is performed automatically. When the index age reaches seven days, data replicas are disabled. When it reaches 30 days, the index is deleted.

PUT _plugins/_ism/policies/rollover_workflow
{
  "policy": {
    "description": "rollover test",
    "default_state": "hot",
    "states": [
      {
        "name": "hot",
        "actions": [
          {
            "rollover": {
              "min_size": "1tb",
              "min_index_age": "1d"
            }
          }
        ],
        "transitions": [
          {
            "state_name": "warm",
            "conditions": {
              "min_index_age": "7d"
            }
          }
        ]
      },
      {
        "name": "warm",
        "actions": [
          {
            "replica_count": {
              "number_of_replicas": 0
            }
          }
        ],
        "transitions": [
          {
            "state_name": "delete",
            "conditions": {
              "min_index_age": "30d"
            }
          }
        ]
      },
      {
        "name": "delete",
        "actions": [
          {
            "delete": {}
          }
        ]
      }
    ],
    "ism_template": {
      "index_patterns": [
        "test*"
      ],
      "priority": 100
    }
  }
}

After a lifecycle policy is created, run the following command to query the policy details:

GET _plugins/_ism/policies/rollover_workflow

Create the index template template_test.

Template description: All the new indexes starting with test are automatically associated with the rollover lifecycle policy rollover_workflow. The alias log_alias is used during rollover.

PUT _template/template_test
{
  "index_patterns": "test*",
  "settings": {
    "number_of_replicas": 1,
    "number_of_shards": 1,
    "index.opendistro.index_state_management.rollover_alias": "log_alias"
  },
  "mappings": {
    "properties": {
      "name": {
        "type": "text"
      }
    }
  }
}

**Table 1** Parameter description
Parameter	Description
number_of_shards	Number of index shards
number_of_replicas	Number of index shard replicas
opendistro.index_state_management.policy_id	Lifecycle policy name
index.opendistro.index_state_management.rollover_alias	Index alias for rollover

After an index template is created, you can run the following command to query the template details:

GET _template/template_test

Create an index, specify aliases, and set is_write_index to true. The index template template_test is automatically used for the index and is associated with the lifecycle policy rollover_workflow based on the index template configuration. In this way, when the index size reaches 1 TB or the index is created for more than one day, the rollover automatically starts. After an index is created for seven days, the data copy is disabled. After an index is created for 30 days, the index is deleted.
The following index is the URL code of <test-{now/d}-000001>. By default, an index name contains the creation date. For example, if an index is created on 2022-06-02, the index name is test-2022.06.02-000001.
```
PUT %3Ctest-%7Bnow%2Fd%7D-000001%3E
{
  "aliases": {
    "log_alias": {
      "is_write_index": true
    }
  }
}
```

The alias log_alias is used to during data write, and log_alias always points to the last index.

POST log_alias/_bulk
{"index":{}}
{"name":"name1"}
{"index":{}}
{"name":"name2"}
{"index":{}}
{"name":"name3"}
{"index":{}}
{"name":"name4"}
{"index":{}}
{"name":"name5"}
{"index":{}}
{"name":"name6"}

Query data and check whether the rollover has taken effect.
- One day after the indexes are created, check the indexes starting with test.
```
GET _cat/indices/test*?s=i
```
  There are supposed to be at least two indexes, for example:
```
green open test-<Date>-000001 r8ab5NX6T3Ox_hoGUanogQ 1 1 6 0 416b 208b
green open test-<Date>-000002 sfwkVgy8RSSEw7W-xYjM2Q 1 1 0 0 209b 209b
```
  In the preceding information, test-<Date>-000001 is the index created in 7, and test-<Date>-000002 is the index generated through rollover.
- To query the index associated with the alias log_alias, run the following command:
```
GET _cat/aliases/log_alias?v
```
  The alias is supposed to point to multiple indexes, for example:
```
alias     index                  filter routing.index routing.search is_write_index
log_alias test-<Date>-000001      -      -             -              false
log_alias test-<Date>-000002      -      -             -              true
```