Automating Index Rollover in an Elasticsearch Cluster Through Index Lifecycle Management
Overview
Time series data is continuously written and increases index size. You can configure the lifecycle to periodically roll over data to new indexes and delete old indexes.
In this section, a lifecycle policy is configured. If the size of an index reaches 1 TB or the index has been created for more than one day, a new index will be automatically generated. Seven days after the index is created, the data copy will be disabled. Thirty days after the index is created, the index will be deleted.
Assume that an index generates about 2.4 TB data every day. The index alias is log-alias. The following figure shows the data format in Elasticsearch. During read, it points to all indexes starting with test. During write, it points to the latest index.
The one day in the rollover time refers to 24 hours after the index creation time, not a calendar day.
Prerequisites
- The CSS cluster is available.
- Elasticsearch 7.6.2 clusters or later are used.
Automating Index Rollover Through Index Lifecycle Management
- Log in to the CSS management console.
- In the navigation tree on the left, choose Clusters > Elasticsearch. The cluster list is displayed.
- Click Access Kibana in the Operation column of a cluster.
- In the navigation tree on the left of Kibana, choose Dev Tools. The command execution page is displayed.
- Create a rollover lifecycle policy named rollover_workflow.
Policy description: When the size of an index reaches 1 TB or the index has been created for more than one day, the index rollover is performed. When the index has been created for seven days, the data copy is disabled. When the index has been created for 30 days, the index is deleted.
PUT _opendistro/_ism/policies/rollover_workflow { "policy": { "description": "rollover test", "default_state": "hot", "states": [ { "name": "hot", "actions": [ { "rollover": { "min_size": "1tb", "min_index_age": "1d" } } ], "transitions": [ { "state_name": "warm", "conditions": { "min_index_age": "7d" } } ] }, { "name": "warm", "actions": [ { "replica_count": { "number_of_replicas": 0 } } ], "transitions": [ { "state_name": "delete", "conditions": { "min_index_age": "30d" } } ] }, { "name": "delete", "actions": [ { "delete": {} } ] } ] } }
After a lifecycle policy is created, run the following command to query the policy details:
GET _opendistro/_ism/policies/rollover_workflow
- Create the index template template_test.
Template description: All the new indexes starting with test are automatically associated with the rollover lifecycle policy rollover_workflow. The alias log_alias is used during rollover.
PUT _template/template_test { "index_patterns": "test*", "settings": { "number_of_replicas": 1, "number_of_shards": 1, "opendistro.index_state_management.policy_id": "rollover_workflow", "index.opendistro.index_state_management.rollover_alias": "log_alias" }, "mappings": { "properties": { "name": { "type": "text" } } } }
Table 1 Parameter description Parameter
Description
number_of_shards
Number of index shards
number_of_replicas
Number of index shard replicas
opendistro.index_state_management.policy_id
Lifecycle policy name
index.opendistro.index_state_management.rollover_alias
Index alias for rollover
After an index template is created, you can run the following command to query the template details:
GET _template/template_test
- Create an index, specify aliases, and set is_write_index to true. The index template template_test is automatically used for the index and is associated with the lifecycle policy rollover_workflow based on the index template configuration. In this way, when the index size reaches 1 TB or the index is created for more than one day, the rollover automatically starts. After an index is created for seven days, the data copy is disabled. After an index is created for 30 days, the index is deleted.
The following index is the URL code of <test-{now/d}-000001>. By default, an index name contains the creation date. For example, if an index is created on 2022-06-02, the index name is test-2022.06.02-000001.
PUT %3Ctest-%7Bnow%2Fd%7D-000001%3E { "aliases": { "log_alias": { "is_write_index": true } } }
- The alias log_alias is used to during data write, and log_alias always points to the last index.
POST log_alias/_bulk {"index":{}} {"name":"name1"} {"index":{}} {"name":"name2"} {"index":{}} {"name":"name3"} {"index":{}} {"name":"name4"} {"index":{}} {"name":"name5"} {"index":{}} {"name":"name6"}
- Query data and check whether the rollover takes effect.
- One day after the indexes are created, check the indexes starting with test.
GET _cat/indices/test*?s=i
There are supposed to be at least two indexes, for example:
green open test-<Date>-000001 r8ab5NX6T3Ox_hoGUanogQ 1 1 6 0 416b 208b green open test-<Date>-000002 sfwkVgy8RSSEw7W-xYjM2Q 1 1 0 0 209b 209b
In the preceding information, test-<Date>-000001 is the index created in 7, and test-<Date>-000002 is the index generated through rollover.
- To query the index associated with the alias log_alias, run the following command:
GET _cat/aliases/log_alias?v
The alias is supposed to point to multiple indexes, for example:
alias index filter routing.index routing.search is_write_index log_alias test-<Date>-000001 - - - false log_alias test-<Date>-000002 - - - true
- One day after the indexes are created, check the indexes starting with test.
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot