ALM-43206 Elasticsearch Shard Document Number Exceeds the Threshold

Alarm Description

The system checks the number of documents in Elasticsearch shards every 60 seconds and compares the number with the threshold. This alarm is generated when the system detects that the number exceeds the threshold for multiple consecutive times (three times by default).

The threshold can be changed by choosing O&M > Alarm > Thresholds > Name of the desired cluster > Elasticsearch > Shard > Elasticsearch Shard Document Number (EsMaster).

If Trigger Count is set to 1, and the number of documents in Elasticsearch shards is less than or equal to the threshold, this alarm is cleared. If Trigger Count is greater than 1, and the number of documents in Elasticsearch shards is less than or equal to 90% of the threshold, this alarm is cleared.

Alarm Attributes

Alarm Parameters

Impact on the System

If the number of Elasticsearch shard documents is too large, the read and write performance of Elasticsearch index data may be affected. When the Elasticsearch process is restarted, the restoration speed of shards with a large amount of data may be slow.

Possible Causes

The configuration of Elasticsearch index shard number is inappropriate.

Handling Procedure

Check the number of documents in shards.

1. Check whether the Elasticsearch cluster is in the security mode.
   Specifically, on FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > Elasticsearch. On the displayed page, click Configurations. Search for ELASTICSEARCH_SECURITY_ENABLE, and check whether the parameter can be queried and its value is true.

   • If yes, go to 2.
   • If no, go to 3.

2. If the security mode is used, configure the permission for running the curl command.
3. Log in to any node where Elasticsearch resides as user root.
4. Run the following command to query the stored shard documents volume of the current cluster: curl -XGET --tlsv1.2 --negotiate -k -v -u : 'https://ip:httpport/_cat/shards?v&s=docs:desc'
   

   • In this command, replace ip with the IP address of any node in the cluster.
   • Replace httpport with the HTTP port number of the Elasticsearch instance, which is specified by SERVER_PORT. To obtain the parameter value, on FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > Elasticsearch. On the displayed page, choose Configurations > All Configurations and search for SERVER_PORT.
   • In normal mode, delete the security authentication parameter --tlsv1.2 --negotiate -k -v -u :, and change https to http.

5. Obtain the indexes with a large number of shard documents and plan the index again as follows:
   • Method 1: Stop writing data to the index and plan a new index to store the written data.
   • Method 2: Migrate the data in the index whose number of shard documents exceeds the threshold to the planned index, and delete the old index.
     

     Deleting a file or folder is a high-risk operation. Ensure that the file or folder is no longer required before performing this operation.

6. After the index planning is complete, Five minutes after the index planning is complete, check whether the alarm is cleared.
   • If yes, no further action is required.
   • If no, go to 7.

Collect fault information.

7. On FusionInsight Manager, choose O&M > Log > Download.
8. Select Elasticsearch in the required cluster for Service.
9. Click in the upper right corner. In the displayed dialog box, set Start Date and End Date to 10 minutes before and after the alarm generation time respectively and click OK. Then, click Download.
10. Contact the O&M engineers and send the collected logs.

Alarm Clearance

After the fault is rectified, the system automatically clears this alarm.

Related Information

None.