ALM-43210 Total Number of Elasticsearch Shards Exceeds the Threshold

Alarm Description

The system checks the total number of Elasticsearch shards every 60 seconds and compares the number with the threshold. This alarm is generated when the system detects that the number exceeds the threshold for multiple consecutive times (five times by default).

The threshold can be changed by choosing O&M > Alarm > Thresholds > Name of the desired cluster > Elasticsearch > Replica Quantity Statistics> Total shard number.

If Trigger Count is set to 1, and the total number of Elasticsearch shards is less than or equal to the threshold, this alarm is cleared. If Trigger Count is greater than 1, and the total number of Elasticsearch shards is less than or equal to 90% of the threshold, this alarm is cleared.

Alarm Attributes

Alarm Parameters

Impact on the System

If the total number of shards in the Elasticsearch cluster is too large, the index data read/write performance of Elasticsearch may be affected, and the shard restoration speed may be slow when the Elasticsearch service is restarted.

Possible Causes

The configuration of the Elasticsearch index shard number is inappropriate.

Handling Procedure

Check the total number of Elasticsearch shards.

1. Check whether the Elasticsearch cluster is in the security mode.

   Specifically, on FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > Elasticsearch. On the displayed page, click Configurations. Search for ELASTICSEARCH_SECURITY_ENABLE, and check whether the parameter can be queried and its value is true.

   • If yes, go to 2.
   • If no, go to 3.

2. If the security mode is used, configure the permission for running the curl command.
3. Log in to any node where Elasticsearch resides as user root.
4. Run the curl -XGET --tlsv1.2 --negotiate -k -v -u : 'https://ip:httpport/_cat/indices?v' command to query the index details in the current cluster.
   

   • In this command, replace ip with the IP address of any node in the cluster.
   • Replace httpport with the HTTP port number of the Elasticsearch instance, which is specified by SERVER_PORT. To obtain the parameter value, on FusionInsight Manager, choose Cluster > Services > Elasticsearch. On the displayed page, choose Configurations > All Configurations and search for SERVER_PORT.
   • In normal mode, delete the security authentication parameter --tlsv1.2 --negotiate -k -v -u : and change https to http.

5. Use either of the following methods:
   • Method 1: Delete the indexes that are no longer used in the cluster.
   • Method 2: Change the threshold of the total number of shards.

6. Five minutes later, check whether the alarm is cleared.
   • If yes, no further action is required.
   • If no, go to 7.

Collect fault information.

7. On FusionInsight Manager, choose O&M > Log > Download.
8. Select Elasticsearch in the required cluster for Service.
9. Click in the upper right corner. In the displayed dialog box, set Start Date and End Date to 10 minutes before and after the alarm generation time respectively and click OK. Then, click Download.
10. Contact the O&M engineers and send the collected logs.

Alarm Clearance

This alarm will be automatically cleared after the fault is rectified.

Related Information

None.