ALM-43208 Elasticsearch Data Directory Usage Exceeds the Threshold

Alarm Description

The system checks the Elasticsearch data directory usage every 60 seconds and compares the usage with the threshold. This alarm is generated when the system detects that the usage exceeds the threshold for multiple consecutive times (three times by default).

The threshold can be changed by choosing O&M > Alarm > Thresholds > Name of the desired cluster > Elasticsearch > Data Directory Usage.

If Trigger Count is set to 1, and the Elasticsearch data directory usage is less than or equal to the threshold, this alarm is cleared. If Trigger Count is greater than 1, and the Elasticsearch data directory usage is less than or equal to 90% of the threshold, this alarm is cleared.

Alarm Attributes

Alarm Parameters

Impact on the System

The remaining space of the Elasticsearch data directory is insufficient, and new data cannot be written.

Possible Causes

The service data volume exceeds the cluster storage capacity.

Handling Procedure

Delete invalid indexes.

1. Check whether the Elasticsearch cluster is in the security mode.

   >Specifically, on FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > Elasticsearch. On the displayed page, click Configurations. Search for ELASTICSEARCH_SECURITY_ENABLE, and check whether the parameter can be queried and its value is true.

   • If yes, go to 2.
   • If no, go to 3.

2. If the security mode is used, configure the permission for running the curl command.
3. Log in to any node where Elasticsearch resides as user root.
4. Run the curl -XGET --tlsv1.2 --negotiate -k -v -u : 'https://ip:httpport/_cat/indices?v&pretty' command to query indexes in the cluster.
   

   • In this command, replace ip with the IP address of any node in the cluster.
   • Replace httpport with the HTTP port number of the Elasticsearch instance, which is specified by SERVER_PORT. To obtain the parameter value, on FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > Elasticsearch. On the displayed page, choose Configurations > All Configurations and search for SERVER_PORT.
   • In normal mode, delete the security authentication parameter --tlsv1.2 --negotiate -k -v -u : and change https to http.
   • The rules also apply to the following curl commands.

5. Run the curl -XDELETE --tlsv1.2 --negotiate -k -u : "https://ip:httpport/index" command to delete invalid or expired indexes if there are.
   

   • In this command, replace index with the index planned to be deleted in the cluster. Regular expression match is supported, and * indicates all indexes.
   • Deleting a file or folder is a high-risk operation. Ensure that the file or folder is no longer required before performing this operation.

6. Five minutes later, check whether the alarm is cleared.
   • If yes, no further action is required.
   • If no, go to 7.

Expand the cluster capacity.

7. Expand the Elasticsearch cluster capacity.

Collect fault information.

8. On FusionInsight Manager, choose O&M > Log > Download.
9. Select Elasticsearch in the required cluster for Service.
10. Click in the upper right corner. In the displayed dialog box, set Start Date and End Date to 10 minutes before and after the alarm generation time respectively and click OK. Then, click Download.
11. Contact the O&M engineers and send the collected logs.

Alarm Clearance

This alarm must be manually cleared after the fault is rectified.

Related Information

None.