ALM-43202 Indices in the Yellow State Exist in Elasticsearch

Alarm Description

The system checks all indices status of all Elasticsearch every 60 seconds. This alarm is generated when an index is in the Yellow state.

Alarm Attributes

Alarm Parameters

Impact on the System

The replicai shards of some Elasticsearch indexes are faulty, which may slow down the read/write performance of Elasticsearch index data.

Possible Causes

The replica shards of some Elasticsearch instances are not allocated.

Handling Procedure

Check whether all instances are normal.

1. Choose Cluster > Name of the desired cluster > Services > Elasticsearch > Instance page, check whether all instances are in normal state.
   • If yes, go to 4.
   • If no, go to 2.

2. Select instances whose Running Status is not Normal. Choose Restart Instance from the drop-down list of More.
   

   When restarting the instance, you need enter the password of the FusionInsight Manager administrator.

3. Check whether the alarm is cleared from the alarm list.
   • If yes, no further action is required.
   • If yes, go to 4.

4. On FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > Elasticsearch. Click Resource. On the displayed page, check indexes with Health Status set to yellow in the Index Information area. Record the index names and number of replicas.
5. Check whether the Elasticsearch cluster is in the security mode.

   Specifically, on FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > Elasticsearch. On the displayed page, click Configurations. Search for ELASTICSEARCH_SECURITY_ENABLE, and check whether the parameter can be queried and its value is true.

   • If yes, go to 6.
   • If no, go to 7.

6. If the security mode is used, configure the permission for running the curl command.
7. Run the curl command to set the replica number.

   curl -XPUT --tlsv1.2 --negotiate -k -v -u : 'https://ip:httpport/index/_settings' -H 'Content-Type: application/json' -d'{"number_of_replicas": "0"}'

   

   • ip: The IP address of any node in the Elasticsearch cluster.
   • httpport: The HTTP port number of the Elasticsearch instance, which is specified by SERVER_PORT. To obtain the parameter value, on FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > Elasticsearch. On the displayed page, choose Configurations > All Configurations and search for SERVER_PORT.
   • index: Name of the index for which the number of copies needs to be reset.
   • In common mode, delete the security authentication parameter --tlsv1.2 --negotiate -k -v -u, and change https to http.
   • These rules also apply to the following curl commands.

8. 10 minutes later, check whether the alarm is cleared.
   • If yes, go to 9.
   • If no, go to 10.

9. Run the curl command to set the replica number to the original value. 10 minutes later, check whether the alarm is cleared.
   • If yes, no further action is required.
   • If no, go to 10.

Collect fault information.

10. On the FusionInsight Manager choose O&M > Log > Download.
11. Select Elasticsearch in the required cluster from the Service list.
12. Click in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.
13. Contact the O&M engineers and send the collected logs.

Alarm Clearance

After the fault is rectified, the system automatically clears this alarm.

Related Information

None.