ALM-43202 Indices in the Yellow State Exist in Elasticsearch

Updated on 2024-11-29 GMT+08:00

View PDF

Alarm Description

The system checks all indices status of all Elasticsearch every 60 seconds. This alarm is generated when an index is in the Yellow state.

Alarm Attributes

Alarm ID	Alarm Severity	Alarm Type	Service Type	Auto Cleared
43202	Major	Quality of service	Elasticsearch	Yes

Alarm Parameters

Type	Parameter	Description
Location Information	Source	Specifies the cluster for which the alarm is generated.
	ServiceName	Specifies the service for which the alarm is generated.
	RoleName	Specifies the role for which the alarm is generated.
	HostName	Specifies the host for which the alarm is generated.
Additional Information	List of indices in the yellow state	Specifies the list of indices in yellow state in Elasticsearch.

NOTE:

The length of the index list is restricted by the character length. When the content exceeds 256 characters, only some index names can be displayed. To query the complete index list, perform the following steps:

On the FusionInsight Manager homepage, choose O&M > Log > Online Search. Enter "alarm 43202" in the Search Content text box. Then, select OMS > Agent for Service, the pluginmonitor file for File, and ERROR for Lowest Log Level. After the configuration, click Search.

Select the latest log in the search result and view the complete list of indices in yellow state. For example:

Impact on the System

The replicai shards of some Elasticsearch indexes are faulty, which may slow down the read/write performance of Elasticsearch index data.

Possible Causes

The replica shards of some Elasticsearch instances are not allocated.

Handling Procedure

Check whether all instances are normal.

Choose Cluster > Name of the desired cluster > Services > Elasticsearch > Instance page, check whether all instances are in normal state.
- If yes, go to 4.
- If no, go to 2.
Select instances whose Running Status is not Normal. Choose Restart Instance from the drop-down list of More.

NOTE:

When restarting the instance, you need enter the password of the FusionInsight Manager administrator.
Check whether the alarm is cleared from the alarm list.
- If yes, no further action is required.
- If yes, go to 4.
On FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > Elasticsearch. Click Resource. On the displayed page, check indexes with Health Status set to yellow in the Index Information area. Record the index names and number of replicas.
Check whether the Elasticsearch cluster is in the security mode.

Specifically, on FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > Elasticsearch. On the displayed page, click Configurations. Search for ELASTICSEARCH_SECURITY_ENABLE, and check whether the parameter can be queried and its value is true.
- If yes, go to 6.
- If no, go to 7.
If the security mode is used, configure the permission for running the curl command.
Run the curl command to set the replica number.

curl -XPUT --tlsv1.2 --negotiate -k -v -u : 'https://ip:httpport/index/_settings' -H 'Content-Type: application/json' -d'{"number_of_replicas": "0"}'
NOTE:
- ip: The IP address of any node in the Elasticsearch cluster.
- httpport: The HTTP port number of the Elasticsearch instance, which is specified by SERVER_PORT. To obtain the parameter value, on FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > Elasticsearch. On the displayed page, choose Configurations > All Configurations and search for SERVER_PORT.
- index: Name of the index for which the number of copies needs to be reset.
- In common mode, delete the security authentication parameter --tlsv1.2 --negotiate -k -v -u, and change https to http.
- These rules also apply to the following curl commands.
10 minutes later, check whether the alarm is cleared.
- If yes, go to 9.
- If no, go to 10.
Run the curl command to set the replica number to the original value. 10 minutes later, check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 10.

Collect fault information.

On the FusionInsight Manager choose O&M > Log > Download.
Select Elasticsearch in the required cluster from the Service list.
Click in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.
Contact the O&M engineers and send the collected logs.