ALM-43203 Indices in the Red State Exist in Elasticsearch

Alarm Description

The system checks all indices status of all Elasticsearch every 60 seconds. This alarm is generated when an index is in the Red state.

Alarm Attributes

Alarm ID	Alarm Severity	Alarm Type	Service Type	Auto Cleared
43203	Critical	Quality of service	Elasticsearch	Yes

Alarm Parameters

Type	Parameter	Description
Location Information	Source	Specifies the cluster for which the alarm is generated.
	ServiceName	Specifies the service for which the alarm is generated.
	RoleName	Specifies the role for which the alarm is generated.
	HostName	Specifies the host for which the alarm is generated.
Additional Information	List of indices in the red state	Specifies the list of indices in red state in Elasticsearch.

The length of the index list is restricted by the character length. When the content exceeds 256 characters, only some index names can be displayed. To query the complete index list, perform the following steps:

On the FusionInsight Manager homepage, choose O&M > Log > Online Search.
Enter "alarm 43203" in the Search Content text box. Then, select OMS > Agent for Service, the pluginmonitor file for File, and ERROR for Lowest Log Level. After the configuration, click Search.

Select the latest log in the search result and view the complete list of indices in red state. For example:

2020-07-10 14:34:00,508 ERROR [monitor_60_1_18_EsMaster] Send alarm 43203. The Elasticsearch cluster has indexes in the Red state, List of indexes in the Red state is [myindex292,myindex200]

Impact on the System

The primary shards of some Elasticsearch indexes are faulty, and the faulty indexes cannot be read or written.

Possible Causes

The primary shard of Elasticsearch is missing.

Handling Procedure

Check whether all the instances are normal.

Specifically, log in to FusionInsight Manager, and choose Cluster > Name of the desired cluster > Services > Elasticsearch. On the displayed page, click Instance and check whether all instances are in normal state.
- If yes, go to 4.
- If no, go to 2.
Select instances whose running status is not Normal and choose Restart Instance from the drop-down list of More.

When restarting the instance, you need enter the password of the FusionInsight Manager administrator.
Check whether the alarm is cleared from the alarm list.
- If yes, no further action is required.
- If yes, go to 4.
On FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > Elasticsearch.
Click Resource. On the displayed page, check indexes with Health Status set to red in the Index Information area.
Check whether the index is an invalid index.
- If yes, go to 7.
- If no, go to 11.
Check whether the Elasticsearch cluster is in the security mode.

Specifically, on FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > Elasticsearch. On the displayed page, click Configurations. Search for ELASTICSEARCH_SECURITY_ENABLE, and check whether the parameter can be queried and its value is true.
- If yes, go to 8.
- If no, go to 9.
If the security mode is used, configure the permission for running the curl command.
Delete the invalid index.

curl -XDELETE --tlsv1.2 --negotiate -k -v -u : 'https://ip:httpport/ index name'
- In this command, replace ip with the IP address of any node in the cluster.
- Replace httpport with the HTTP port number of the Elasticsearch instance, which is specified by SERVER_PORT. To obtain the parameter value, on FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > Elasticsearch. On the displayed page, choose Configurations > All Configurations and search for SERVER_PORT.
- Replace index name with the name of the index to be deleted.
- In common mode, delete the security authentication parameter --tlsv1.2 --negotiate -k -v -u, and change https to http.
- Deleting a file or folder is a high-risk operation. Ensure that the file or folder is no longer required before performing this operation.
5 minutes later, check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 11.

Collect fault information.

On the FusionInsight Manager, choose O&M> Log > Download.
Select Elasticsearch in the required cluster from the Service list.
Click in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.
Contact the O&M engineers and send the collected logs.