ALM-43207 Elasticsearch Has Indexes Without Replicas

Alarm Description

The system checks whether there are indexes without replicas in Elasticsearch every 10 minutes. This alarm is generated when there are.

Alarm Attributes

Alarm ID	Alarm Severity	Alarm Type	Service Type	Auto Cleared
43207	Major	Quality of service	Elasticsearch	Yes

Alarm Parameters

Type	Parameter	Description
Location Information	Source	Specifies the cluster for which the alarm is generated.
	ServiceName	Specifies the service for which the alarm is generated.
	RoleName	Specifies the role for which the alarm is generated.
	HostName	Specifies the host for which the alarm is generated.
Additional Information	Index List	Specifies the index list without replicas.

Impact on the System

Some index data of Elasticsearch has only one copy, which affects the data reliability of Elasticsearch. If a single node is faulty, data may be lost.

Possible Causes

No replica is configured when an index is created, or the number of index replicas is modified and not restored.

Handling Procedure

Check indexes.

Check whether the Elasticsearch cluster is in the security mode.
Specifically, on FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > Elasticsearch. On the displayed page, click Configurations. Search for ELASTICSEARCH_SECURITY_ENABLE, and check whether the parameter can be queried and its value is true.
- If yes, go to 2.
- If no, go to 3.
If the security mode is used, configure the permission for running the curl command.
Log in to any node where Elasticsearch resides as user root.
Run the following command to query index information in the current cluster:
curl -XGET --tlsv1.2 --negotiate -k -v -u : 'https://ip:httpport/_cat/indices?v&pretty'
```
health status index   uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   test    s8wOFxAARtKkhEGSc5vgEQ   3   0          0            0      1.5kb           783b
```
- In this command, replace ip with the IP address of any node in the cluster.
- Replace httpport with the HTTP port number of the Elasticsearch instance, which is specified by SERVER_PORT. To obtain the parameter value, on FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > Elasticsearch. On the displayed page, choose Configurations > All Configurations and search for SERVER_PORT.
- In normal mode, delete the security authentication parameter --tlsv1.2 --negotiate -k -v -u: and change https to http.
- These rules also apply to the following curl commands.

Obtain the index not configured with replicas and configure index replicas as follows:
1. Run the following command to configure the number of index replicas:
  curl -XPUT --tlsv1.2 --negotiate -k -v -u : 'https://ip:httpport/index/_settings' -H 'Content-Type: application/json' -d'{ "number_of_replicas": 1 }'
2. Run the following command to query indexes in the cluster and ensure that replicas are configured for the index.
  curl -XGET --tlsv1.2 --negotiate -k -v -u : 'https://ip:httpport/_cat/indices?v&pretty'
```
health status index   uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   test    s8wOFxAARtKkhEGSc5vgEQ   3   1          0            0      1.5kb           783b
```
  In this command, replace index with the index not configured with replicas in the cluster. Regular expression match is supported, and * indicates all indexes.
  
  Replace number_of_replicas with the number of index replicas. The value 1 is recommended.

Manually clear the alarm.

Collect fault information.

On FusionInsight Manager, choose O&M > Log > Download.
Select Elasticsearch in the required cluster for Service.
Click in the upper right corner. In the displayed dialog box, set Start Date and End Date to 10 minutes before and after the alarm generation time respectively and click OK. Then, click Download.
Contact the O&M engineers and send the collected logs.