There Is a Sudden Spike in Cluster Load

Symptom

For a long time, many of a cluster's tasks are rejected and a large number of tasks are suspended. The Cerebro console shows a sudden, sharp increase in the cluster's load values.

Possible Causes

Possible causes are as follows:

• Query threads are executed slowly because a large amount of data is obtained.
• Threads are suspended caused by high read pressures.

Troubleshooting Procedure

Method 1: Using Cerebro

1. Log in to the CSS management console.
2. In the navigation pane on the left, expand Clusters. Select a cluster type based on the target cluster. The cluster list is displayed.
3. Locate the cluster whose load has spiked, and choose More > Cerebro in the Operation column.
4. Check the CPU and heap metrics. If the values of these two metrics are too high, the cluster is overloaded. In this case, reduce the number of requests sent by the client and wait until the cluster load decreases.
5. Check the number and size of shards. Each shard is recommended to be 20 GB to 40 GB and not exceed 50 GB. On a single node, up to five shards can use the same index.

Method 2: Using Kibana

1. Log in to the CSS management console.
2. In the navigation pane on the left, expand Clusters. Select a cluster type based on the target cluster. The cluster list is displayed.
3. For an Elasticsearch cluster, click Kibana in the Operation column to log in to Kibana. For an OpenSearch cluster, click Dashboards in the Operation column to log in to OpenSearch Dashboards.
4. Expand the menu in the upper-left corner, and choose Dev Tools.
5. Run the following command to check which threads are having tasks piling up and locate the cause of the sudden spike in cluster load.

   GET _cat/thread_pool?v

6. Run the following command to check which threads occupy excessive CPU resources and take a long time to execute, and locate the cause of task delaying.

   GET /_nodes/hot_threads