The Peak Heap Memory of an Elasticsearch Cluster Remains High (Over 90%)

Symptom

The peak heap memory of an Elasticsearch cluster remains high (over 90%). If the heap memory usage of a node does not remain above 90%, the cluster is normal. If the heap memory usage remains above this rate for a long time, the cluster faces a certain risk of unavailability.

Possible Causes

• Check whether there are many tasks waiting in the write and query queues of the cluster.

  GET /_cat/thread_pool/write?v

  GET /_cat/thread_pool/search?v

• View the cluster monitoring information. Check the metrics related to the write and query tasks of the cluster.
• If the heap memory usage of the cluster remains high for a long time, check the cluster size and the number of nodes, and scale out the cluster if necessary.

Solution

• Optimize the write and query programs on the client based on the task queuing statistics.
• If the cluster is heavily loaded for a long time, it may cause slow write, query, and frequent node disconnections. To avoid these problems, you can increase nodes or redesign the cluster.
• If the heap memory fluctuates around 95% and nodes are sometimes disconnected, use the traffic control function as needed. For more information, see Configuring Flow Control for an Elasticsearch Cluster.