Logs Cannot Be Written In Due to High CPU Usage of Elasticsearch

Symptom

The CPU usage of an Elasticsearch cluster is high, an error message Elasticsearch Unreachable is reported by Logstash, and logs cannot be written to Elasticsearch.

Possible Causes

The indexes have only one shard. This could easily overload individual nodes. When the job queue is full, later jobs are rejected.

Procedure

1. Log in to the CSS management console.
2. In the navigation pane on the left, choose Clusters > Elasticsearch.
3. In the cluster list, find the target cluster, and choose More > Cerebro in the Operation column. Log in to Cerebro.
4. In Cerebro, check the number of shards in the cluster and metrics such as the CPU, load, head, and disk of each node.
5. Analyze the possible causes based on metrics and tune your system accordingly.
   1. Increase the number of queues and reduce rejected jobs by changing the value of write.queue_size.
      1. Click the name of the target cluster. The cluster information page is displayed.
      2. Choose Cluster Settings > Parameter Settings.
      3. Expand Custom, find the write.queue_size parameter and increase its value.

         If this parameter does not exist, add it under Custom.

         For more information, see Parameter Settings.

   2. Rebuild the indexes to ensure that the number of shards is greater than that of nodes in the cluster.
6. If the number of shards and queues are appropriate but the CPU usage and load are still high, you are advised to scale out the cluster.