What Do I Do If Logs Cannot Be Written to CSS Due to High CPU Usage of Elasticsearch?

Symptom

The CPU usage of Elasticsearch is high, an error message "Elasticsearch Unreachable" is displayed on Logstash, and logs cannot be written to Elasticsearch.

Possible Causes

The customer index has only one shard. The node of the shard is overloaded, and the job queue is full. Later jobs are rejected.

Procedure

1. Log in to the CSS management console.
2. Choose Clusters in the navigation pane. The cluster list is displayed.
3. Locate the target cluster and choose More > Access Cerebro in the Operation column.

   If the cluster is in security mode, you need to enter the login username (admin) and password.

4. In Cerebro, view the number of shards in the cluster and metrics such as the CPU, load, head, and dis of each node.
5. Analyze the possible causes based on metrics and tune your system accordingly.
   1. Increase the number of queues and reduce rejected jobs by changing the value of write.queue_size.
      1. Click the name of the target cluster whose parameters you want to modify. The basic information page of the cluster is displayed.
      2. Click Parameter Configurations, search for write.queue_size, and change its value.

         If this parameter does not exist, add it in the Customize area. For details, see Configuring Parameters.

   2. Rebuild the indexes to ensure that the number of shards is greater than that of nodes in the cluster.
6. If the number of shards and queues are appropriate but the CPU usage and load are still high, you are advised to scale out the cluster.