Insufficient Disk Space

Symptom

Insufficient disk space occurs when one or more disk partitions of an Elasticsearch/OpenSearch cluster approach or exceed a predefined usage threshold. This condition can degrade performance, cause service malfunctions, or lead to a complete outage for workloads dependent on those partitions. Typical symptoms include:

• The disk usage of cluster data nodes (including cold data nodes) exceeds 90%. Nodes exceeding this threshold cannot allocate new shards. If all nodes exceed the 90% threshold, the cluster is unable to allocate new shards, and creating an index will cause the cluster status to turn red.
• The disk usage of cluster data nodes (including cold data nodes) exceeds 95%. When the disk usage of any node hosting a shard exceeds 95%, the index enters read-only protection mode and cannot accept data writes.
• A large volume of data merging or excessively fast writes cause the index to fail to enter read-only protection mode in time, resulting in 100% disk usage. This leads to process failures. 100% disk usage is a critical issue. In this case, expand the disk capacity.

Possible Causes

• Uneven shard distribution: Index shards are not evenly distributed across the cluster nodes, causing significantly higher disk usage on some nodes compared to others.
• Insufficient initial disk capacity: The storage capacity originally allocated to the cluster nodes is inadequate for the actual data volumes or workloads.

Solutions

1. Check the number of shards on each node to determine whether they are balanced across all nodes.

   GET _cat/shards?v

   If there is unbalanced shard distribution, perform the following steps to fix the issue:

   1. Use the _cluster/reroute API to manually migrate shards to balance the load.

      POST _cluster/reroute
      {
        "commands": [
          {
            "move": {
              "index": "index_name",	//Destination index name
              "shard": 0,		//Destination shard ID
              "from_node": "node1",	//Source node ID
              "to_node": "node2" //Destination node ID, indicating a node with relatively low load
            }
          }
        ]
      }

   2. Reduce the number of replicas to lower storage overhead.

      PUT /index_name/_settings
      {
        "number_of_replicas": 1
      }

2. If shards are balanced across the cluster nodes, try to reclaim storage by deleting obsolete data.

   Delete obsolete indexes.

   DELETE {index_name}

3. If these steps do not work, expand the cluster's storage capacity.