Insufficient Disk Space

Symptom

Insufficient disk space occurs when one or more disk partitions of an Elasticsearch/OpenSearch cluster approach or exceed a predefined usage threshold. This condition can degrade performance, cause service malfunctions, or lead to a complete outage for workloads dependent on those partitions. Typical symptoms include:

• A cluster's disk usage reaches 85% of its total capacity, and no new indexes can be created.
• A cluster's disk usage reaches 90% of its total capacity, and the cluster becomes read-only.
• Data is ingested faster than a cluster can enter read-only mode, and its disk usage reaches 100%. This leads to process failures.

Possible Causes

• Uneven shard distribution: Index shards are not evenly distributed across the cluster nodes, causing significantly higher disk usage on some nodes compared to others.
• Insufficient initial disk capacity: The storage capacity originally allocated to the cluster nodes is inadequate for the actual data volumes or workloads.

Solutions

1. Check the number of shards on each node to determine whether they are balanced across all nodes.

   GET _cat/shards?v

   If there is unbalanced shard distribution, perform the following steps to fix the issue:

   1. Use the _cluster/reroute API to manually migrate shards to balance the load.

      POST _cluster/reroute
      {
        "commands": [
          {
            "move": {
              "index": "index_name",	//Destination index name
              "shard": 0,		//Destination shard ID
              "from_node": "node1",	//Source node ID
              "to_node": "node2" //Destination node ID, indicating a node with relatively low load
            }
          }
        ]
      }

   2. Reduce the number of replicas and shards to reduce the storage overhead.

      PUT /index_name/_settings
      {
        "number_of_replicas": 1
      }

2. If shards are balanced across the cluster nodes, try to reclaim storage by deleting obsolete data.

   Delete obsolete indexes.

   DELETE {index_name}

3. If these steps do not work, expand the cluster's storage capacity.