Insufficient Disk Space
Symptom
Insufficient disk space occurs when one or more disk partitions of an Elasticsearch/OpenSearch cluster approach or exceed a predefined usage threshold. This condition can degrade performance, cause service malfunctions, or lead to a complete outage for workloads dependent on those partitions. Typical symptoms include:
- The disk usage of cluster data nodes (including cold data nodes) exceeds 90%. Nodes exceeding this threshold cannot allocate new shards. If all nodes exceed the 90% threshold, the cluster is unable to allocate new shards, and creating an index will cause the cluster status to turn red.
- The disk usage of cluster data nodes (including cold data nodes) exceeds 95%. When the disk usage of any node hosting a shard exceeds 95%, the index enters read-only protection mode and cannot accept data writes.
- A large volume of data merging or excessively fast writes cause the index to fail to enter read-only protection mode in time, resulting in 100% disk usage. This leads to process failures. 100% disk usage is a critical issue. In this case, expand the disk capacity.
Possible Causes
- Uneven shard distribution: Index shards are not evenly distributed across the cluster nodes, causing significantly higher disk usage on some nodes compared to others.
- Insufficient initial disk capacity: The storage capacity originally allocated to the cluster nodes is inadequate for the actual data volumes or workloads.
Solutions
- Check the number of shards on each node to determine whether they are balanced across all nodes.
GET _cat/shards?v
If there is unbalanced shard distribution, perform the following steps to fix the issue:
- Use the _cluster/reroute API to manually migrate shards to balance the load.
POST _cluster/reroute { "commands": [ { "move": { "index": "index_name", //Destination index name "shard": 0, //Destination shard ID "from_node": "node1", //Source node ID "to_node": "node2" //Destination node ID, indicating a node with relatively low load } } ] } - Reduce the number of replicas to lower storage overhead.
PUT /index_name/_settings { "number_of_replicas": 1 }
- Use the _cluster/reroute API to manually migrate shards to balance the load.
- If shards are balanced across the cluster nodes, try to reclaim storage by deleting obsolete data.
DELETE {index_name} - If these steps do not work, expand the cluster's storage capacity.