HDFS Displays Insufficient Disk Space But 10% Disk Space Remains

Symptom

1. The alarm "HDFS Disk Usage Exceeds the Threshold" is reported.
2. On the HDFS page, high disk space usage is displayed.

Cause Analysis

The dfs.datanode.du.reserved.percentage parameter is set in HDFS, indicating the percentage of the reserved space of each disk to the total disk space. The DataNode reserves space you set for NodeManager running and computing of other components, for example, Yarn, or for upgrades.

As 10% disk space is reserved, the HDFS DataNode regards that there is no available disk space when the disk usage reaches 90%.

Solution

1. Expand the capacity when the HDFS DataNode disk usage reaches 80%. For details, see Scaling Out a Cluster. After the scale-out is complete, balance data. For details, see Configuring HDFS DataNode Data Balancing.
2. If the disk capacity cannot be expanded in time, delete useless data in HDFS to release disk space.