HDFS Capacity Reaches 100%, Causing Unavailable Upper-Layer Services Such as HBase and Spark

Issue

The HDFS capacity usage of the cluster reaches 100%, and the HDFS service status is read-only. As a result, upper-layer services such as HBase and Spark are unavailable.

Symptom

The HDFS capacity usage is 100%, the disk capacity usage is only about 85%, and the HDFS service status is read-only. As a result, upper-layer services such as HBase and Spark are unavailable.

Cause Analysis

Currently, NodeManager and DataNode share data disks. By default, MRS reserves 15% of data disk space for non-HDFS. You can change the percentage of data disk space by setting the HDFS parameter dfs.datanode.du.reserved.percentage.

If the HDFS disk usage is 100%, you can set dfs.datanode.du.reserved.percentage to a smaller value to restore services and then expand disk capacity.

Procedure

1. Log in to any Master node in the cluster.
2. Run the source /opt/client/bigdata_env command to initialize environment variables.
   

   If it is a security cluster, run the kinit -kt <keytab file> <Principal name> command for authentication.

3. Run the hdfs dfs -put ./startDetail.log /tmp command to check whether HDFS fails to write files.

   19/05/12 10:07:32 WARN hdfs.DataStreamer: DataStreamer Exception
   org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /tmp/startDetail.log._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1).  There are 3 datanode(s) running and no node(s) are excluded in this operation.

4. Run the hdfs dfsadmin -report command to check the used HDFS capacity. The command output shows that the HDFS capacity usage has reached 100%.

   Configured Capacity: 5389790579100 (4.90 TB)
   Present Capacity: 5067618628404 (4.61 TB)
   DFS Remaining: 133350196 (127.17 MB)
   DFS Used: 5067485278208 (4.61 TB)
   DFS Used%: 100.00%
   Under replicated blocks: 10
   Blocks with corrupt replicas: 0
   Missing blocks: 0
   Missing blocks (with replication factor 1): 0
   Pending deletion blocks: 0

5. When the HDFS capacity usage reaches 100%, change the percentage of data disk space by setting HDFS parameter dfs.datanode.du.reserved.percentage.
   1. Go to the service configuration page.
      • MRS Manager: Log in to MRS Manager and choose Services > HDFS > Configuration.
      • FusionInsight Manager: Log in to FusionInsight Manager and choose Cluster > Services > HDFS > Configurations.
   2. Click All Configurations and search for dfs.datanode.du.reserved.percentage.
   3. Change the value of this parameter to 10.

6. After the modification, increase the number of disks of the Core node.