The MRS Cluster Reports an Alarm Indicating Insufficient HDFS Disk Space, But the Disk Space Is Not Used Up

Symptom

1. ALM-14001 HDFS Disk Usage Exceeds the Threshold is reported for the MRS cluster.
2. On the HDFS client or HDFS web UI, the disk space is consumed heavily but not fully used up.

Cause Analysis

In HDFS, you can configure the percentage of the reserved disk space of DataNodes by setting the dfs.datanode.du.reserved.percentage parameter. Its default value is 10%.

When DataNodes store data, it calculates and reserves some disk space based on this parameter. The disk space is not used to store HDFS data blocks but is reserved for the OS, other applications, temporary files, or upgrades.

By default, 10% of the disk space is reserved. When the disk usage reaches 90%, the cluster reports an alarm indicating that the HDFS disk space is insufficient.

Procedure

1. If the cluster has reported an alarm, adjust the alarm threshold to a proper value by referring to the alarm help.
2. Log in to the HDFS client and run the following commands to delete unnecessary files or directories from the HDFS.
   1. Run the following command to check files or sub-directories in the folder.

      hdfs dfs -ls /Folder name

   2. Run the following command to query the HDFS file directory size.

      hdfs dfs -du HDFS file directory

      For example, if you run the above command to query the size of files or sub-directories in the /tmp folder, the following command output is returned, indicating that the /tmp/hadoop-yarn directory occupies 622,222,017 bytes.

      0          0          /tmp/.testHDFS
      207407339  622222017  /tmp/hadoop-yarn
      0          0          /tmp/hive
      0          0          /tmp/hive-scratch
      0          0          /tmp/logs
      ...

   3. Run the following command to delete confirmed unnecessary HDFS files.

      hdfs dfs -rm -r HDFS file or directory path

      For example, you can delete expired logs and temporary files, such as historical logs of MapReduce/YARN and temporary output of Spark jobs. Generally, you can delete files generated 30 days ago.

      You can also delete temporary directories (for example, /user/test) generated during development and test and data tables of decommissioned services after confirming with the upper-layer service side that they are no longer needed.

      

      Deleting files is a high-risk operation. Ensure that the files to be deleted are unnecessary files to prevent data loss.

   4. By default, the HDFS recycle bin is enabled. If the space is insufficient, you can manually clear the recycle bin.

      # Check the recycle bin path of the current user.
      hdfs dfs -ls /user/UserA/.Trash 
      # Clear the recycle bin.
      hdfs dfs -rm -r /user/UserA/.Trash/*

3. Archive or migrate cold data.

   You are advised to migrate data that is not frequently accessed but needs to be retained to low-cost storage (such as OBS), using the tools like Distcp.

4. Scale out the MRS cluster.

   If more than 80% of the disk space is used on HDFS DataNodes, scale out the MRS cluster by adding DataNode groups. For details, see Scaling Out an MRS Cluster.

   After DataNodes are added, balance data on the DataNodes. For details, see Configuring HDFS DataNode Data Balancing.

   After data balancing is complete, run the hdfs dfsadmin -report command to check that the disk usage of each DataNode is balanced.