ALM-14006 Number of HDFS Files Exceeds the Threshold

Alarm Description

The system periodically checks the number of HDFS files every 30 seconds and compares the number of HDFS files with the threshold. This alarm is generated when the system detects that the number of HDFS files exceeds the threshold.

If Trigger Count is 1, this alarm is cleared when the number of HDFS files is less than or equal to the threshold. If Trigger Count is greater than 1, this alarm is cleared when the number of HDFS files is less than or equal to 90% of the threshold.

Alarm Attributes

Alarm Parameters

Impact on the System

Disk storage space is insufficient, which may result in data import failure. The performance of the HDFS system is affected.

Possible Causes

The number of HDFS files exceeds the threshold.

Handling Procedure

Check the number of files in the system.

1. On FusionInsight Manager, check the number of HDFS files. Specifically, choose Cluster > Name of the desired cluster > Services > HDFS. Click the drop-down menu in the upper right corner of Chart, choose Customize > File and Block, and select HDFS File and Total Blocks.
2. Choose Cluster > Name of the desired cluster > Services > HDFS > Configurations > All Configurations, and search for the GC_OPTS parameter under NameNode.
3. Configure the threshold of the number of configuration file objects. Specifically, change the value of Xmx (GB) in the GC_OPTS parameter. The threshold (specified by y) is calculated as follows: y = 0.2007 x Xmx - 0.6312, where x indicates the memory capacity Xmx (GB) and y indicates the number of files (unit: kW). Adjust the memory size as required.
4. Confirm that the value of GC_PROFILE is custom so that the GC_OPTS configuration takes effect. Click Save and choose More > Restart Instance to restart the service.
5. Check whether the alarm is cleared.
   • If yes, no further action is required.
   • If no, go to 6.

Check whether needless files exist in the system.

6. Log in to the HDFS client as user root. Run cd to switch to the client installation directory, and run source bigdata_env to configure the environment variables.

   If the cluster uses the security mode, perform security authentication.

   Run the kinit hdfs command and enter the password as prompted. Obtain the password from the MRS cluster administrator.

7. Run hdfs dfs -ls file or directory to check whether the files in the directory can be deleted.
   • If yes, go to 8.
   • If no, go to 9.

8. Run the hdfs dfs -rm -r file or directory path command. After deleting unnecessary files, wait until the files are retained in the recycle bin for a period longer than the value of fs.trash.interval on the NameNode. Then check whether the alarm is cleared.
   

   Deleting a file or folder is a high-risk operation. Ensure that the file or folder is no longer required before performing this operation.

   • If yes, no further action is required.
   • If no, go to 9.

Collect the fault information.

9. On FusionInsight Manager, choose O&M > Log > Download.
10. Expand the drop-down list next to the Service field. In the Services dialog box that is displayed, select HDFS for the target cluster.
11. Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
12. Contact O&M personnel and provide the collected logs.

Alarm Clearance

This alarm is automatically cleared after the fault is rectified.

Related Information

Configuration rules of the NameNode JVM parameter

Default value of the NameNode JVM parameter GC_OPTS:

-Xms2G -Xmx4G -XX:NewSize=128M -XX:MaxNewSize=256M -XX:MetaspaceSize=128M -XX:MaxMetaspaceSize=128M -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=65 -XX:+PrintGCDetails -Dsun.rmi.dgc.client.gcInterval=0x7FFFFFFFFFFFFFE -Dsun.rmi.dgc.server.gcInterval=0x7FFFFFFFFFFFFFE -XX:-OmitStackTraceInFastThrow -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=1M -Djdk.tls.ephemeralDHKeySize=3072 -Djdk.tls.rejectClientInitiatedRenegotiation=true -Djava.io.tmpdir=${Bigdata_tmp_dir}