ALM-14006 Number of HDFS Files Exceeds the Threshold

Updated on 2024-11-13 GMT+08:00

View PDF

Alarm Description

The system periodically checks the number of HDFS files every 30 seconds and compares the number of HDFS files with the threshold. This alarm is generated when the system detects that the number of HDFS files exceeds the threshold.

If Trigger Count is 1, this alarm is cleared when the number of HDFS files is less than or equal to the threshold. If Trigger Count is greater than 1, this alarm is cleared when the number of HDFS files is less than or equal to 90% of the threshold.

Alarm Attributes

Alarm ID	Alarm Severity	Auto Cleared
14006	Minor (versions earlier than MRS 3.3.1) Major (MRS 3.3.1 and later versions)	Yes

Alarm ID

Alarm Severity

Auto Cleared

14006

Minor (versions earlier than MRS 3.3.1)

Major (MRS 3.3.1 and later versions)

Yes

Alarm Parameters

Parameter	Description
Source	Specifies the cluster for which the alarm was generated.
ServiceName	Specifies the service for which the alarm was generated.
RoleName	Specifies the role for which the alarm was generated.
HostName	Specifies the host for which the alarm was generated.
NameServiceName	Specifies the NameService for which the alarm was generated.
Trigger Condition	Specifies the threshold for triggering the alarm.

Impact on the System

If there are too many HDFS files, the HDFS system may respond slowly or the disk space may be used up.

Possible Causes

The number of HDFS files exceeds the threshold.

Handling Procedure

Check the number of files in the system.

On FusionInsight Manager, check the number of HDFS files. Specifically, choose Cluster > Name of the desired cluster > Services > HDFS. Click the drop-down menu in the upper right corner of Chart, choose Customize > File and Block, and select HDFS File and Total Blocks.
Choose Cluster > Name of the desired cluster > Services > HDFS > Configurations > All Configurations, and search for the GC_OPTS parameter under NameNode.
Configure the threshold of the number of configuration file objects. Specifically, change the value of Xmx (GB) in the GC_OPTS parameter. The threshold (specified by y) is calculated as follows: y = 0.2007 x Xmx - 0.6312, where x indicates the memory capacity Xmx (GB) and y indicates the number of files (unit: kW). Adjust the memory size as required.
Check that the value of GC_PROFILE is custom so that the GC_OPTS configuration can take effect. Click Save, click More, and select Restart Service to restart the service.

NOTE:

The service is unavailable during the restart. Upper-layer services that depend on the service are also affected.
Check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 6.

Check whether needless files exist in the system.

Log in to the HDFS client as user root. Run cd to switch to the client installation directory, and run source bigdata_env to configure the environment variables.

If the cluster uses the security mode, perform security authentication.

Run the kinit hdfs command and enter the password as prompted. Obtain the password from the MRS cluster administrator.
Run hdfs dfs -ls file or directory to check whether the files in the directory can be deleted.
- If yes, go to 8.
- If no, go to 9.
Run the hdfs dfs -rm -r file or directory path command. After deleting unnecessary files, wait until the files are retained in the recycle bin for a period longer than the value of fs.trash.interval on the NameNode. Then check whether the alarm is cleared.

NOTE:

Deleting a file or folder is a high-risk operation. Ensure that the file or folder is no longer required before performing this operation.
- If yes, no further action is required.
- If no, go to 9.

Collect fault information.

On FusionInsight Manager, choose O&M > Log > Download.
Expand the drop-down list next to the Service field. In the Services dialog box that is displayed, select HDFS for the target cluster.
Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
Contact O&M personnel and provide the collected logs.

Alarm Clearance

This alarm is automatically cleared after the fault is rectified.

Related Information

Configuration rules of the NameNode JVM parameter

Default value of the NameNode JVM parameter GC_OPTS:

-Xms2G -Xmx4G -XX:NewSize=128M -XX:MaxNewSize=256M -XX:MetaspaceSize=128M -XX:MaxMetaspaceSize=128M -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=65 -XX:+PrintGCDetails -Dsun.rmi.dgc.client.gcInterval=0x7FFFFFFFFFFFFFE -Dsun.rmi.dgc.server.gcInterval=0x7FFFFFFFFFFFFFE -XX:-OmitStackTraceInFastThrow -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=1M -Djdk.tls.ephemeralDHKeySize=3072 -Djdk.tls.rejectClientInitiatedRenegotiation=true -Djava.io.tmpdir=${Bigdata_tmp_dir}

The number of NameNode files is proportional to the used memory size of the NameNode. When file objects change, you need to change -Xms2G -Xmx4G -XX:NewSize=128M -XX:MaxNewSize=256M in the default value. The following table lists the reference values.

**Table 1** NameNode JVM configuration
Number of File Objects	Reference Value
10,000,000	-Xms6G -Xmx6G -XX:NewSize=512M -XX:MaxNewSize=512M
20,000,000	-Xms12G -Xmx12G -XX:NewSize=1G -XX:MaxNewSize=1G
50,000,000	-Xms32G -Xmx32G -XX:NewSize=3G -XX:MaxNewSize=3G
100,000,000	-Xms64G -Xmx64G -XX:NewSize=6G -XX:MaxNewSize=6G
200,000,000	-Xms96G -Xmx96G -XX:NewSize=9G -XX:MaxNewSize=9G
300,000,000	-Xms164G -Xmx164G -XX:NewSize=12G -XX:MaxNewSize=12G