ALM-14007 NameNode Heap Memory Usage Exceeds the Threshold

Alarm Description

The system checks the HDFS NameNode Heap Memory usage every 30 seconds and compares the actual Heap memory usage with the threshold. The HDFS NameNode Heap Memory usage has a default threshold. This alarm is generated when the HDFS NameNode Heap Memory usage exceeds the threshold.

You can change the threshold in O&M > Alarm > Thresholds > Name of the desired cluster > HDFS.

When the Trigger Count is 1, this alarm is cleared when the HDFS NameNode Heap memory usage is less than or equal to the threshold. When the Trigger Count is greater than 1, this alarm is cleared when the HDFS NameNode Heap memory usage is less than or equal to 95% of the threshold.

Alarm Attributes

Alarm Parameters

Impact on the System

The HDFS NameNode Heap Memory usage is too high, which affects the data read/write performance of the HDFS.

Possible Causes

The HDFS NameNode Heap Memory is insufficient.

Handling Procedure

Delete unnecessary files.

1. Log in to the HDFS client as user root. Run cd to switch to the client installation directory, and run source bigdata_env.

   If the cluster uses the security mode, perform security authentication.

   Run the kinit hdfs command and enter the password as prompted. Obtain the password from the administrator.

2. Run the hdfs dfs -rm -r file or directory command to delete unnecessary files.
3. Check whether the alarm is cleared.
   • If yes, no further action is required.
   • If no, go to 4.

Check the NameNode JVM memory usage and configuration.

4. On the FusionInsight Manager portal, choose Cluster > Name of the desired cluster > Services > HDFS.
5. In the Basic Information area, click NameNode(Active) to go to the HDFS WebUI.
   

   By default, the admin user does not have the permissions to manage other components. If the page cannot be opened or the displayed content is incomplete when you access the native UI of a component due to insufficient permissions, you can manually create a user with the permissions to manage that component.

6. On the HDFS WebUI, click the Overview tab. In Summary, check the numbers of files, directories, and blocks in the HDFS.
7. On the FusionInsight Manager portal, choose Cluster > Name of the desired cluster > Services > HDFS > Configurations > All Configurations. In Search, enter GC_OPTS to check the GC_OPTS memory parameter of HDFS->NameNode.

Adjust the configuration in the system.

8. Check whether the memory is configured properly based on the number of files in 6 and the NameNode Heap Memory parameters in 7.
   • If yes, go to 9.
   • If no, go to 11.
   

   The recommended mapping between the number of HDFS file objects (filesystem objects = files + blocks) and the JVM parameters configured for NameNode is as follows:

   • If the number of file objects reaches 10,000,000, you are advised to set the JVM parameters as follows: -Xms6G -Xmx6G -XX:NewSize=512M -XX:MaxNewSize=512M
   • If the number of file objects reaches 20,000,000, you are advised to set the JVM parameters as follows: -Xms12G -Xmx12G -XX:NewSize=1G -XX:MaxNewSize=1G
   • If the number of file objects reaches 50,000,000, you are advised to set the JVM parameters as follows: -Xms32G -Xmx32G -XX:NewSize=3G -XX:MaxNewSize=3G
   • If the number of file objects reaches 100,000,000, you are advised to set the JVM parameters as follows: -Xms64G -Xmx64G -XX:NewSize=6G -XX:MaxNewSize=6G
   • If the number of file objects reaches 200,000,000, you are advised to set the JVM parameters as follows: -Xms96G -Xmx96G -XX:NewSize=9G -XX:MaxNewSize=9G
   • If the number of file objects reaches 300,000,000, you are advised to set the JVM parameters as follows: -Xms164G -Xmx164G -XX:NewSize=12G -XX:MaxNewSize=12G

9. Modify the heap memory parameters of the NameNode based on the mapping between the number of file objects and the memory. Click Save and choose Dashboard > More > Restart Service.
10. Check whether the alarm is cleared.
    • If yes, no further action is required.
    • If no, go to 11.

Collect fault information.

11. On the FusionInsight Manager portal, choose O&M > Log > Download.
12. Select the following nodes in the required cluster from the Service:
    • ZooKeeper
    • HDFS

13. Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
14. Contact the O&M engineers and send the collected logs.

Alarm Clearance

After the fault is rectified, the system automatically clears this alarm.

Related Information

None.