ALM-14014 NameNode GC Time Exceeds the Threshold

Description

The system checks the garbage collection (GC) duration of the NameNode process every 60 seconds. This alarm is generated when the GC duration exceeds the threshold (12 seconds by default).

This alarm is cleared when the GC duration is less than the threshold.

Attribute

Parameters

Impact on the System

A long GC duration of the NameNode process may interrupt the services and users cannot read or write files.

Possible Causes

The heap memory of the NameNode instance is overused or the heap memory is inappropriately allocated. As a result, GCs occur frequently.

Procedure

Check the GC duration.

1. On the FusionInsight Manager portal, choose O&M > Alarm > Alarms. On the displayed interface, click the drop-down button of ALM-14014 NameNode GC Time Exceeds the Threshold. Then check the role name in Location and confirm the IP address of the instance.
2. On the Home page of FusionInsight Manager, choose Cluster > Name of the target cluster > Services > HDFS. On the page that is displayed, click the Instance tab. In the instance list, select NameNode (IP address of the instance for which this alarm is generated). Click the drop-down list in the upper right corner of the chart area, choose Customize > Garbage Collection, and select NameNode Garbage Collection (GC). Check the GC duration statistics of the NameNode process every minute.
   Figure 1 NameNode Garbage Collection (GC)
   

3. Check whether the GC duration of the NameNode process collected every minute exceeds the threshold (12 seconds by default).
   • If yes, go to Step 4.
   • If no, go to Step 7.

4. On FusionInsight Manager, choose Cluster, click the name of the desired cluster, and choose Services > HDFS > Configurations > All Configurations > NameNode > System. Increase the value of the GC_OPTS parameter as required.
   

   The recommended mapping between the number of HDFS file objects (filesystem objects = files + blocks) and the JVM parameters configured for NameNode is as follows:

   • If the number of file objects reaches 10,000,000, you are advised to set the JVM parameters as follows: -Xms6G -Xmx6G -XX:NewSize=512M -XX:MaxNewSize=512M
   • If the number of file objects reaches 20,000,000, you are advised to set the JVM parameters as follows: -Xms12G -Xmx12G -XX:NewSize=1G -XX:MaxNewSize=1G
   • If the number of file objects reaches 50,000,000, you are advised to set the JVM parameters as follows: -Xms32G -Xmx32G -XX:NewSize=3G -XX:MaxNewSize=3G
   • If the number of file objects reaches 100,000,000, you are advised to set the JVM parameters as follows: -Xms64G -Xmx64G -XX:NewSize=6G -XX:MaxNewSize=6G
   • If the number of file objects reaches 200,000,000, you are advised to set the JVM parameters as follows: -Xms96G -Xmx96G -XX:NewSize=9G -XX:MaxNewSize=9G
   • If the number of file objects reaches 300,000,000, you are advised to set the JVM parameters as follows: -Xms164G -Xmx164G -XX:NewSize=12G -XX:MaxNewSize=12G

5. Save the configuration and restart the NameNode instance.
   

   • During the restart of the active NameNode, a NameNode active/standby switchover occurs. As a result, no active node is available in the system for a short period of time (in the transition period of the active/standby switchover), and an alarm indicating that the HDFS service is unavailable may be generated. In addition, an error is reported during the running read and write tasks, but services are not interrupted.
   • Services are not affected after the standby NameNode is restarted.

6. Check whether the alarm is cleared.
   • If yes, no further action is required.
   • If no, go to Step 7.

Collect fault information.

7. On the FusionInsight Manager portal, choose O&M > Log > Download.
8. Select NameNode in the required cluster from the Service.
9. Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
10. Contact the O&M personnel and send the collected logs.

Alarm Clearing

After the fault is rectified, the system automatically clears this alarm.

Related Information

None