ALM-18011 NodeManager GC Time Exceeds the Threshold

Alarm Description

The system checks the garbage collection (GC) duration of the NodeManager process every 60 seconds. This alarm is generated when the GC duration exceeds the threshold.

This alarm is cleared when the GC duration is less than the threshold.

Alarm Attributes

Alarm Parameters

Impact on the System

A long GC duration of the NodeManager process may interrupt the services.

Possible Causes

The heap memory of the NodeManager instance is overused or the heap memory is inappropriately allocated. As a result, GCs occur frequently.

Handling Procedure

Check the GC duration.

1. On Manager, choose O&M > Alarm > Alarms to view the alarm details and record the IP address of the instance for which the alarm is generated.

   For details about how to log in to FusionInsight Manager, see Accessing MRS Manager.

2. On Manager, choose Cluster > Services > Yarn > Instances > NodeManager (the instance for which this alarm is generated). Click the drop-down list in the upper right corner of the Chart area, choose Customize > Garbage Collection, and select Garbage Collection (GC) Time of NodeManager. Check the GC duration statistics of the NodeManager process every minute.
   Figure 1 Garbage Collection (GC) Time of NodeManager
   

3. Check whether the GC duration of the NodeManager process collected every minute exceeds the threshold (12 seconds by default).
   • If yes, go to Step 4.
   • If no, go to Step 7.

4. On Manager, choose Cluster > Services > Yarn > Configurations > All Configurations > NodeManager > System. Increase the value of the GC_OPTS parameter as required.
   

   The mapping between the number of NodeManager instances in a cluster and the memory size of NodeManager is as follows:

   • If there are more than 100 NodeManager instances in the cluster, you are advised to set the JVM parameter of NodeManager instances to -Xms2G -Xmx4G -XX:NewSize=512M -XX:MaxNewSize=1G.
   • If there are more than 200 NodeManager instances in the cluster, you are advised to set the JVM parameter of NodeManager instances to -Xms4G -Xmx4G -XX:NewSize=512M -XX:MaxNewSize=1G.
   • If there are more than 500 NodeManager instances in the cluster, you are advised to set the JVM parameter of NodeManager instances to -Xms8G -Xmx8G -XX:NewSize=1G -XX:MaxNewSize=2G.

5. Save the configuration and restart the NodeManager instance.
   

   During NodeManager restart, containers submitted to this node may be retried to other nodes.

6. Check whether the alarm is cleared 5 minutes later.
   • If yes, no further action is required.
   • If no, go to Step 7.

Collect fault information.

7. On Manager, choose O&M > Log > Download.
8. Expand the Service drop-down list, and select NodeManager for the target cluster.
9. Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes before and after the alarm generation time, respectively. Then, click Download.
10. Send the collected fault logs to O&M personnel for help.

Alarm Clearance

This alarm is automatically cleared after the fault is rectified.

Related Information

None