ALM-18010 ResourceManager GC Time Exceeds the Threshold

Alarm Description

The system checks the garbage collection (GC) duration of the ResourceManager process every 60 seconds. This alarm is generated when the GC duration exceeds the threshold.

This alarm is cleared when the GC duration is less than the threshold.

Alarm Attributes

Alarm Parameters

Impact on the System

A long GC duration of the ResourceManager process may interrupt the services.

Possible Causes

The heap memory of the ResourceManager instance is overused or the heap memory is inappropriately allocated. As a result, GCs occur frequently.

Handling Procedure

Check the GC duration.

1. On Manager, choose O&M > Alarm > Alarms to view details about the current alarm and record the IP address of the instance for which the alarm is reported.

   For details about how to log in to FusionInsight Manager, see Accessing MRS Manager.

2. On Manager, choose Cluster > Services > Yarn > Instances > ResourceManager (the instance for which this alarm is generated). Click the drop-down list in the upper right corner of the Chart area, choose Customize > Garbage Collection, and select Garbage Collection (GC) Time of ResourceManager. Check the GC duration statistics of the ResourceManager process every minute.
   Figure 1 Garbage Collection (GC) Time of ResourceManager
   

3. Check whether the GC duration of the ResourceManager process collected every minute exceeds the threshold (12 seconds by default).
   • If yes, go to Step 4.
   • If no, go to Step 7.

4. On Manager, choose Cluster > Services > Yarn > Configurations > All Configurations > ResourceManager > System. Increase the value of the GC_OPTS parameter as required.

   Table 1 Mapping between the number of NodeManager instances and the ResourceManager memory size

   Number of NodeManager Instances in a Cluster	Recommended JVM Parameter Settings for ResourceManager Instances
   100	-Xms4G -Xmx4G -XX:NewSize=512M -XX:MaxNewSize=1G
   200	-Xms6G -Xmx6G -XX:NewSize=512M -XX:MaxNewSize=1G
   500	-Xms10G -Xmx10G -XX:NewSize=1G -XX:MaxNewSize=2G
   1000	-Xms20G -Xmx20G -XX:NewSize=1G -XX:MaxNewSize=2G
   2000	-Xms40G -Xmx40G -XX:NewSize=2G -XX:MaxNewSize=4G
   3000	-Xms60G -Xmx60G -XX:NewSize=2G -XX:MaxNewSize=4G
   4000	-Xms80G -Xmx80G -XX:NewSize=2G -XX:MaxNewSize=4G
   5000	-Xms100G -Xmx100G -XX:NewSize=3G -XX:MaxNewSize=6G

5. Save the configuration and restart the ResourceManager instance.
   

   • Restarting the active ResourceManager instance will trigger a active/standby switchover of the ResourceManager instance. During the switchover, Yarn cannot submit new jobs, but the submitted jobs are not affected. The Yarn component and components that depend on Yarn generate service unavailable alarms for a short period of time.
   • Restart the standby ResourceManager instance. Services are not affected.

6. Check whether the alarm is cleared 5 minutes later.
   • If yes, no further action is required.
   • If no, go to Step 7.

Collect fault information.

7. On Manager, choose O&M > Log > Download.
8. Expand the Service drop-down list, and select ResourceManager for the target cluster.
9. Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes before and after the alarm generation time, respectively. Then, click Download.
10. Send the collected fault logs to O&M personnel for help.

Alarm Clearance

This alarm is automatically cleared after the fault is rectified.

Related Information

None