Help Center/ MapReduce Service/ User Guide/ MRS Cluster O&M/ MRS Cluster Alarm Handling Reference/ ALM-18008 Heap Memory Usage of ResourceManager Exceeds the Threshold
Updated on 2024-09-23 GMT+08:00
View PDF
Share

ALM-18008 Heap Memory Usage of ResourceManager Exceeds the Threshold

Description

The system checks the heap memory usage of Yarn ResourceManager every 30 seconds and compares the actual usage with the threshold. The alarm is generated when the heap memory usage of Yarn ResourceManager exceeds the threshold (95% of the maximum memory by default).

Users can choose O&M > Alarm > Thresholds > Name of the desired cluster > Yarn to change the threshold.

When the Trigger Count is 1, this alarm is cleared when the heap memory usage of Yarn ResourceManager is less than or equal to the threshold. When the Trigger Count is greater than 1, this alarm is cleared when the heap memory usage of Yarn ResourceManager is less than or equal to 95% of the threshold.

Attribute

Alarm ID

Alarm Severity

Automatically Cleared

18008

Major

Yes

Parameters

Name

Meaning

Source

Specifies the cluster for which the alarm is generated.

ServiceName

Specifies the service name for which the alarm is generated.

RoleName

Specifies the role name for which the alarm is generated.

HostName

Specifies the object (host ID) for which the alarm is generated.

Trigger Condition

Specifies the threshold triggering the alarm. If the current indicator value exceeds this threshold, the alarm is generated.

Impact on the System

When the heap memory usage of Yarn ResourceManager is overhigh, the performance of Yarn task submission and operation is affected. In addition, a memory overflow may occur so that the Yarn service is unavailable.

Possible Causes

The heap memory of the Yarn ResourceManager instance on the node is overused or the heap memory is inappropriately allocated. As a result, the usage exceeds the threshold.

Procedure

Check the heap memory usage.

  1. On the FusionInsight Manager portal, choose O&M > Alarm > Alarms > Heap Memory Usage of Yarn ResourceManager Exceeds the Threshold > Location. Check the HostName of the instance for which the alarm is generated.
  2. On the FusionInsight Manager portal, choose Cluster > Name of the desired cluster > Services > Yarn > Instance > ResourceManager (Indicates the host name of the instance for which the alarm is generated). Click the drop-down menu in the upper right corner of Chart, choose Customize > ResourceManager > Percentage of Used Memory of the ResourceManager. Check the heap memory usage.

    Figure 1 Percentage of Used Memory of the ResourceManager

  3. Check whether the used heap memory of ResourceManager reaches 95% of the maximum heap memory specified for ResourceManager.

    • If yes, go to 4.
    • If no, go to 6.

  4. On the FusionInsight Manager portal, choose Cluster > Name of the desired cluster > Services > Yarn > Configurations > All Configurations > ResourceManager > System. Increase the value of GC_OPTS parameter as required, click Save. Restart the role instance.

    • Restarting the active ResourceManager instance will trigger a active/standby switchover of the ResourceManager instance. During the switchover, Yarn cannot submit new jobs, but submitted jobs are not affected. The Yarn component and components that depend on Yarn generate temporary service unavailability alarms.

      Restart the standby ResourceManager instance. Services are not affected.

    • The mapping between the number of NodeManager instances in a cluster and the memory size of ResourceManager is as follows:
      • If the number of NodeManager instances in the cluster reaches 100, the recommended JVM parameters of the ResourceManager instance are as follows: -Xms4G -Xmx4G -XX:NewSize=512M -XX:MaxNewSize=1G
      • If the number of NodeManager instances in the cluster reaches 200, the recommended JVM parameters of the ResourceManager instance are as follows: -Xms6G -Xmx6G -XX:NewSize=512M -XX:MaxNewSize=1G
      • If the number of NodeManager instances in the cluster reaches 500, the recommended JVM parameters of the ResourceManager instance are as follows: -Xms10G -Xmx10G -XX:NewSize=1G -XX:MaxNewSize=2G
      • If the number of NodeManager instances in the cluster reaches 1000, the recommended JVM parameters of the ResourceManager instance are as follows: -Xms20G -Xmx20G -XX:NewSize=1G -XX:MaxNewSize=2G
      • If the number of NodeManager instances in the cluster reaches 2000, the recommended JVM parameters of the ResourceManager instance are as follows: -Xms40G -Xmx40G -XX:NewSize=2G -XX:MaxNewSize=4G
      • If the number of NodeManager instances in the cluster reaches 3000, the recommended JVM parameters of the ResourceManager instance are as follows: -Xms60G -Xmx60G -XX:NewSize=2G -XX:MaxNewSize=4G
      • If the number of NodeManager instances in the cluster reaches 4000, the recommended JVM parameters of the ResourceManager instance are as follows: -Xms80G -Xmx80G -XX:NewSize=2G -XX:MaxNewSize=4G
      • If the number of NodeManager instances in the cluster reaches 5000, the recommended JVM parameters of the ResourceManager instance are as follows: -Xms100G -Xmx100G -XX:NewSize=3G -XX:MaxNewSize=6G

  5. Check whether the alarm is cleared.

    • If yes, no further action is required.
    • If no, go to 6.

Collect fault information.

  1. On the FusionInsight Manager portal, choose O&M > Log > Download.
  2. Select the following node in the required cluster from the Service.

    • NodeAgent
    • Yarn

  3. Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
  4. Contact the O&M personnel and send the collected logs.

Alarm Clearing

After the fault is rectified, the system automatically clears this alarm.

Related Information

None