ALM-18017 Non Heap Memory Usage of NodeManager Exceeds the Threshold

Alarm Description

The system checks the Non Heap memory usage of Yarn NodeManager every 30 seconds and compares the actual usage with the threshold. The alarm is generated when the Non Heap memory usage of Yarn NodeManager exceeds the threshold.

Users can choose O&M > Alarm > Thresholds > Name of the desired cluster > Yarn to change the threshold.

The alarm is cleared when the Non Heap memory usage is less than or equal to the threshold.

Alarm Attributes

Alarm Parameters

Impact on the System

When the Non Heap memory usage of Yarn NodeManager is overhigh, the performance of Yarn task submission and operation is affected. In addition, a memory overflow may occur so that the Yarn service is unavailable.

Possible Causes

The Non Heap memory of the Yarn NodeManager instance on the node is overused or the Non Heap memory is inappropriately allocated. As a result, the usage exceeds the threshold.

Handling Procedure

Check the Non Heap memory usage.

1. On the FusionInsight Manager portal, choose O&M > Alarm > Alarms > ALM-18017 Non Heap Memory Usage of Yarn NodeManager Exceeds the Threshold > Location. Check the HostName of the instance for which the alarm is generated.
2. On the FusionInsight Manager portal, choose Cluster > Name of the desired cluster > Services > Yarn > Instance > NodeManager. Click the drop-down menu in the upper right corner of Chart, choose Customize > Resource > Percentage of Used Memory of the NodeManager. NodeManager indicates the corresponding HostName of the instance for which the alarm is generated. Check the Non Heap memory usage.
3. Check whether the used Non Heap memory of NodeManager reaches 90% of the maximum Non Heap memory specified for NodeManager by default.
   • If yes, go to 4.
   • If no, go to 6.

4. On the FusionInsight Manager portal, choose Cluster > Name of the desired cluster > Services > Yarn > Configurations > All Configurations > NodeManager > System. Adjust the GC_OPTS memory parameter of NodeManager, click Save, and click OK, and restart the role instance.
   

   The mapping between the number of NodeManager instances in a cluster and the memory size of NodeManager is as follows:

   • If the number of NodeManager instances in the cluster reaches 100, the recommended JVM parameters for NodeManager instances are as follows: -Xms2G -Xmx4G -XX:NewSize=512M -XX:MaxNewSize=1G
   • If the number of NodeManager instances in the cluster reaches 200, the recommended JVM parameters for NodeManager instances are as follows: -Xms4G -Xmx4G -XX:NewSize=512M -XX:MaxNewSize=1G
   • If the number of NodeManager instances in the cluster reaches 500, the recommended JVM parameters for NodeManager instances are as follows: -Xms8G -Xmx8G -XX:NewSize=1G -XX:MaxNewSize=2G

5. Check whether the alarm is cleared.
   • If yes, no further action is required.
   • If no, go to 6.

Collect fault information.

6. On the FusionInsight Manager portal, choose O&M > Log > Download.
7. Select the following node in the required cluster from the Service.
   • NodeAgent
   • Yarn

8. Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
9. Contact the O&M engineers and send the collected logs.

Alarm Clearance

After the fault is rectified, the system automatically clears this alarm.

Related Information

None.