ALM-18009 Heap Memory Usage of JobHistoryServer Exceeds the Threshold

Alarm Description

The system checks the heap memory usage of Mapreduce JobHistoryServer every 30 seconds and compares the actual usage with the threshold. The alarm is generated when the heap memory usage of Mapreduce JobHistoryServer exceeds the threshold.

Users can choose O&M > Alarm > Thresholds > Name of the desired cluster > MapReduce to change the threshold.

When the Trigger Count is 1, this alarm is cleared when the heap memory usage of MapReduce JobHistoryServer is less than or equal to the threshold. When the Trigger Count is greater than 1, this alarm is cleared when the heap memory usage of MapReduce JobHistoryServer is less than or equal to the threshold.

Alarm Attributes

Alarm Parameters

Impact on the System

When the heap memory usage of Mapreduce JobHistoryServer is overhigh, the performance of Mapreduce log archiving is affected. In addition, a memory overflow may occur so that the Yarn service is unavailable.

Possible Causes

The heap memory of the Mapreduce JobHistoryServer instance on the node is overused or the heap memory is inappropriately allocated. As a result, the usage exceeds the threshold.

Handling Procedure

Check the memory usage.

1. On the FusionInsight Manager portal, choose O&M > Alarm > Alarms > ALM-18009 Heap Memory Usage of MapReduce JobHistoryServer Exceeds the Threshold > Location. Check the HostName of the instance for which the alarm is generated.
2. On the FusionInsight Manager portal, choose Cluster > Name of the desired cluster > Services > Mapreduce > Instance > JobHistoryServer. Click the drop-down menu in the upper right corner of Chart, choose Customize > JobHistoryServer heap memory usage statistics. JobHistoryServer indicates the corresponding HostName of the instance for which the alarm is generated. Check the heap memory usage.
3. Check whether the used heap memory of JobHistoryServer reaches 95% of the maximum heap memory specified for JobHistoryServer.
   • If yes, go to 4.
   • If no, go to 6.

4. On the FusionInsight Manager portal, choose Cluster > Name of the desired cluster > Services > Mapreduce > Configurations > All Configurations > JobHistoryServer > System. Increase the value of GC_OPTS parameter as required, click Save. Click OK and restart the role instance.
   

   The mapping between the number of historical tasks (10000) and the memory of JobHistoryServer is as follows:

   -Xms30G -Xmx30G -XX:NewSize=1G -XX:MaxNewSize=2G

5. Check whether the alarm is cleared.
   • If yes, no further action is required.
   • If no, go to 6.

Collect fault information.

6. On the FusionInsight Manager portal, choose O&M > Log > Download.
7. Select the following node in the required cluster from the Service.
   • NodeAgent
   • Mapreduce

8. Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
9. Contact the O&M engineers and send the collected logs.

Alarm Clearance

After the fault is rectified, the system automatically clears this alarm.

Related Information

None.