ALM-18009 Heap Memory Usage of JobHistoryServer Exceeds the Threshold

Alarm Description

The system checks the heap memory usage of MapReduce JobHistoryServer every 30 seconds and compares the actual usage with the threshold. This alarm is generated when the heap memory usage of MapReduce JobHistoryServer exceeds the threshold.

Users can choose O&M > Alarm > Thresholds > Name of the desired cluster > MapReduce to change the threshold.

When the Trigger Count is 1, this alarm is cleared when the heap memory usage of MapReduce JobHistoryServer is less than or equal to the threshold. When the Trigger Count is greater than 1, this alarm is cleared when the heap memory usage of MapReduce JobHistoryServer is less than or equal to 95% of the threshold.

Alarm Attributes

Alarm Parameters

Impact on the System

When the heap memory usage of MapReduce JobHistoryServer is overhigh, the performance of MapReduce log archiving is affected. What is more, a memory overflow occurs so that the YARN service is unavailable.

Possible Causes

The heap memory of the MapReduce JobHistoryServer instance on the node is overused or the heap memory is inappropriately allocated. As a result, the usage exceeds the threshold.

Handling Procedure

Check the memory usage.

1. On Manager, choose O&M > Alarm > Alarms to view details about the current alarm and record the host name of the instance for which the alarm is generated.

   For details about how to log in to FusionInsight Manager, see Accessing MRS Manager.

2. On Manager, choose Cluster > Services > Mapreduce and click the Instances tab. On the displayed page, select JobHistoryServer (the host name of the instance for which the alarm is generated) and select Customize > Resource from the drop-down list in the upper right corner of the Chart area. On the displayed page, select JobHistoryServer Heap Memory Usage Statistics. Check the heap memory usage.
3. Check whether the used heap memory of JobHistoryServer reaches 95% of the maximum heap memory specified for JobHistoryServer.
   • If yes, go to Step 4.
   • If no, go to Step 6.

4. On Manager, choose Cluster > Services > Mapreduce > Configurations > All Configurations > JobHistoryServer > System. Increase the value of GC_OPTS parameter as required, click Save. Click OK to restart the JobHistoryServer.
   

   • The mapping between the number of historical tasks (10000) and the memory of the JobHistoryServer is as follows:

     -Xms30G -Xmx30G -XX:NewSize=1G -XX:MaxNewSize=2G

   • During the restart of JobHistoryServer, the status of tasks such as Hive cannot be queried. As a result, the query result may be inaccurate.

5. Check whether the alarm is cleared 5 minutes later.
   • If yes, no further action is required.
   • If no, go to Step 6.

Collect fault information.

6. On Manager, choose O&M > Log > Download.
7. In the Service area, select the following nodes of the desired cluster.
   • NodeAgent
   • MapReduce

8. Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes before and after the alarm generation time, respectively. Then, click Download.
9. Send the collected fault logs to O&M personnel for help.

Alarm Clearance

This alarm is automatically cleared after the fault is rectified.

Related Information

None