ALM-18015 JobHistoryServer Direct Memory Usage Exceeds the Threshold

Description

The system checks the direct memory usage of the MapReduce service every 30 seconds. This alarm is generated when the direct memory usage of a JobHistoryServer instance exceeds the threshold (90% of the maximum memory).

The alarm is cleared when the direct memory usage is less than the threshold.

Attribute

Parameters

Impact on the System

If the available direct memory of the MapReduce service is insufficient, a memory overflow occurs and the service breaks down.

Possible Causes

The direct memory of the JobHistoryServer instance is overused or the direct memory is inappropriately allocated.

Procedure

Check the direct memory usage.

1. On FusionInsight Manager, choose O&M > Alarm > Alarms > ALM-18015 JobHistory Direct Memory Usage Exceeds the Threshold > Location. View the IP address of the alarmed instance.
2. On FusionInsight Manager, choose Cluster > Name of the target cluster > Services > MapReduce and click the Instance tab. On the displayed page, select JobHistoryServer (the IP address of the instance for which the alarm is generated). Click the drop-down menu in the upper right corner of Chart, choose Customize > Resource, select Memory Usage Status of JobHistoryServer. Check the direct memory usage.
3. Check whether the used direct memory of MapReduce reaches 90% of the maximum direct memory specified for the MapReduce service.
   • If yes, go to Step 4.
   • If no, go to Step 9.

4. On the FusionInsight Manager portal, choose Cluster > Name of the desired cluster > Services > MapReduce > Configurations > All Configurations > JobHistoryServer > System. Check whether -XX:MaxDirectMemorySize exists in the GC_OPTS parameter.
   • If yes, go to Step 5.
   • If no, go to Step 7.

5. In the GC_OPTS parameter, delete "-XX:MaxDirectMemorySize".
6. Save the configuration and restart the JobHistoryServer instance.
   

   During the restart of JobHistoryServer, the status query of tasks such as Hive is affected, and the query result may be inaccurate.

7. Check whether the ALM-18009 Heap Memory Usage of JobHistoryServer Exceeds the Threshold exists.
   • If yes, handle the alarm by referring to ALM-18009 Heap Memory Usage of JobHistoryServer Exceeds the Threshold.
   • If no, go to Step 8.

8. Check whether the alarm is cleared.
   • If yes, no further action is required.
   • If no, go to Step 9.

Collect fault information.

9. On the FusionInsight Manager portal, choose O&M > Log > Download.
10. Select JobHistoryServer in the required cluster from the Service.
11. Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
12. Contact the O&M personnel and send the collected logs.

Alarm Clearing

After the fault is rectified, the system automatically clears this alarm.

Related Information

None