ALM-43006 Heap Memory Usage of the JobHistory Process Exceeds the Threshold (For MRS 2.x or Earlier)

Description

The system checks the JobHistory process status every 30 seconds. The alarm is generated when the heap memory usage of the JobHistory process exceeds the threshold (90% of the maximum memory).

Attribute

Parameters

Impact on the System

If the available JobHistory process heap memory is insufficient, a memory overflow occurs and the service breaks down.

Possible Causes

The heap memory of the JobHistory process is overused or the heap memory is inappropriately allocated.

Procedure

1. Check the heap memory usage.
   1. Go to the cluster details page and choose Alarms.
   2. Select the alarm whose Alarm ID is 43006 and view the IP address and role name of the instance in Location.
   3. Choose Components > Spark > Instance > JobHistory (IP address of the instance for which the alarm is generated) > Customize > Heap Memory Statistics of the JobHistory Process. Click OK to view the heap memory usage.
   4. Check whether the used heap memory of JobHistory reaches 90% of the maximum heap memory specified for JobHistory.
      • If yes, go to 1.e.
      • If no, go to 2.
   5. Choose Components > Spark > Service Configuration. Set Type to All and choose JobHistory > Default. Increase the value of SPARK_DAEMON_MEMORY as required.
   6. Click Save Configuration and select Restart the affected services or instances. Click OK.
   7. Check whether the alarm is cleared.
      • If yes, no further action is required.
      • If no, go to 2.

2. Collect fault information.
   1. On MRS Manager, choose System > Export Log.
   2. Contact the O&M engineers and send the collected logs.

Reference

None