ALM-43007 Non-Heap Memory Usage of the JobHistory Process Exceeds the Threshold

Alarm Description

The system checks the non-heap memory usage of the JobHistory process every 30 seconds. A major alarm is reported when the non-heap memory usage is greater than 95% of the maximum memory. A minor alarm is reported when the usage is greater than 85% and less than 95% of the maximum memory.

Alarm Attributes

Alarm Parameters

Impact on the System

If the non-heap memory usage of the JobHistory process is too high, the performance deteriorates, and the process even becomes unavailable due to memory overflow. When it is unavailable, execution records of Spark tasks cannot be queried.

Possible Causes

The non-heap memory of the JobHistory process is overused or the non-heap memory is inappropriately allocated.

Handling Procedure

Check non-heap memory usage.

1. On FusionInsight Manager, choose O&M > Alarm > Alarms and select the alarm whose ID is 43007. Check the role name and the IP address of the host where the alarm is generated in Location.
2. On FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > Spark > Instance. Click the JobHistory that reported the alarm to go to the Dashboard page. Click the drop-down list in the upper right corner of the chart area, and choose Customize > Memory > JobHistory Memory Usage Statistics > OK. Check whether the non-heap memory used by the process reaches the maximum non-heap memory threshold (95% by default).
   • If yes, go to 3.
   • If no, go to 7.

3. On FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > Spark > Instance. Click the JobHistory that reported the alarm to go to the Dashboard page. Click the drop-down list in the upper right corner of the chart area, and choose Customize > Memory > Statistics for the non-heap memory of the JobHistory process > OK. Based on the alarm generation time, check the values of the used non-heap memory of the process in the corresponding period and obtain the maximum value.
4. On FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > Spark > Configurations, click All Configurations, select Default, and change the value of -XX:MaxMetaspaceSize in the SPARK_DAEMON_JAVA_OPTS parameter based on the ratio of the maximum non-heap memory usage to the threshold specified by JobHistory Non-Heap Memory Usage Statistics (JobHistory) in the alarm period.
   

   On FusionInsight Manager, you can choose O&M > Alarm > Thresholds > Name of the desired cluster > Spark > Memory > Spark Non-Heap Memory Usage Statistics (Spark) to view the threshold.

5. Restart all JobHistory instances.
6. Check whether the alarm is cleared 10 minutes later.
   • If yes, no further action is required.
   • If no, go to 7.

Collect fault Information.

7. On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
8. Expand the Service drop-down list, and select Spark for the target cluster.
9. Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
10. Contact O&M engineers and provide the collected logs.

Alarm Clearance

This alarm is automatically cleared after the fault is rectified.

Related Information

None.