ALM-43009 JobHistory Process GC Duration Exceeds the Threshold

Alarm Description

The system checks the GC duration of JobHistory every 60 seconds. A major alarm is reported when the GC duration exceeds 12 seconds for three consecutive times. A minor alarm is reported when the duration exceeds 9.6 seconds for three consecutive times. To change the threshold, choose O&M > Alarm > Thresholds > Name of the desired cluster > Spark > GC Time > JobHistory Total GC time. This alarm is cleared when the JobHistory GC duration is shorter than or equal to the threshold.

Alarm Attributes

Alarm Parameters

Impact on the System

The process performance deteriorates, and the process can even be unavailable. Historical execution records of Spark tasks cannot be queried.

Possible Causes

The heap memory of the JobHistory process is overused or inappropriately allocated, causing frequent occurrence of the GC process.

Handling Procedure

Check the GC duration.

1. On FusionInsight Manager, choose O&M > Alarm > Alarms and select the alarm whose ID is 43009. Check the role name and the IP address of the host where the alarm is generated in Location.
2. On FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > Spark > Instance and click the JobHistory for which the alarm is generated to enter the Dashboard page. Click the drop-down menu in the Chart area and choose Customize > GC Time > Garbage Collection (GC) Time of JobHistory from the drop-down list box in the upper right corner and click OK to check whether the GC duration is longer than the threshold (default value: 12 seconds).
   • If yes, go to 3.
   • If no, go to 6.

3. On FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > Spark > Configurations, click All Configurations, and select Default. Increase the value of SPARK_DAEMON_MEMORY (4 GB by default) by 0.5 times if this alarm is generated occasionally. Double the value if the alarm is reported frequently.
4. Restart all JobHistory instances.
5. Check whether the alarm is cleared 10 minutes later.
   • If yes, no further action is required.
   • If no, go to 6.

Collect fault Information.

6. On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
7. Expand the Service drop-down list, and select Spark for the target cluster.
8. Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
9. Contact O&M engineers and provide the collected logs.

Alarm Clearance

This alarm is automatically cleared after the fault is rectified.

Related Information

None.