ALM-43010 Heap Memory Usage of the JDBCServer Process Exceeds the Threshold

Alarm Description

The system checks the heap memory usage of the JDBCServer processes every 30 seconds.

A major alarm is reported when the usage is greater than 85% and less than 95% of the maximum memory. A critical alarm is reported when the usage is greater than the 95% of the maximum memory.

Alarm Attributes

Alarm Parameters

Impact on the System

If the heap memory usage of the JDBCServer process is too high, the performance deteriorates, and even memory overflow occurs. As a result, the JDBCServer process is unavailable, and Spark JDBC tasks are slow or fail to run.

Possible Causes

The heap memory of the JDBCServer process is overused or the heap memory is inappropriately allocated.

Handling Procedure

Check the heap memory usage.

1. On FusionInsight Manager, choose O&M > Alarm > Alarms and select the alarm whose ID is 43010. Check the role name and the IP address of the host where the alarm is generated in Location.
2. On FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > Spark > Instance. Click the JDBCServer that reported the alarm to go to the Dashboard page. Click the drop-down list in the upper right corner of the chart area, and choose Customize > JDBCServer Memory Usage Statistics > OK. Check whether the heap memory used by the process reaches the maximum heap memory threshold (95% by default).
   • If the threshold is reached, go to 3.
   • If the threshold is not reached, go to 7.

3. On FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > Spark > Instance. Click the JDBCServer that reported the alarm to go to the Dashboard page. Click the drop-down list in the upper right corner of the chart area, and choose Customize > Statistics for the heap memory of the JDBCServer process > OK. Based on the alarm generation time, check the values of the used heap memory of the process in the corresponding period and obtain the maximum value.
4. On FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > Spark > Configurations > All Configurations > JDBCServer > Performance. The default value of the SPARK_DRIVER_MEMORY parameter is 4 GB. You can change the value based on the ratio of the maximum heap memory used by the JDBCServer process to the threshold specified by JDBCServer Heap Memory Usage Statistics (JDBCServer) in the alarm period. If the alarm persists after the parameter value is changed, increase the value by 0.5 times. If the alarm is generated frequently, increase the value by one time. In the case of large service volume and high service concurrency, you are advised to add instances.
   

   On FusionInsight Manager, you can choose O&M > Alarm > Thresholds > Name of the desired cluster > Spark > Memory > JDBCServer Heap Memory Usage Statistics (JDBCServer) to view the threshold.

5. Restart all JDBCServer instances.
6. Check whether the alarm is cleared 10 minutes later.
   • If yes, no further action is required.
   • If the threshold is not reached, go to 7.

Collect fault Information.

7. On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
8. Expand the Service drop-down list, and select Spark for the target cluster.
9. Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
10. Contact O&M engineers and provide the collected logs.

Alarm Clearance

This alarm is automatically cleared after the fault is rectified.

Related Information

None.