Help Center/ MapReduce Service/ User Guide/ MRS Cluster O&M/ MRS Cluster Alarm Handling Reference/ ALM-43017 JDBCServer2x Process Full GC Number Exceeds the Threshold
Updated on 2024-09-23 GMT+08:00
View PDF
Share

ALM-43017 JDBCServer2x Process Full GC Number Exceeds the Threshold

Description

The system checks the number of Full garbage collection (GC) times of the JDBCServer2x process every 60 seconds. This alarm is generated when the detected Full GC number exceeds the threshold (exceeds 12 for three consecutive checks.) You can change the threshold by choosing O&M > Alarm > Thresholds > Spark2x > GC number > Full GC Number of JDBCServer2x. This alarm is cleared when the Full GC number of the JDBCServer2x process is less than or equal to the threshold.

In MRS 3.3.0-LTS and later versions, the Spark2x component is renamed Spark, and the role names in the component are also changed. For example, JDBCServer2x is changed to JDBCServer. Refer to the descriptions and operations related to the component name and role names in the document based on your MRS version.

Attribute

Alarm ID

Alarm Severity

Auto Clear

43017

Major

Yes

Parameters

Name

Description

Source

Specifies the cluster for which the alarm is generated.

ServiceName

Specifies the service for which the alarm is generated.

RoleName

Specifies the role for which the alarm is generated.

HostName

Specifies the host for which the alarm is generated.

Trigger Condition

Specifies the threshold for triggering the alarm.

Impact on the System

If the full GC times exceeds the threshold, the performance of the JDBCServer2x process deteriorates, and the process can even be unavailable. As a result, Spark JDBC tasks are slow or fail to run.

Possible Causes

The heap memory usage of the JDBCServer2x process is excessively large, or the heap memory is inappropriately allocated. As a result, Full GC occurs frequently.

Procedure

Check the number of Full GCs.

  1. Log in to FusionInsight Manager, choose O&M > Alarm > Alarms, select this alarm, and check the RoleName in Location and confirm the IP address of HostName.
  2. Choose Cluster > Services > Spark2x > Instance. On the displayed page, click the JDBCServer2x for which the alarm is reported. On the Dashboard page that is displayed, click the drop-down menu in the Chart area and choose Customize > GC Number > Full GC Number of JDBCServer2x in the upper right corner and click OK. Check whether the number of Full GCs of the JDBCServer2x process is greater than the threshold(default value: 12).

    • If it is, go to 3.
    • If it is not, go to 6.
    Figure 1 Full GC Number of JDBCServer2x

  3. Choose Cluster > Services > Spark2x > Configurations > All Configurations. On the displayed page, choose JDBCServer2x > Tuning. The default value of SPARK_DRIVER_MEMORY is 4GB. You can change the value according to the following rules: If this alarm is generated occasionally, increase the value by 0.5 times. If the alarm is frequently reported, increase the value by 1 time. In the case of large service volume and high concurrency, add instances.
  4. Restart all JDBCServer2x instances.

    When the instance is rebooted, it cannot be used and any tasks running on the current instance node will fail.

  5. After 10 minutes, check whether the alarm is cleared.

    • If it is, no further action is required.
    • If it is not, go to 6.

Collect fault information.

  1. Log in to FusionInsight Manager, and choose O&M > Log > Download.
  2. Select Spark2x in the required cluster from the Service drop-down list.
  3. Click in the upper right corner. In the displayed dialog box, set Start Date and End Date to 10 minutes before and after the alarm generation time respectively and click OK. Then, click Download.
  4. Contact the O&M personnel and send the collected logs.

Alarm Clearing

This alarm will be automatically cleared after the fault is rectified.

Related Information

None