Help Center/ MapReduce Service/ User Guide (Ankara Region)/ Alarm Reference/ ALM-43023 IndexServer Process Full GC Number Exceeds the Threshold
Updated on 2024-11-29 GMT+08:00
View PDF
Share

ALM-43023 IndexServer Process Full GC Number Exceeds the Threshold

Alarm Description

The system checks the number of IndexServer Full GC times every 60 seconds. A critical alarm is reported when the number exceeds 12 for three consecutive times. A major alarm is reported when the number exceeds 12 x 0.8 (rounded down) for three consecutive times. You can change the threshold by choosing O&M > Alarm > Thresholds > Name of the desired cluster > Spark > GC Number > Full GC Number of IndexServer. This alarm is cleared when the Full GC times of the JDBCServer process is less than or equal to the threshold. This alarm is cleared when the Full GC times of the IndexServer process is less than or equal to the threshold.

Alarm Attributes

Alarm ID

Alarm Severity

Alarm Type

Service Type

Auto Cleared

43023

Minor (default threshold: 9 for three consecutive times)

Critical (default threshold: 12 for three consecutive times)

Quality of service

Spark

Yes

Alarm Parameters

Type

Parameter

Description

Location Information

Source

Identifies the cluster for which the alarm is generated.

ServiceName

Specifies the service for which the alarm is generated.

RoleName

Specifies the role for which the alarm is generated.

HostName

Specifies the host for which the alarm is generated.

Additional Information

Trigger Condition

Specifies the alarm triggering condition.

Impact on the System

If the GC times exceeds the threshold, the performance of the IndexServer process deteriorates, and the process can even be unavailable. As a result, Carbon tasks with indexing enabled are slow or fail to run.

Possible Causes

The heap memory of the IndexServer process is overused or inappropriately allocated, causing frequent occurrence of the full GC process.

Handling Procedure

Check the number of Full GCs.

  1. On FusionInsight Manager, choose O&M > Alarm > Alarms and select the alarm whose ID is 43023. Check the role name and the IP address of the host where the alarm is generated in Location.
  2. On FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > Spark > Instance and click the IndexServer for which the alarm is generated to enter the Dashboard page. Click the drop-down menu in the chart area and choose Customize > Full GC Number of IndexServer from the drop-down list box in the upper right corner and click OK to check whether the full GC times is larger than the threshold (default value: 12).

    • If the threshold is reached, go to 3.
    • If the threshold is not reached, go to 6.

  3. On FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > Spark > Configurations > All Configurations > IndexServer > Performance. The default value of the SPARK_DRIVER_MEMORY is 4 GB. You can increase the value by 0.5 times if this alarm is generated occasionally. Double the value if the alarm is reported frequently. In the case of large service volume and high service concurrency, you are advised to add instances.
  4. Restart all IndexServer instances.
  5. Check whether the alarm is cleared 10 minutes later.

    • If the alarm is cleared, no further action is required.
    • If the alarm is not cleared, go to 6.

Collect fault information.

  1. On FusionInsight Manager, choose O&M > Log > Download.
  2. Expand the Service drop-down list, and select Spark for the target cluster.
  3. Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time respectively. Then, click Download.
  4. Contact O&M engineers and provide the collected logs.

Alarm Clearance

After the fault is rectified, the system automatically clears this alarm.

Related Information

None.

Parent topic: Alarm Reference