ALM-43022 IndexServer2x Process GC Time Exceeds the Threshold

Description

The system checks the GC time of the IndexServer2x process every 60 seconds. This alarm is generated when the detected GC time exceeds the threshold (12 seconds) for three consecutive times. To change the threshold, choose O&M > Alarm > Thresholds > Name of the desired cluster > Spark2x > GC Time > IndexServer2x Total GC time. This alarm is cleared when the IndexServer2x GC time is shorter than or equal to the threshold.

In MRS 3.3.0-LTS and later versions, the Spark2x component is renamed Spark, and the role names in the component are also changed. For example, IndexServer2x is changed to IndexServer. Refer to the descriptions and operations related to the component name and role names in the document based on your MRS version.

Attribute

Parameters

Impact on the System

If the GC duration exceeds the threshold, the performance of the IndexServer2x process deteriorates, and the process can even be unavailable. As a result, Carbon tasks with indexing enabled are slow or fail to run.

Possible Causes

The heap memory of the IndexServer2x process is overused or the heap memory is inappropriately allocated. As a result, GC occurs frequently.

Procedure

Check the GC time.

1. On FusionInsight Manager, choose O&M > Alarm > Alarms and select the alarm whose ID is 43022. Check the role name and the IP address of the host where the alarm is generated in Location.
2. On FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > Spark2x > Instance and click the IndexServer2x for which the alarm is generated to enter the Dashboard page. Click the drop-down menu in the Chart area and choose Customize > GC Time > Garbage Collection (GC) Time of IndexServer2x from the drop-down list box in the upper right corner and click OK to check whether the GC time is longer than the threshold (12 seconds by default).
   • If the threshold is reached, go to Step 3.
   • If the threshold is not reached, go to Step 6.
   Figure 1 IndexServer2x total GC duration
   

3. On FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > Spark2x > Configurations > All Configurations > IndexServer2x > Default. The default value of the SPARK_DRIVER_MEMORY is 4 GB. You can increase the value 1.5 times to its default. If this alarm is still generated occasionally after the adjustment, increase the value by 0.5 times. Double the value if the alarm is reported frequently.
4. Restart all IndexServer2x instances.
   

   When the instance is rebooted, it cannot be used and any tasks running on the current instance node will fail.

5. Check whether the alarm is cleared in 10 minutes.
   • If the alarm is cleared, no further action is required.
   • If the alarm is not cleared, go to Step 6.

Collect fault information.

6. On FusionInsight Manager, choose O&M > Log > Download.
7. Expand the Service drop-down list, and select Spark2x for the target cluster.
8. Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time respectively. Then, click Download.
9. Send the collected fault logs to O&M personnel for help.

Alarm Clearing

After the fault is rectified, the system automatically clears this alarm.

Reference

None