ALM-43023 IndexServer2x Process Full GC Number Exceeds the Threshold

Description

The system checks the Full GC number of the IndexServer2x process every 60 seconds. This alarm is generated when the detected Full GC number exceeds the threshold (12) for three consecutive times. You can change the threshold by choosing O&M > Alarm > Thresholds > Spark2x > GC Number > Full GC Number of IndexServer2x. This alarm is cleared when the Full GC number of the IndexServer2x process is less than or equal to the threshold. This alarm is cleared when the Full GC number of the IndexServer2x process is less than or equal to the threshold.

In MRS 3.3.0-LTS and later versions, the Spark2x component is renamed Spark, and the role names in the component are also changed. For example, IndexServer2x is changed to IndexServer. Refer to the descriptions and operations related to the component name and role names in the document based on your MRS version.

Attribute

Alarm ID	Severity	Auto Clear
43023	Major	Yes

Parameters

Parameter	Description
Source	Specifies the cluster for which the alarm is generated.
ServiceName	Specifies the service for which the alarm is generated.
RoleName	Specifies the role for which the alarm is generated.
HostName	Specifies the host for which the alarm is generated.
Trigger Condition	Specifies the threshold for triggering the alarm.

Impact on the System

If the GC times exceeds the threshold, the performance of the IndexServer2x process deteriorates, and the process can even be unavailable. As a result, Carbon tasks with indexing enabled are slow or fail to run.

Possible Causes

The heap memory of the IndexServer2x process is overused or the heap memory is inappropriately allocated. As a result, Full GC occurs frequently.

Procedure

Check the number of Full GCs.

On FusionInsight Manager, choose O&M > Alarm > Alarms. In the displayed alarm list, choose the alarm with the ID 43023, and check the RoleName in Location and confirm the IP address of HostName.
On FusionInsight Manager, choose Cluster > Services > Spark2x > Instance and click the IndexServer2x for which the alarm is generated to go to the Dashboard page. Click the drop-down menu in the chart area and choose Customize > GC Number > Full GC Number of IndexServer2x from the drop-down list box in the upper right corner and click OK to check whether the GC number is larger than the threshold (default value: 12).
- If the threshold is reached, go to 3.
- If the threshold is not reached, go to 6.
Figure 1 Full GC Number of IndexServer2x
On FusionInsight Manager, choose Cluster > Services > Spark2x > Configurations > All Configurations > IndexServer2x > Tuning. The default value of the SPARK_DRIVER_MEMORY is 4 GB. You can change the value according to the following rules: If this alarm is generated occasionally, increase the value by 0.5 times. Double the value if the alarm is reported frequently. In the case of large service volume and high service concurrency, you are advised to add instances.
Restart all IndexServer2x instances.

When the instance is rebooted, it cannot be used and any tasks running on the current instance node will fail.
After 10 minutes, check whether the alarm is cleared.
- If the alarm is cleared, no further action is required.
- If the alarm is not cleared, go to 6.

Collect fault information.

On FusionInsight Manager, choose O&M > Log > Download.
Expand the Service drop-down list, and select Spark2x for the target cluster.
Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time respectively. Then, click Download.
Contact the O&M personnel and send the collected fault logs.