ALM-51203 GC Duration of the LakeSearch Instance Exceeds the Threshold

Alarm Description

The system checks the garbage collection (GC) duration of the LakeSearch instance process every 60 seconds. This alarm is generated when the GC duration exceeds the threshold (30 seconds by default).

If Trigger Count is 1, this alarm is cleared when the GC duration of the LakeSearch instance process is less than or equal to the threshold. If Trigger Count is greater than 1, this alarm is cleared when the GC duration of the LakeSearch instance process is less than or equal to 90% of the threshold.

Alarm Attributes

Alarm Parameters

Impact on the System

A long GC duration of the LakeSearch instance process may interrupt the services.

Possible Causes

Service load of the LakeSearch instance on the node is high or the heap memory is not properly configured. As a result, GC frequently occurs.

Handling Procedure

Check the heap memory of the instance.

1. Log in to FusionInsight Manager, choose O&M > Alarm > Alarms, expand alarm "GC Duration of the LakeSearch Instance Exceeds the Threshold", and check the HostName in Location.
2. Choose Cluster > Services > LakeSearch, click Instances, and click the SearchServer and SearchFactory instance names corresponding to the host for which the alarm is generated. On the displayed page, check whether the GC duration is continuously greater than the threshold by checking SearchServer GC Duration Statistics and SearchFactory GC Duration Statistics displayed in charts. If related charts do not exist, click the triangle icon next to the time editing button in the chart area, click Customize. On the displayed page, click Garbage Collection, select the charts, and click OK.
   • If the GC duration is continuously greater than the threshold, go to 3.
   • If the GC duration threshold is properly set, go to 11.

3. Choose Cluster > Services > LakeSearch. Click Configurations and click All Configurations.
4. In the upper right corner of the configuration page, enter GC_OPTS in the search box. The GC_OPTS parameters of all instances are displayed.
5. Select the instance whose parameter GC_OPTS needs to be modified, and check whether the differentiated configuration icon is displayed next to the instance value configuration box.
   • If the icon is displayed, go to 6.
   • If the icon is not displayed, go to 7.

6. Click . In the displayed dialog box, click on the right and click OK.
7. Change the values of -Xms and -Xmx of the GC_OPTS parameter by referring to the following note.
   

   Suggestions on GC parameter settings for LakeSearch instances:

   • Change the values of -Xms and -Xmx of the GC_OPTS parameter to 8G.
   • Set -Xms and -Xmx to the same value to prevent dynamic adjustment of the heap memory size, which may affect performance.

8. After the modification, click Save in the upper left corner. In the displayed dialog box, click OK.
9. Click Instances, select the instances whose configuration status is Expired, and choose More > Restart Instance to restart them as prompted.
10. Wait for one minute and check whether the alarm is cleared.
    • If it is cleared, no further action is required.
    • If it persists, go to 11.

Collect fault information.

11. On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
12. Expand the Service drop-down list, and select LakeSearch for the target cluster.
13. Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
14. Contact O&M personnel and provide the collected logs.

Alarm Clearance

This alarm is automatically cleared after the fault is rectified.

Related Information

None