Help Center> MapReduce Service> User Guide (Kuala Lumpur Region)> FusionInsight Manager Operation Guide (Applicable to 3.x)> Alarm Reference (Applicable to MRS 3.x)> ALM-13003 GC Duration of the ZooKeeper Process Exceeds the Threshold
Updated on 2022-08-12 GMT+08:00
View PDF

ALM-13003 GC Duration of the ZooKeeper Process Exceeds the Threshold

Description

The system checks the garbage collection (GC) duration of the ZooKeeper process every 60 seconds. This alarm is generated when the GC duration exceeds the threshold (12 seconds by default).

This alarm is cleared when the GC duration is less than the threshold.

Attribute

Alarm ID

Alarm Severity

Auto Clear

13003

Major

Yes

Parameters

Name

Meaning

Source

Specifies the cluster for which the alarm is generated.

ServiceName

Specifies the service for which the alarm is generated.

RoleName

Specifies the role for which the alarm is generated.

HostName

Specifies the host for which the alarm is generated.

Trigger Condition

Specifies the threshold triggering the alarm. If the current indicator value exceeds this threshold, the alarm is generated.

Impact on the System

A long GC duration of the ZooKeeper process may interrupt the services.

Possible Causes

The heap memory of the ZooKeeper instance is overused or the heap memory is inappropriately allocated. As a result, GCs occur frequently.

Procedure

Check the GC duration.

  1. On the FusionInsight Manager portal, choose O&M > Alarm > Alarms. On the displayed interface, click the drop-down button of GC Duration of the ZooKeeper Process Exceeds the Threshold. Check the IP address of the instance for which the alarm is generated.
  2. On the FusionInsight Manager portal, choose Cluster > Name of the desired cluster > Services > ZooKeeper > Instance > quorumpeer(IP address for which the alarm is generated). Click the drop-down menu in the upper right corner of Chart, choose Customize > GC, and select ZooKeeper GC Duration per Minute, click OK to check the GC duration statistics of the ZooKeeper process collected every minute.
  3. Check whether the GC duration of the ZooKeeper process collected every minute exceeds the threshold (12 seconds by default).

    • If yes, go to 4.
    • If no, go to 8.

  4. Check whether memory leakage occurs in the application program.
  5. On the FusionInsight Manager portal, choose Cluster > Name of the desired cluster > Services > ZooKeeper > Configurations > All Configurations > quorumpeer > System to increase the value of GC_OPTS parameter as required.

    Generally, the value of -Xmx is twice of the ZooKeeper data capacity. You are advised to set MaxDirectMemorySize to half of the data capacity. If the ZooKeeper capacity reaches 2G, you are advised to set GC_OPTS to:

    -Xms4G -Xmx4G -XX:NewSize=512M -XX:MaxNewSize=512M -XX:MetaspaceSize=64M -XX:MaxMetaspaceSize=64M -XX:CMSFullGCsBeforeCompaction=1

  6. Save the configuration and restart the ZooKeeper service.
  7. Check whether the alarm is cleared.

    • If yes, no further action is required.
    • If no, go to 8.

Collect fault information.

  1. On the FusionInsight Manager portal, choose O&M > Log > Download.
  2. Select ZooKeeper in the required cluster from the Service.
  3. Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
  4. Contact the O&M personnel and send the collected logs.

Alarm Clearing

After the fault is rectified, the system automatically clears this alarm.

Related Information

None