Help Center/ MapReduce Service/ User Guide/ MRS Cluster O&M/ MRS Cluster Alarm Handling Reference/ ALM-13003 GC Duration of the ZooKeeper Process Exceeds the Threshold
Updated on 2024-09-23 GMT+08:00
View PDF
Share

ALM-13003 GC Duration of the ZooKeeper Process Exceeds the Threshold

Alarm Description

The system checks the garbage collection (GC) duration of the ZooKeeper process every 60 seconds. This alarm is generated when the GC duration exceeds the threshold (12 seconds by default).

This alarm is cleared when the GC duration is less than the threshold.

Alarm Attributes

Alarm ID

Alarm Severity

Auto Cleared

13003

Major

Yes

Alarm Parameters

Parameter

Description

Source

Specifies the cluster for which the alarm was generated.

ServiceName

Specifies the service for which the alarm was generated.

RoleName

Specifies the role for which the alarm was generated.

HostName

Specifies the host for which the alarm was generated.

Trigger Condition

Specifies the threshold for triggering the alarm.

Impact on the System

The ZooKeeper process may respond slowly. Services of upper-layer components (such as Yarn, Flink, and Spark) may fail.

Possible Causes

The heap memory of the ZooKeeper process is overused or inappropriately allocated, causing frequent occurrence of the GC process.

Handling Procedure

Check the GC duration.

  1. On FusionInsight Manager, choose O&M > Alarm > Alarms. On the displayed page, click the drop-down list of GC Duration of the ZooKeeper Process Exceeds the Threshold. View the IP address of the instance for which the alarm is generated.
  2. On FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > ZooKeeper > Instance > quorumpeer. Click the drop-down list in the upper right corner of Chart, choose Customize > GC, select ZooKeeper GC Duration per Minute, and click OK to check the GC duration statistics of the ZooKeeper process collected every minute.

    Figure 1 ZooKeeper GC duration

  3. Check whether the GC duration of the ZooKeeper process collected every minute exceeds the threshold (12 seconds by default).

    • If yes, go to 4.
    • If no, go to 8.

  4. Check whether memory leakage occurs in the application.
  5. On FusionInsight Manager, choose Cluster, click the name of the desired cluster, and choose Services > ZooKeeper > Configurations > All Configurations > quorumpeer > System. Increase the value of the GC_OPTS parameter as required.

    Generally, -Xmx is twice of ZooKeeper data capacity. If the capacity of ZooKeeper reaches 2 GB, set GC_OPTS as follows:

    -Xms4G -Xmx4G -XX:NewSize=512M -XX:MaxNewSize=512M -XX:MetaspaceSize=64M -XX:MaxMetaspaceSize=64M -XX:CMSFullGCsBeforeCompaction=1

  6. Save the configuration and restart the ZooKeeper service.

    The service will be unavailable during the restart. In addition, upper-layer services that depend on the service are affected.

  7. Check whether the alarm is cleared.

    • If yes, no further action is required.
    • If no, go to 8.

Collect the fault information.

  1. On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
  2. Expand the Service drop-down list, and select ZooKeeper for the target cluster.
  3. Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
  4. Contact O&M personnel and provide the collected logs.

Alarm Clearance

This alarm is automatically cleared after the fault is rectified.

Related Information

None