Help Center/ MapReduce Service/ User Guide/ MRS Cluster O&M/ MRS Cluster Alarm Handling Reference/ ALM-13008 ZooKeeper Znode Usage Exceeds the Threshold
Updated on 2024-11-13 GMT+08:00
View PDF
Share

ALM-13008 ZooKeeper Znode Usage Exceeds the Threshold

Description

The system checks the level-2 Znode status in the ZooKeeper data directory every hour (every 10 minutes in MRS 3.5.0 and later versions). This alarm is generated when the system detects that the level-2 Znode usage exceeds the threshold.

Attribute

Alarm ID

Alarm Severity

Automatically Cleared

13008

Major

Yes

Parameters

Name

Meaning

Source

Specifies the cluster for which the alarm is generated.

ServiceName

Specifies the service name for which the alarm is generated.

ServiceDirectory

Specifies the directory for which the alarm is generated.

RoleName

Specifies the role for which the alarm is generated.

Trigger Condition

Specifies the cause of the alarm.

Impact on the System

When a large amount of data is written to the ZooKeeper data directory space, ZooKeeper cannot provide services for external systems. As a result, services of upstream components (such as Yarn, Flink, and Spark) that depend on the alarm directory are abnormal.

Possible Causes

  • A large amount of data is written to the ZooKeeper data directory.
  • The user-defined threshold is inappropriate.

Procedure

Check whether a large amount of data is written into the directory for which the alarm is generated.

  1. Log in to FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > ZooKeeper, and click Resource. Click By Znode quantity in Used Resources (By Second-Level Znode), and check whether a large amount of data is written to the top Znode.

    • If yes, go to 2.
    • If no, go to 4.

  2. Log in to FusionInsight Manager, choose O&M > Alarm > Alarms, select Location from the drop-down list box next to ALM-13008 ZooKeeper Znode Quantity Usage Exceeds Threshold, and obtain the Znode path in ServiceDirectory.
  3. Log in to the ZooKeeper client as a cluster user and delete unnecessary data from the Znode corresponding to the alarm.
  4. Log in to FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > ZooKeeper > Configurations > All Configurations, and search for max.znode.count, which is the maximum number of ZooKeeper directories. The alarm threshold is 80% of this parameter. Increase the value of this parameter, click Save, and restart the service for the configuration to take effect.

    The service will be unavailable during the restart. In addition, upper-layer services that depend on the service are affected.

  5. Check whether the alarm is cleared.

    • If yes, no further action is required.
    • If no, go to 6.

Collect fault information.

  1. On the FusionInsight Manager portal, choose O&M > Log > Download.
  2. Select ZooKeeper in the required cluster from the Service.
  3. Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
  4. Contact the O&M personnel and send the collected logs.

Alarm Clearing

After the fault is rectified, the system automatically clears this alarm.

Related Information

None