ALM-13008 ZooKeeper Znode Usage Exceeds the Threshold

Alarm Description

The system checks the level-2 Znode status in the ZooKeeper data directory every hour. This alarm is generated when the system detects that the level-2 Znode usage exceeds the threshold.

Alarm Attributes

Alarm Parameters

Impact on the System

When a large amount of data is written to the ZooKeeper data directory space, ZooKeeper cannot provide services for external systems. As a result, services of upstream components (such as Yarn, Flink, and Spark) that depend on the alarm directory are abnormal.

Possible Causes

• A large amount of data is written to the ZooKeeper data directory.
• The user-defined threshold is inappropriate.

Procedure

Check whether a large amount of data is written into the directory for which the alarm is generated.

1. Log in to FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > ZooKeeper, and click Resource. Click By Znode quantity in Used Resources (By Second-Level Znode), and check whether a large amount of data is written to the top Znode.
   • If yes, go to 2.
   • If no, go to 4.

2. Log in to FusionInsight Manager, choose O&M > Alarm > Alarms, select Location from the drop-down list box next to ALM-13008 ZooKeeper Znode Quantity Usage Exceeds Threshold, and obtain the Znode path in ServiceDirectory.
3. Log in to the ZooKeeper client as a cluster user and delete unnecessary data from the Znode corresponding to the alarm.
4. Log in to FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > ZooKeeper > Configurations > All Configurations, and search for max.znode.count, which is the maximum number of ZooKeeper directories. The alarm threshold is 80% of this parameter. Increase the value of this parameter, click Save, and restart the service for the configuration to take effect.
5. Check whether the alarm is cleared.
   • If yes, no further action is required.
   • If no, go to 6.

Collect fault information.

6. On the FusionInsight Manager portal, choose O&M > Log > Download.
7. Select ZooKeeper in the required cluster from the Service.
8. Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
9. Contact the O&M engineers and send the collected logs.

Alarm Clearance

After the fault is rectified, the system automatically clears this alarm.

Related Information

None.