ALM-13008 ZooKeeper Znode Usage Exceeds the Threshold

Alarm Description

The system checks the level-2 Znode status in the ZooKeeper data directory every hour (every 10 minutes in MRS 3.5.0 and later versions). This alarm is generated when the system detects that the level-2 Znode usage exceeds the threshold.

Alarm Attributes

Alarm Parameters

Impact on the System

When a large amount of data is written to the ZooKeeper data directory space, ZooKeeper cannot provide services for external systems. As a result, services of upstream components (such as Yarn, Flink, and Spark) that depend on the alarm directory are abnormal.

Possible Causes

• A large amount of data is written to the ZooKeeper data directory.
• The user-defined threshold is inappropriate.

Handling Procedure

Check whether a large amount of data is written into the directory for which the alarm is generated.

1. Log in to FusionInsight Manager, choose Cluster > Services > ZooKeeper, and click Resource. Click By Znode quantity in Used Resources (By Second-Level Znode), and check whether a large amount of data is written to the top Znode.
   • If yes, go to Step 2.
   • If no, go to Step 4.

2. Log in to FusionInsight Manager, choose O&M > Alarm > Alarms, select Location from the drop-down list box next to ALM-13008 ZooKeeper Znode Quantity Usage Exceeds Threshold, and obtain the Znode path in ServiceDirectory.
3. Log in to the ZooKeeper client as a cluster user and delete unnecessary data from the Znode corresponding to the alarm.

   For details about operations on the ZooKeeper client, see Using the ZooKeeper Client.

4. Log in to FusionInsight Manager, choose Cluster > Services > ZooKeeper > Configurations > All Configurations, and search for max.znode.count, which is the maximum number of ZooKeeper directories. The alarm threshold is 80% of this parameter. Increase the value of this parameter, click Save, and restart the service for the configuration to take effect.
   

   The service will be unavailable during the restart. In addition, upper-layer services that depend on the service are affected.

5. Check whether the alarm is cleared.
   • If yes, no further action is required.
   • If no, go to Step 6.

Collect fault information.

6. On the FusionInsight Manager portal, choose O&M > Log > Download.
7. Select ZooKeeper in the required cluster from the Service.
8. Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
9. Contact the O&M personnel and send the collected logs.

Alarm Clearance

After the fault is rectified, the system automatically clears this alarm.

Related Information

None