ALM-13010 Znode Usage of a Directory with Quota Configured Exceeds the Threshold

Alarm Description

The system checks the Znode usage of all service directories with quota configured every hour. This alarm is generated when the system detects that the level-2 Znode usage exceeds the threshold.

Alarm Attributes

Alarm Parameters

Impact on the System

When a large amount of data is written to the ZooKeeper data directory space, ZooKeeper cannot provide services for external systems. As a result, services of upstream components (such as Yarn, Flink, and Spark) that depend on the alarm directory are abnormal.

Possible Causes

• A large amount of data is written to the ZooKeeper data directory.
• The user-defined threshold is inappropriate.

Handling Procedure

Check whether a large amount of data is written into the directory for which the alarm is generated.

1. On FusionInsight Manager, choose O&M > Alarm > Alarms. Confirm the Znode for which the alarm is generated in Location of this alarm.
2. Choose Cluster > Name of the desired cluster > Services > ZooKeeper and click Resource. In Used Resources (By Second-Level Znode), check whether a large amount of data is written into the top Znode.
   • If yes, go to 3.
   • If no, go to 5.

3. Log in to FusionInsight Manager, choose O&M > Alarm > Alarms, select Location from the drop-down list box next to ALM-13010 Znode Usage of a Directory with Quota Configured Exceeds the Threshold, and obtain the Znode path in ServiceDirectory.
4. Log in to the ZooKeeper client as a cluster user and delete unwanted data in the Znode for which the alarm is generated.
5. Log in to FusionInsight Manager, and choose Cluster > Name of the desired cluster > Services > Component of the top Znode for which the alarm isgenerated. Choose Configurations > All Configurations, search for zk.quota.number, increase its value, click Save.
   

   If the Component of the top Znode for which the alarm isgenerated is ClickHouse, change the value of clickhouse.zookeeper.quota.node.count.

6. Check whether the alarm is cleared.
   • If yes, no further action is required.
   • If no, go to 7.

Collect fault information.

7. On the FusionInsight Manager portal, choose O&M > Log > Download.
8. Select ZooKeeper in the required cluster from the Service.
9. Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
10. Contact the O&M engineers and send the collected logs.

Alarm Clearance

After the fault is rectified, the system automatically clears this alarm.

Related Information

None.