Help Center/ MapReduce Service/ User Guide/ MRS Cluster O&M/ MRS Cluster Alarm Handling Reference/ ALM-13006 Znode Number or Capacity Exceeds the Threshold
Updated on 2024-09-23 GMT+08:00
View PDF
Share

ALM-13006 Znode Number or Capacity Exceeds the Threshold

Description

The system periodically detects the status of secondary Znode in the ZooKeeper service data directory every four hours. This alarm is generated when the number or capacity of secondary Znodes exceeds the threshold.

Attribute

Alarm ID

Alarm Severity

Automatically Cleared

13006

Minor

Yes

Parameters

Name

Meaning

Source

Specifies the cluster for which the alarm is generated.

ServiceName

Specifies the service name for which the alarm is generated.

ServiceDirectory

Specifies the directory for which the alarm is generated.

Trigger Condition

Specifies the cause of the alarm.

Impact on the System

A large amount of data is written to the ZooKeeper data directory space. As a result, services of upstream components (such as Yarn, Flink, and Spark) that depend on this directory are abnormal. For details, see the alarm location information.

Possible Causes

A large amount of data is written to the ZooKeeper data directory. The threshold is not appropriate.

Procedure

Check whether a large amount of data is written to the directory for which the alarm is generated.

  1. On FusionInsight Manager, choose O&M > Alarm > Alarms. On the displayed interface, click the drop-down button of Znode Number or Capacity Exceeds the Threshold. Confirm the Znode for which the alarm is generated in Location Information.
  2. Log in to FusionInsight Manager, open the ZooKeeper service interface, and select Resource. In the table Used Resources (By Second-Level Znode), check whether a large amount of data is written to the top-level Znode for which the alarm is reported.

    • If it is, go to 3.
    • If it is not, go to 4.
    Figure 1 Used Resources (By Second-Level Znode)

  3. Log in to the ZooKeeper client and delete the data in the top-level Znode.
  4. Log in to FusionInsight Manager and open the ZooKeeper service interface. On the Resource page, choose > By Znode quantity in Used Resources (By Second-Level Znode). Threshold Configuration of By Znode quantity is displayed. Click Modify under Operation. Increase the threshold by referringto the value of max.Znode.count by choosing Cluster > Name of the desired cluster > Services > ZooKeeper > Configurations > All Configurations > Quota.

    Figure 2 Modify Rule

  5. In the Used Resources (By Second-Level Znode), choose > By capacity. The Threshold Settings page of By Capacity is displayed. Click Modify under Operation. Increase the threshold by referring to the value of max.data.size by choosing Cluster > Name of the desired cluster > Services > ZooKeeper > Configurations > All Configurations > Quota.
  6. Check whether the alarm is cleared.

    • If it is, no further action is required.
    • If it is not, go to 7.

Collect fault information.

  1. On the FusionInsight Manager portal, choose O&M > Log > Download.
  2. Select ZooKeeper in the required cluster from the Service.
  3. Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
  4. Contact the O&M personnel and send the collected logs.

Alarm Clearing

After the fault is rectified, the system automatically clears this alarm.

Related Information

None