ALM-13007 Available ZooKeeper Client Connections Are Insufficient

Alarm Description

The system detects the number of active processes between the ZooKeeper client and the ZooKeeper server every 60 seconds. This alarm is generated when the number of connections exceeds the threshold.

Alarm Attributes

Alarm Parameters

Impact on the System

A large number of processes are connected to ZooKeeper, and the ZooKeeper connections are used up. As a result, upstream components (such as YARN, Flink, and Spark) cannot provide services properly.

Possible Causes

A large number of client processes are connected to ZooKeeper. Or, the threshold is set improperly.

Handling Procedure

Check whether there are a large number of client processes connected to ZooKeeper.

1. On FusionInsight Manager, choose O&M > Alarm > Alarms. On the displayed page, click the drop-down button in the row where the alarm "Available ZooKeeper Client Connections Are Insufficient" is located. In the Location area, check the IP address of the node where the host for which the alarm is generated resides.
2. Choose Cluster > Name of the desired cluster > Services > ZooKeeper and click Resources. On the displayed page, locate the client IP address for which the alarm is generated and check whether its value of Number of Connections (by Client IP Address) is large.
   • If yes, go to Step 3.
   • If no, go to Step 4.

3. Check whether connection leakage occurs on the client process.
4. Click in the Number of Connections (by Client IP Address) to enter the Thresholds page, and click Modify under Operation. Increase the threshold by referring to the value of maxClientCnxns. To obtain the value, choose Cluster > Name of the desired cluster > Services > ZooKeeper > Configurations > All Configurations > quorumpeer.
5. Check whether the alarm is cleared.
   • If yes, no further action is required.
   • If no, go to Step 6.

Collect fault information.

6. On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
7. Expand the Service drop-down list, and select ZooKeeper for the target cluster.
8. Click in the upper right corner, set Start Date and End Date for log collection to 10 minutes before and after the alarm generation time, respectively, and click Download.
9. Send the collected fault logs to O&M personnel for help.

Alarm Clearance

This alarm is automatically cleared after the fault is rectified.

Related Information

None