ALM-13001 Available ZooKeeper Connections Are Insufficient (For MRS 2.x or Earlier)

Description

The system checks ZooKeeper connections every 30 seconds. This alarm is generated when the system detects that the number of used ZooKeeper instance connections exceeds the threshold (80% of the maximum connections).

This alarm is cleared when the number of used ZooKeeper instance connections is less than the threshold.

Attribute

Parameters

Impact on the System

Available ZooKeeper connections are insufficient. When the connection usage reaches 100%, external connections cannot be handled.

Possible Causes

The number of connections to the ZooKeeper node exceeds the threshold. Connection leakage occurs on some connection processes, or the maximum number of connections does not meet the requirement of the actual scenario.

Procedure

1. Check the connection status.
   1. On the MRS cluster details page, choose Alarms > ALM-13001 Available ZooKeeper Connections Are Insufficient > Location. Check the IP address of the node for which the alarm is generated.
   2. Obtain the PID of the ZooKeeper process. Log in to the node for which this alarm is generated and run the pgrep -f proc_zookeeper command.
   3. Check whether the PID can be successfully obtained.
      • If yes, go to 1.d.
      • If no, go to 2.
   4. Obtain all the IP addresses connected to the ZooKeeper instance and the number of connections and check 10 IP addresses with top connections. Run the lsof -i|grep $pid | awk '{print $9}' | cut -d : -f 2 | cut -d \> -f 2 | awk '{a[$1]++} END {for(i in a){print i,a[i] | "sort -r -g -k 2"}}' | head -10 command based on the obtained PID value. ($pid is the PID obtained in the preceding step.)
   5. Check whether the node IP addresses and the number of connections are successfully obtained.
      • If yes, go to 1.f.
      • If no, go to 2.
   6. Obtain the ID of the port connected to the process. Run the lsof -i|grep $pid | awk '{print $9}'|cut -d \> -f 2 |grep $IP| cut -d : -f 2 command based on the obtained PID and IP address. ($pid and $IP are the PID and IP address obtained in the preceding step.)
   7. Check whether the port ID is successfully obtained.
      • If yes, go to 1.h.
      • If no, go to 2.
   8. Obtain the ID of the connected process. Log in to each IP address and run the following command based on the obtained port ID: lsof -i|grep $port. ($port is the port ID obtained in the preceding step.)
   9. Check whether the process ID is successfully obtained.
      • If yes, go to 1.j.
      • If no, go to 2.
   10. Check whether connection leakage occurs on the process based on the obtained process ID.
       • If yes, go to 1.k.
       • If no, go to 1.l.
   11. Close the process where connection leakage occurs and check whether the alarm is cleared.
       • If yes, no further action is required.
       • If no, go to 1.l.
   12. On the MRS cluster details page, choose Components > ZooKeeper > Service Configuration. Set Type to All, choose quorumpeer > Performance, and change the value of maxCnxns to 20000 or more.
   13. Check whether the alarm is cleared.
       • If yes, no further action is required.
       • If no, go to 2.

2. Collect fault information.
   1. On MRS Manager, choose System > Export Log.
   2. Contact the O&M engineers and send the collected logs.

Reference

None