Help Center > > User Guide> Managing Active Clusters> Alarm Reference> ALM-13001 Available ZooKeeper Connections Are Insufficient

ALM-13001 Available ZooKeeper Connections Are Insufficient

Updated at: Mar 31, 2020 GMT+08:00

Description

The system checks ZooKeeper connections every 30 seconds. This alarm is generated when the system detects that the number of used ZooKeeper instance connections exceeds the threshold (80% of the maximum connections).

This alarm is cleared when the number of used ZooKeeper instance connections is less than the threshold.

Attribute

Alarm ID

Alarm Severity

Automatically Cleared

13001

Major

Yes

Parameters

Parameter

Description

ServiceName

Specifies the service for which the alarm is generated.

RoleName

Specifies the role for which the alarm is generated.

HostName

Specifies the host for which the alarm is generated.

Trigger Condition

Generates an alarm when the actual indicator value exceeds the specified threshold.

Impact on the System

Available ZooKeeper connections are insufficient. When the connection usage reaches 100%, external connections cannot be handled.

Possible Causes

  • The number of connections to the ZooKeeper node exceeds the threshold.
  • Connection leakage occurs on some connection processes.
  • The maximum number of connections does not meet the requirement of the actual scenario.

Procedure

  1. Check the connection status.

    1. On the MRS cluster details page, choose Alarms > ALM-13001 Available ZooKeeper Connections Are Insufficient > Location. Check the IP address of the alarm node.

      For MRS 1.8.10 or earlier, log in to MRS Manager to view alarm information.

    2. Obtain the PID of the ZooKeeper process. Log in to the alarm node and run the pgrep -f proc_zookeeper command.
    3. Check whether the PID can be successfully obtained.
      • If yes, go to 1.d.
      • If no, go to 2.
    4. Obtain all the IP addresses connected to the ZooKeeper instance and the number of connections. Check 10 IP addresses with the top connections. Run the following command based on the obtained PID and IP address: lsof -i|grep $pid | awk '{print $9}' | cut -d : -f 2 | cut -d \> -f 2 | awk '{a[$1]++} END {for(i in a){print i,a[i] | "sort -r -g -k 2"}}' | head -10. ($pid is the PID obtained in the preceding step.)
    5. Check whether the node IP addresses and the number of connections are successfully obtained.
      • If yes, go to 1.f.
      • If no, go to 2.
    6. Obtain the ID of the port connected to the process. Run the following command based on the obtained PID and IP address: lsof -i|grep $pid | awk '{print $9}' |cut -d \> -f 2 |grep $IP | cut -d :-f 2. ($pid and $IP are the PID and IP address obtained in the preceding step.)
    7. Check whether the port ID is successfully obtained.
      • If yes, go to 1.h.
      • If no, go to 2.
    8. Obtain the ID of the connected process. Log in to each IP address and run the following command based on the obtained port ID: lsof -i|grep $port. ($port is the port ID obtained in the preceding step.)
    9. Check whether the process ID is successfully obtained.
      • If yes, go to 1.j.
      • If no, go to 2.
    10. Check whether connection leakage occurs on the process based on the obtained process ID.
      • If yes, go to 1.k.
      • If no, go to 2.
    11. Close the process where connection leakage occurs and check whether the alarm is cleared.
      • If yes, no further action is required.
      • If no, go to 2.
    12. On the MRS cluster details page, choose Components > ZooKeeper > Service Configuration > All > quorumpeer > Performance and change the value of maxCnxns to 20000 or more.

      For MRS 1.8.10 or earlier, log in to MRS Manager, choose Services > ZooKeeper > Service Configuration > All > quorumpeer > Performance and change the value of maxCnxns to 20000 or more.

    13. Check whether the alarm is cleared.
      • If yes, no further action is required.
      • If no, go to 2.

  2. Collect fault information.

    1. On MRS Manager, choose System > Export Log.
    2. Contact the O&M personnel and send the collected log information.

Related Information

N/A

Did you find this page helpful?

Submit successfully!

Thank you for your feedback. Your feedback helps make our documentation better.

Failed to submit the feedback. Please try again later.

Which of the following issues have you encountered?







Please complete at least one feedback item.

Content most length 200 character

Content is empty.

OK Cancel