ALM-45441 Zookeeper Disconnected

Alarm Description

The system checks the connection between ClickHouse and ZooKeeper every minute. This alarm is generated when the connection fails. The alarm is reported because the ZooKeeper connection is abnormal. If the connection fails for three consecutive times, the system generates an alarm.

This alarm is automatically cleared when the system detects that the connection is normal.

Alarm Attributes

Alarm Parameters

Impact on the System

If ClickHouse is disconnected from ZooKeeper, the ClickHouse service cannot be used.

Possible Causes

• The ZooKeeper service is abnormal.
• The ClickHouse service is overloaded.

Handling Procedure

Check whether ZooKeeper is normal.

1. On FusionInsight Manager, choose Cluster > Services > ZooKeeper > quorumpeer.
2. Check whether ZooKeeper instances are normal.
   • If yes, go to 6.
   • If no, go to 3.

3. Select instances whose status is not good and choose More > Restart Instance.
4. Check whether the instance status is good after restart.
   • If yes, go to 5.
   • If no, go to 10.

5. Choose O&M > Alarm > Alarms and check whether the alarm is cleared.
   • If yes, no further action is required.
   • If no, go to 6.

Check whether the ClickHouse service load is heavy.

6. Log in to FusionInsight Manager, choose O&M > Alarm > Alarms, and view the role name and the IP address for the hostname in Location.
7. Log in to the node where the client is installed as the client installation user and run the following commands:

   cd {Client installation path}

   source bigdata_env

   • For a cluster with Kerberos authentication enabled (security mode):

     kinit Component service user

     clickhouse client --host IP address of the ClickHouseServer instance that reports the alarm --port 9440 --secure

   • For a cluster with Kerberos authentication disabled (normal mode):

     clickhouse client --host IP address of the ClickHouseServer instance that reports the alarm --user User name --password --port 9440

8. Run the following statement to check whether data is frequently written to the system table. If yes, wait until the service execution is complete and check whether the alarm is cleared.

   SELECT query_id, user, FQDN(), elapsed, query FROM system.processes ORDER BY query_id;

   • If yes, no further action is required.
   • If no, go to 9.

9. Check whether a large amount of data is written. If yes, wait until the task is complete and check whether the alarm is cleared.
   • If yes, no further action is required.
   • If no, go to 10.

Collect fault information.

10. On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
11. Expand the Service drop-down list, and select ClickHouse for the target cluster.
12. Expand the Hosts drop-down list. In the Select Host dialog box that is displayed, select the abnormal host, and click OK.
13. Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.
14. Contact O&M personnel and provide the collected logs.

Alarm Clearance

This alarm is automatically cleared after the fault is rectified.

Related Information

None.