ALM-13000 ZooKeeper Service Unavailable

Alarm Description

The system checks the ZooKeeper service status every 60 seconds. This alarm is generated when the ZooKeeper service is unavailable.

This alarm is cleared when the ZooKeeper service recovers.

Alarm Attributes

Alarm Parameters

Impact on the System

ZooKeeper cannot provide coordination services for upper layer components and the components (such as Yarn and Flink) that depend on ZooKeeper may not run properly.

Possible Causes

• The DNS is installed on the ZooKeeper node.
• The network is faulty.
• The KrbServer service is abnormal.
• The ZooKeeper instance is abnormal.
• The disk capacity is insufficient.

Handling Procedure

Check the DNS.

1. Check whether the DNS is installed on the node where the ZooKeeper instance is located. On the Linux node where the ZooKeeper instance is located, run the cat /etc/resolv.conf command to check whether the file is empty.
   • If yes, go to 2.
   • If no, go to 3.

2. Run the service named status command to check whether the DNS is started.
   • If yes, go to 3.
   • If no, go to 5.

3. Run the service named stop command to stop the DNS service. If "Shutting down name server BIND waiting for named to shut down (28s)" is displayed, the DNS service is stopped successfully. Comment out the content (if any) in /etc/resolv.conf.
4. On the O&M > Alarm > Alarms tab, check whether the alarm is cleared.
   • If yes, no further action is required.
   • If no, go to 5.

Check the network status.

5. On the Linux node where the ZooKeeper instance is located, run the ping command to check whether the host names of other nodes where the ZooKeeper instance is located can be pinged successfully.
   • If yes, go to 9.
   • If no, go to 6.

6. Modify the IP addresses in /etc/hosts and add the host name and IP address mapping.
7. Run the ping command again to check whether the host names of other nodes where the ZooKeeper instance is located can be pinged successfully.
   • If yes, go to 8.
   • If no, go to 23.

8. On the O&M > Alarm > Alarms tab, check whether the alarm is cleared.
   • If yes, no further action is required.
   • If no, go to 9.

Check the KrbServer service status (Skip this step if the normal mode is used).

9. On FusionInsight Manager, choose Cluster > Name of the desired cluster > Services.
10. Check whether the KrbServer service is normal.
    • If yes, go to 13.
    • If no, go to 11.

11. Perform operations based on "ALM-25500 KrbServer Service Unavailable" and check whether the KrbServer service is recovered.
    • If yes, go to 12.
    • If no, go to 23.

12. On the O&M > Alarm > Alarms tab, check whether the alarm is cleared.
    • If yes, no further action is required.
    • If no, go to 13.

Check the ZooKeeper service instance status.

13. On FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > ZooKeeper > quorumpeer.
14. Check whether the ZooKeeper instances are normal.
    • If yes, go to 18.
    • If no, go to 15.

15. Select instances whose status is not good, and choose More > Restart Instance.
16. Check whether the instance status is good after restart.
    • If yes, go to 17.
    • If no, go to 18.

17. On the O&M > Alarm > Alarms tab, check whether the alarm is cleared.
    • If yes, no further action is required.
    • If no, go to 18.

Check disk status.

18. On FusionInsight Manager, choose Cluster > Name of the desired cluster > Service > ZooKeeper > quorumpeer, and check the node host information of the ZooKeeper instance.
19. On FusionInsight Manager, click Host.
20. In the Disk column, check whether the disk space of each node where ZooKeeper instances are located is insufficient (disk usage exceeds 80%).
    • If yes, go to 21.
    • If no, go to 23.

21. Expand disk capacity. For details, see "ALM-12017 Insufficient Disk Capacity".
22. On the O&M > Alarm > Alarms tab, check whether the alarm is cleared.
    • If yes, no further action is required.
    • If no, go to 23.

Collect fault information.

23. On the FusionInsight Manager portal, choose O&M > Log > Download.
24. Select the following nodes in the required cluster from the Service: (KrbServer logs do not need to be downloaded in normal mode.)
    • ZooKeeper
    • KrbServer

25. Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
26. Contact the O&M engineers and send the collected log information.

Alarm Clearance

After the fault is rectified, the system automatically clears this alarm.

Related Information

None.