Help Center > > User Guide> Managing Active Clusters> Alarm Reference> ALM-13000 ZooKeeper Service Unavailable

ALM-13000 ZooKeeper Service Unavailable

Updated at: Mar 31, 2020 GMT+08:00

Description

The system checks the ZooKeeper service status every 30 seconds. This alarm is generated when the ZooKeeper service is unavailable and is cleared when the ZooKeeper service recovers.

Attribute

Alarm ID

Alarm Severity

Automatically Cleared

13000

Critical

Yes

Parameters

Parameter

Description

ServiceName

Specifies the service for which the alarm is generated.

RoleName

Specifies the role for which the alarm is generated.

HostName

Specifies the host for which the alarm is generated.

Impact on the System

  • ZooKeeper fails to provide coordination services for upper-layer components.
  • Components dependent on ZooKeeper may not run properly.

Possible Causes

  • A ZooKeeper instance is abnormal.
  • The disk capacity is insufficient.
  • The network is faulty.
  • The DNS is installed on the ZooKeeper node.

Procedure

Check the ZooKeeper service instance status.

  1. On the MRS cluster details page, choose Components > ZooKeeper > quorumpeer.

    For MRS 1.8.10 or earlier, log in to MRS Manager and choose Services > ZooKeeper > quorumpeer.

  2. Check whether the ZooKeeper instances are normal.

    • If yes, go to 6.
    • If no, go to 3.

  3. Select instances whose status is not good and choose More > Restart Instance.
  4. Check whether the instance status is good after restart.

    • If yes, go to 5.
    • If no, go to 19.

  5. On the Alarms tab, check whether the alarm is cleared.

    • If yes, no further action is required.
    • If no, go to 6.

    Check the disk status.

  6. On the MRS cluster details page, choose Components > ZooKeeper > quorumpeer, and check the host information of the ZooKeeper instance on each node.

    For MRS 1.8.10 or earlier, log in to MRS Manager, choose Services > ZooKeeper > quorumpeer and check the host information of the ZooKeeper instance on each node.

  7. On the MRS cluster details page, click Nodes.

    For MRS 1.8.10 or earlier, log in to MRS Manager and click Hosts.

  8. In the Disk Usage column, check whether the disk space of each node that contains ZooKeeper instances is insufficient (where disk usage exceeds 80%).

    • If yes, go to 9.
    • If no, go to 11.

  9. Expand disk capacity. For details, see ALM-12017 Insufficient Disk Capacity.
  10. On the Alarms tab, check whether the alarm is cleared.

    • If yes, no further action is required.
    • If no, go to 11.

    Check the network status.

  11. On the Linux node that contains the ZooKeeper instance, run the ping command to check whether the host names of other nodes that contain the ZooKeeper instance can be pinged successfully.

    • If yes, go to 15.
    • If no, go to 12.

  12. Modify the IP addresses in /etc/hosts and add the host name and IP address mapping.
  13. Run the ping command again to check whether the host names of other nodes that contain the ZooKeeper instance can be pinged successfully.

    • If yes, go to 14.
    • If no, go to 19.

  14. On the Alarms tab, check whether the alarm is cleared.

    • If yes, no further action is required.
    • If no, go to 15.

    Check the DNS.

  15. Check whether the DNS is installed on the node that contains the ZooKeeper instance. On the Linux node that contains the ZooKeeper instance, run the cat /etc/resolv.conf command to check whether the file is empty.

    • If yes, go to 16.
    • If no, go to 19.

  16. Run the service named status command to check whether the DNS is started.

    • If yes, go to 17.
    • No, go to 19.

  17. Run the service named stop command to stop the DNS service. If "Shutting down name server BIND waiting for named to shut down (28s)" is displayed, the DNS service is stopped successfully. Comment out any content in /etc/resolv.conf.
  18. On the Alarms tab, check whether the alarm is cleared.

    • If yes, no further action is required.
    • If no, go to 19.

  19. Collect fault information.

    1. On MRS Manager, choose System > Export Log.
    2. Contact the O&M personnel and send the collected log information.

Related Information

N/A

Did you find this page helpful?

Submit successfully!

Thank you for your feedback. Your feedback helps make our documentation better.

Failed to submit the feedback. Please try again later.

Which of the following issues have you encountered?







Please complete at least one feedback item.

Content most length 200 character

Content is empty.

OK Cancel