Help Center > > User Guide> MRS Manager Operation Guide> Alarm Reference> ALM-13000 ZooKeeper Service Unavailable

ALM-13000 ZooKeeper Service Unavailable

Updated at: Dec 31, 2019 GMT+08:00

Description

The system checks the ZooKeeper service status every 30 seconds. This alarm is generated when the ZooKeeper service is unavailable and is cleared when the ZooKeeper service recovers.

Attribute

Alarm ID

Alarm Severity

Automatically Cleared

13000

Critical

Yes

Parameters

Parameter

Description

ServiceName

Specifies the service for which the alarm is generated.

RoleName

Specifies the role for which the alarm is generated.

HostName

Specifies the host for which the alarm is generated.

Impact on the System

  • ZooKeeper fails to provide coordination services for upper-layer components.
  • Components dependent on ZooKeeper may not run properly.

Possible Causes

  • A ZooKeeper instance is abnormal.
  • The disk capacity is insufficient.
  • The network is faulty.
  • The DNS is installed on the ZooKeeper node.

Procedure

Check the ZooKeeper service instance status.

  1. On MRS Manager, choose Services > ZooKeeper > quorumpeer.
  2. Check whether the ZooKeeper instances are normal.

    • If yes, go to 6.
    • If no, go to 3.

  3. Select instances whose status is not good and choose More > Restart Instance.
  4. Check whether the instance status is good after restart.

    • If yes, go to 5.
    • If no, go to 19.

  5. On the Alarms tab, check whether the alarm is cleared.

    • If yes, no further action is required.
    • If no, go to 6.

    Check the disk status.

  6. On MRS Manager, choose Services > ZooKeeper > quorumpeer, and check the host information of the ZooKeeper instance on each node.
  7. On MRS Manager, click Hosts.
  8. In the Disk Usage column, check whether the disk space of each node that contains ZooKeeper instances is insufficient (where disk usage exceeds 80%).

    • If yes, go to 9.
    • If no, go to 11.

  9. Expand disk capacity. For details, see ALM-12017 Insufficient Disk Capacity.
  10. On the Alarms tab, check whether the alarm is cleared.

    • If yes, no further action is required.
    • If no, go to 11.

    Check the network status.

  11. On the Linux node that contains the ZooKeeper instance, run the ping command to check whether the host names of other nodes that contain the ZooKeeper instance can be pinged successfully.

    • If yes, go to 15.
    • If no, go to 12.

  12. Modify the IP addresses in /etc/hosts and add the host name and IP address mapping.
  13. Run the ping command again to check whether the host names of other nodes that contain the ZooKeeper instance can be pinged successfully.

    • If yes, go to 14.
    • If no, go to 19.

  14. On the Alarms tab, check whether the alarm is cleared.

    • If yes, no further action is required.
    • If no, go to 15.

    Check the DNS.

  15. Check whether the DNS is installed on the node that contains the ZooKeeper instance. On the Linux node that contains the ZooKeeper instance, run the cat /etc/resolv.conf command to check whether the file is empty.

    • If yes, go to 16.
    • If no, go to 19.

  16. Run the service named status command to check whether the DNS is started.

    • If yes, go to 17.
    • No, go to 19.

  17. Run the service named stop command to stop the DNS service. If "Shutting down name server BIND waiting for named to shut down (28s)" is displayed, the DNS service is stopped successfully. Comment out any content in /etc/resolv.conf.
  18. On the Alarms tab, check whether the alarm is cleared.

    • If yes, no further action is required.
    • If no, go to 19.

  19. Collect fault information.

    1. On MRS Manager, choose System > Export Log.
    2. Contact the O&M personnel and send the collected log information.

Related Information

N/A

Did you find this page helpful?

Submit successfully!

Thank you for your feedback. Your feedback helps make our documentation better.

Failed to submit the feedback. Please try again later.

Which of the following issues have you encountered?







Please complete at least one feedback item.

Content most length 200 character

Content is empty.

OK Cancel