ALM-45736 Guardian Service Unavailable

Alarm Description

The alarm module checks the Guardian service status every 60 seconds. This alarm is generated if Guardian is unavailable.

This alarm is cleared after Guardian recovers.

Alarm Attributes

Alarm Parameters

Impact on the System

Guardian cannot work properly, and OBS cannot be accessed.

Possible Causes

• The HDFS or Zookeeper service on which the Guardian service depends is abnormal.
• The TokenServer role instance is abnormal.

Handling Procedure

Check the HDFS service status.

1. Log in to FusionInsight Manager and choose O&M > Alarm > Alarms. On the page that is displayed, check whether "ALM-14000 HDFS Service Unavailable" is reported.
   • If yes, go to 2.
   • If no, go to 3.

2. Clear "ALM-14000 HDFS Service Unavailable" according to the alarm help.
   After the alarm is cleared, wait a few minutes and check whether the alarm GuardianService Unavailable is cleared.

   • If yes, no further action is required.
   • If no, go to 3.

Check the Zookeeper status.

3. Log in to FusionInsight Manager and choose O&M > Alarm > Alarms. On the page that is displayed, check whether "ALM-13000 ZooKeeper Service Unavailable" is reported.
   • If yes, go to 4.
   • If no, go to 5.

4. Clear "ALM-13000 ZooKeeper Service Unavailable" according to the alarm help.
   After those alarms are cleared, wait a few minutes and check whether this alarm is cleared.

   • If yes, no further action is required.
   • If no, go to 5.

Check all TokenServer instances.

5. Log in to the node where the TokenServer instance resides as user omm and run the ps -ef|grep "guardian.token.server.Server" command to check whether the TokenServer process exists on the node.
   • If yes, go to 7.
   • If no, restart the faulty TokenServer instance and go to 6.

6. In the alarm list, check whether the alarm "Guardian Service Unavailable" is cleared.
   • If yes, no further action is required.
   • If no, go to 7.

Collect fault information.

7. On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
8. Expand the Service drop-down list, and select Guardian for the destination cluster.
9. Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.
10. Contact O&M engineers and provide the collected logs.

Alarm Clearance

This alarm is automatically cleared after the fault is rectified.

Related Information

None.