ALM-14000 HDFS Service Unavailable

Description

The system checks the NameService service status every 60 seconds. This alarm is generated when all the NameService services are abnormal and the system considers that the HDFS service is unavailable.

This alarm is cleared when at least one NameService service is normal and the system considers that the HDFS service recovers.

Attribute

Parameters

Impact on the System

HDFS fails to provide services for HDFS service-based upper-layer components, such as HBase and MapReduce. As a result, users cannot read or write files.

Possible Causes

• The ZooKeeper service is abnormal.
• All NameService services are abnormal.

• The number of service requests is too large, and the HDFS health check fails to read and write files.
• The health check fails due to HDFS FullGC.

Procedure

Check the ZooKeeper service status.

1. On the FusionInsight Manager portal, choose O&M > Alarm > Alarms. On the Alarm page, check whether ALM-13000 ZooKeeper Service Unavailable is reported.
   • If yes, go to Step 2.
   • If no, go to Step 4.

2. Rectify the ZooKeeper service fault by following the instructions in ALM-13000 ZooKeeper Service Unavailable. Then check whether the running status of the ZooKeeper service is normal.
   • If yes, go to Step 3.
   • If no, go to Step 13.

3. On the O&M > Alarm > Alarms page, check whether the alarm is cleared.
   • If yes, no further action is required.
   • If no, go to Step 4.

Handle the NameService service exception alarm.

4. On the FusionInsight Manager portal, choose O&M > Alarm > Alarms. On the Alarms page, check whether ALM-14010 NameService Service Unavailable is reported.
   • If yes, go to Step 5.
   • If no, go to Step 7.

5. Handle the abnormal NameService services by referring to the handling method provided in ALM-14010 NameService Service Is Abnormal. Then, check whether the alarm is cleared.
   • If yes, go to Step 6.
   • If no, go to Step 13.

6. On the O&M > Alarm > Alarms page, check whether the alarm is cleared.
   • If yes, no further action is required.
   • If no, go to Step 7.

Check whether the HDFS health check fails to read or write files due to a large number of service requests.

7. On FusionInsight Manager, choose O&M > Alarm > Alarms, and check whether ALM-14021 NameNode Average RPC Processing Time Exceeds the Threshold or ALM-14022 NameNode Average RPC Queuing Time Exceeds the Threshold is generated.
   • If yes, go to Step 8.
   • If no, go to Step 10.

8. Handle the abnormal NameService services by following the instructions provided in ALM-14021 NameNode Average RPC Processing Time Exceeds the Threshold and ALM-14022 NameNode Average RPC Queuing Time Exceeds the Threshold, and check whether the alarm is cleared for each NameService.
   • If yes, go to Step 9.
   • If no, go to Step 13.

9. On the O&M > Alarm > Alarms page, check whether the alarm is cleared.
   • If yes, no further action is required.
   • If no, go to Step 10.

Check whether the health check fails due to HDFS FullGC.

10. On the FusionInsight Manager portal, choose O&M > Alarm > Alarms. On the Alarms page, check whether ALM-14014 NameNode GC Time Exceeds the Threshold is reported.
    • If yes, go to Step 11.
    • If no, go to Step 13.

11. Handle the abnormal NameService services by referring to the handling method provided in ALM-14014 NameNode GC Time Exceeds the Threshold. Then check whether the alarm is cleared for each NameService service.
    • If yes, go to Step 12.
    • If no, go to Step 13.

12. On the O&M > Alarm > Alarms page, check whether the alarm is cleared.
    • If yes, no further action is required.
    • If no, go to Step 13.

Collect fault information.

13. On the FusionInsight Manager portal, choose O&M > Log > Download.
14. Select the following nodes in the required cluster from the Service:
    • ZooKeeper
    • HDFS

15. Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
16. Contact the O&M personnel and send the collected logs.

Alarm Clearing

After the fault is rectified, the system automatically clears this alarm.

Related Information

None