ALM-16004 Hive Service Unavailable

Description

This alarm is generated when the HiveServer service is unavailable. The system checks the HiveServer service status every 60 seconds.

This alarm is cleared when the HiveServer service is normal.

Attribute

Parameters

Impact on the System

The system cannot provide data loading, query, and extraction services.

Possible Causes

• Hive service unavailability may be related to the faults of the Hive process as well as basic services, such as ZooKeeper, Hadoop distributed file system (HDFS), Yarn, and DBService.
  • The ZooKeeper service is abnormal.
  • The HDFS service is abnormal.
  • The Yarn service is abnormal.
  • The DBService service is abnormal.
  • The Hive service process is abnormal. If the alarm is caused by Hive process fault, the alarm report has a delay of about 5 minutes.
• The network communication between the Hive and basic services is interrupted.
• The permission on the HDFS temporary directory of Hive is abnormal.
• The local disk space of the Hive node is insufficient.

Procedure

Check the HiveServer/MetaStore process status.

1. On the FusionInsight Manager portal, click Cluster > Name of the desired cluster > Services > Hive > Instance. In the Hive instance list, check whether the HiveServer or MetaStore instances are in the Unknown state.
   • If yes, go to 2.
   • If no, go to 4.

2. In the Hive instance list, choose More > Restart Instance to restart the HiveServer/MetaStore process.
   

   During HiveServer or MetaStore instance restart, the instance cannot provide services for external systems. SQL tasks that are being executed on the instance may fail.

3. In the alarm list, check whether Hive Service Unavailable is cleared.
   • If yes, no further action is required.
   • If no, go to 4.

Check the ZooKeeper service status.

4. On the FusionInsight Manager, check whether the alarm list contains Process Fault.
   • If yes, go to 5.
   • If no, go to 8.

5. In the Process Fault, check whether ServiceName is ZooKeeper.
   • If yes, go to 6.
   • If no, go to 8.

6. Rectify the fault by following the steps provided in "ALM-12007 Process Fault".
7. In the alarm list, check whether Hive Service Unavailable is cleared.
   • If yes, no further action is required.
   • If no, go to 8.

Check the HDFS service status.

8. On the FusionInsight Manager, check whether the alarm list contains HDFS Service Unavailable.
   • If yes, go to 9.
   • If no, go to 11.

9. Rectify the fault by following the steps provided in "ALM-14000 HDFS Service Unavailable".
10. In the alarm list, check whether Hive Service Unavailable is cleared.
    • If yes, no further action is required.
    • If no, go to 11.

Check the Yarn service status.

11. In FusionInsight Manager alarm list, check whether Yarn Service Unavailable is generated.
    • If yes, go to 12.
    • If no, go to 14.

12. Rectify the fault. For details, see "ALM-18000 Yarn Service Unavailable".
13. In the alarm list, check whether Hive Service Unavailable is cleared.
    • If yes, no further action is required.
    • If no, go to 14.

Check the DBService service status.

14. In FusionInsight Manager alarm list, check whether DBService Service Unavailable is generated.
    • If yes, go to 15.
    • If no, go to 17.

15. Rectify the fault. For details, see "ALM-27001 DBService Service Unavailable".
16. In the alarm list, check whether Hive Service Unavailable is cleared.
    • If yes, no further action is required.
    • If no, go to 17.

Check the network connection between the Hive and ZooKeeper, HDFS, Yarn, and DBService.

17. On the FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > Hive.
18. Click Instance.

    The HiveServer instance list is displayed.

19. Click Host Name in the row of HiveServer.

    The active HiveServer host status page is displayed.

20. Record the IP address under Basic Information.
21. Use the IP address obtained in 20 to log in to the host where the active HiveServer runs as user omm.

22. Run the ping command to check whether communication between the host that runs the active HiveServer and the hosts that run the ZooKeeper, HDFS, Yarn, and DBService services is normal. (Obtain the IP addresses of the hosts that run the ZooKeeper, HDFS, Yarn, and DBService services in the same way as that for obtaining the IP address of the active HiveServer.)
    • If yes, go to 31.
    • If no, go to 23.

23. Contact the administrator to restore the network.
24. In the alarm list, check whether Hive Service Unavailable is cleared.
    • If yes, no further action is required.
    • If no, go to 31.

Check the permission on the HDFS temporary directory.

25. Log in to the node where the HDFS client is located and run the following command to go to the HDFS client installation directory:

    cd Client installation directory

    source bigdata_env

    kinit user with the supergroup permission (Skip this step for common clusters.)

26. Run the following command to check whether the permission on the data warehouse directory is 770:

    hdfs dfs -ls /tmp | grep hive-scratch

    • If yes, go to 29.
    • If no, go to 27.

27. Run the following command to restore the default data warehouse permission:

    hdfs dfs -chmod 770 /tmp/hive-scratch

28. Wait for several minutes and check whether the Hive Service Unavailable alarm is cleared.
    • If yes, no further action is required.
    • If no, go to 29.

Check whether the local disk space is normal.

29. Run the df -h command to check the root directory and check whether the disk usage of the /srv, /var, and /opt directories exceeds 95%.
    • If yes, go to 30.
    • If no, go to 31.

30. Clear unnecessary information in the corresponding directory to ensure that the available disk space is greater than 80%. Wait for several minutes and check whether the Hive Service Unavailable alarm is cleared.
    • If yes, no further action is required.
    • If no, go to 31.

Collect fault information.

31. On the FusionInsight Manager, choose O&M > Log > Download.
32. Select the following nodes in the required cluster from the Service:
    • ZooKeeper
    • HDFS
    • Yarn
    • DBService
    • Hive

33. Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
34. Contact the O&M personnel and send the collected logs.

Alarm Clearing

After the fault is rectified, the system automatically clears this alarm.

Related Information

None