ALM-16004 Hive Service Unavailable

Alarm Description

The system checks the Hive service status every 60 seconds. This alarm is generated when the Hive service is unavailable.

This alarm is cleared when the Hive service recovers.

Alarm Attributes

Alarm Parameters

Impact on the System

The system cannot provide data loading, query, and extraction services.

Possible Causes

• Hive service unavailability may be related to the faults of the Hive process as well as basic services, such as ZooKeeper, Hadoop distributed file system (HDFS), YARN, and DBService.
  • The ZooKeeper service is abnormal.
  • The HDFS service is abnormal.
  • The YARN service is abnormal.
  • The DBService service is abnormal.
  • The Hive service process is faulty. If the alarm is caused by the Hive process fault, the alarm reporting may be delayed for about 5 minutes.
• The network communication between the Hive service and basic services is interrupted.
• The permission on the HDFS temporary directory of Hive is abnormal.
• The local disk space of the Hive node is insufficient.

Handling Procedure

Check the HiveServer/MetaStore process status.

1. On Manager, click Cluster > Services > Hive > Instances. On the displayed page, check whether there are HiveServer or MetaStore instances in the Unknown state.

   For details about how to log in to FusionInsight Manager, see Accessing MRS Manager.

   • If yes, go to Step 2.
   • If no, go to Step 4.

2. Above the Hive instance list, choose More > Restart Instance to restart the HiveServer/MetaStore process.
   

   The HiveServer/MetaStore instance that is being restarted cannot provide services for external systems, and the SQL tasks being executed on the instance may fail.

3. In the alarm list, check whether Hive Service Unavailable is cleared.
   • If yes, no further action is required.
   • If no, go to Step 4.

Check the ZooKeeper service status.

4. In the alarm list on Manager, check whether there are Process Fault alarms.
   • If yes, go to Step 5.
   • If no, go to Step 8.

5. Check whether the Process Fault alarm is generated for ZooKeeper in the alarm location information.
   • If yes, go to Step 6.
   • If no, go to Step 8.

6. Rectify the fault by following the steps provided in ALM-12007 Process Fault.
7. In the alarm list, check whether Hive Service Unavailable is cleared.
   • If yes, no further action is required.
   • If no, go to Step 8.

Check the HDFS service status.

8. In the alarm list, check whether there is an HDFS Service Unavailable alarm.
   • If yes, go to Step 9.
   • If no, go to Step 11.

9. Rectify the fault by following the steps provided in ALM-14000 HDFS Service Unavailable.
10. In the alarm list, check whether Hive Service Unavailable is cleared.
    • If yes, no further action is required.
    • If no, go to Step 11.

Check the YARN service status.

11. In the alarm list, check whether there is a Yarn Service Unavailable alarm.
    • If yes, go to Step 12.
    • If no, go to Step 14.

12. Rectify the fault by following the steps provided in ALM-18000 Yarn Service Unavailable.
13. In the alarm list, check whether Hive Service Unavailable is cleared.
    • If yes, no further action is required.
    • If no, go to Step 14.

Check the DBService service status.

14. In the alarm list, check whether there is a DBService Service Unavailable alarm.
    • If yes, go to Step 15.
    • If no, go to Step 17.

15. Rectify the fault. For details, see ALM-27001 DBService Service Unavailable.
16. In the alarm list, check whether Hive Service Unavailable is cleared.
    • If yes, no further action is required.
    • If no, go to Step 17.

Check the network connection between the Hive and ZooKeeper, HDFS, Yarn, and DBService.

17. On the Manager homepage, choose Cluster > Services > Hive.
18. Click Instances to go to the Hive instance list page.
19. Click the host name in the row where HiveServer is located to go to the HiveServer host status page.
20. Record the IP address in the Basic Information area.
21. Log in to the host where HiveServer resides as user omm using the IP address obtained in Step 20.

22. Run the ping command to check whether the network connection between the host that runs HiveServer and the hosts that run the ZooKeeper, HDFS, Yarn, and DBService services is normal. (Obtain the IP addresses of the hosts that run the ZooKeeper, HDFS, Yarn, and DBService services in the same way as that for obtaining the IP address of the HiveServer.)
    • If yes, go to Step 25.
    • If no, go to Step 23.

23. Contact the network administrator to restore the network.
24. In the alarm list, check whether Hive Service Unavailable is cleared.
    • If yes, no further action is required.
    • If no, go to Step 25.

Check the permission on the HDFS temporary directory.

25. Log in to the node where the HDFS client is located and run the following command to go to the HDFS client installation directory:

    Go to the client installation directory.

    cd Client installation directory

    Configure environment variables.

    source bigdata_env

    If Kerberos authentication is enabled for the cluster (in security mode), run the following command to authenticate the user:

    kinit User with the permissions of the supergroup group

26. Run the following command to check whether the permission on the data warehouse directory is 770:

    hdfs dfs -ls /tmp | grep hive-scratch

    • If yes, go to Step 29.
    • If no, go to Step 27.

27. Run the following command to restore the default data warehouse permission:

    hdfs dfs -chmod 770 /tmp/hive-scratch

28. Wait for several minutes and check whether the "Hive Service Unavailable" alarm is cleared.
    • If yes, no further action is required.
    • If no, go to Step 29.

Check whether the local disk space is normal.

29. Run the following command to check whether the disk usage of the /srv, /var, and /opt directories exceeds 95% in the root directory.

    df -h

    • If yes, go to Step 30.
    • If no, go to Step 31.

30. Clear unnecessary information in the corresponding directory to ensure that the available disk space is greater than 80%. Wait for several minutes and check whether the "Hive Service Unavailable" alarm is cleared.
    • If yes, no further action is required.
    • If no, go to Step 31.

Collect fault information.

31. On Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
32. In the Service area, select the following nodes of the desired cluster.
    • ZooKeeper
    • HDFS
    • Yarn
    • DBService
    • Hive

33. Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes before and after the alarm generation time, respectively. Then, click Download.
34. Send the collected fault logs to O&M personnel for help.

Alarm Clearance

This alarm is automatically cleared after the fault is rectified.

Related Information

None