ALM-45000 HetuEngine Service Unavailable

Alarm Description

The system checks the HetuEngine service status every 300 seconds. This alarm is generated when the HetuEngine service is unavailable.

This alarm is cleared when the HetuEngine service recovers.

Alarm Attributes

Alarm Parameters

Impact on the System

HetuEngine tasks fail to execute.

Possible Causes

• The KrbServer service is abnormal.
• The ZooKeeper service is abnormal.
• The HDFS service is abnormal.
• The Yarn service is abnormal.
• The DBService service is abnormal.
• The Hive service is abnormal.
• There are no HSBroker instances in HetuEngine.

Handling Procedure

Check the KrbServer service status.

1. On FusionInsight Manager, choose O&M > Alarm > Alarm.
2. In the alarm list, check whether the "ALM-25500 KrbServer Service Unavailable" alarm is generated.
   • If yes, go to 3.
   • If no, go to 5.

3. Clear "ALM-25500 KrbServer Service Unavailable" according to the alarm help.
4. In the alarm list, check whether the alarm "ALM-45000 HetuEngine Service Unavailable" is cleared.
   • If yes, no further action is required.
   • If no, go to 5.

Check the ZooKeeper service status.

5. In the alarm list, check whether the alarm "ALM-12007 Process Fault" is generated.
   • If yes, go to 6.
   • If no, go to 9.

6. In the alarm list, click in the row that contains the "Process Fault" alarm. Check whether the name of the service for which the alarm is generated is ZooKeeper in Location Information.
   • If yes, go to 7.
   • If no, go to 9.

7. Clear "ALM-12007 Process Fault" according to the alarm help.
8. In the alarm list, check whether the alarm "ALM-45000 HetuEngine Service Unavailable" is cleared.
   • If yes, no further action is required.
   • If no, go to 9.

Check the HDFS service status.

9. In the alarm list, check whether the "ALM-14000 HDFS Service Unavailable" alarm is generated.
   • If yes, go to 10.
   • If no, go to 12.

10. Clear "ALM-14000 HDFS Service Unavailable" according to the alarm help.
11. In the alarm list, check whether the "ALM-45000 HetuEngine Service Unavailable" alarm is cleared.
    • If yes, no further action is required.
    • If no, go to 12.

Check the YARN service status.

12. In the alarm list, check whether the "ALM-18000 YARN Service Unavailable" alarm is generated.
    • If yes, go to 13.
    • If no, go to 15.

13. Clear "ALM-18000 YARN Service Unavailable" according to the alarm help.
14. In the alarm list, check whether the "ALM-45000 HetuEngine Service Unavailable" alarm is cleared.
    • If yes, no further action is required.
    • If no, go to 15.

Check the DBService service status.

15. In the alarm list, check whether the "ALM-27001 DBService Service Unavailable" alarm is generated.
    • If yes, go to 16.
    • If no, go to 20.

16. Clear "ALM-27001 DBService Service Unavailable" according to the alarm help.
17. In the alarm list, check whether the "ALM-45000 HetuEngine Service Unavailable" alarm is cleared.
    • If yes, no further action is required.
    • If no, go to 20.

Check the Hive service status.

18. In the alarm list, check whether the "ALM-16004 Hive Service Unavailable" alarm is generated.
    • If yes, go to 19.
    • If no, go to 20.

19. Clear "ALM-16004 Hive Service Unavailable" according to the alarm help.
20. In the alarm list, check whether the "ALM-45000 HetuEngine Service Unavailable" alarm is cleared.
    • If yes, no further action is required.
    • If no, go to 20.

Check whether there are no HSBroker instances in HetuEngine.

21. On FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > HetuEngine. On the page that is displayed, click the Instance tab.
22. Check whether there are no HSBroker instances.
    • If yes, click Add Instance to add one.
    • If no, go to 23.

23. In the alarm list, check whether the "ALM-45000 HetuEngine Service Unavailable" alarm is cleared.
    • If yes, no further action is required.
    • If no, go to 23.

Check the network connection between HetuEngine and ZooKeeper, HDFS, YARN, DBService, and Hive.

24. On FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > HetuEngine. On the page that is displayed, click the Instance tab.
25. Click the host name in the HSBroker row and record the management IP address in the Basic Information area.
26. Log in to the host where HSBroker resides as user omm using the IP address obtained in 25.

27. Run the ping command to check whether the network connection between the host where HSBroker resides and the hosts where ZooKeeper, HDFS, Yarn, DBService, and Hive reside is in the normal state.
    • If yes, go to 30.
    • If no, go to 28.

28. Contact the network administrator to restore the network.
29. In the alarm list, check whether the "ALM-45000 HetuEngine Service Unavailable" alarm is cleared.
    • If yes, no further action is required.
    • If no, go to 30.

Collect fault information.

30. On FusionInsight Manager, choose O&M > Log > Download.
31. Expand the Service drop-down list. In the Services dialog box that is displayed, select HetuEngine under the target cluster name, and click OK.
32. Expand the Hosts drop-down list. In the Select Host dialog box that is displayed, select the hosts to which the role belongs, and click OK.
33. Click in the upper right corner, and set Start Date and End Date for log collection to 30 minutes ahead of and after the alarm generation time respectively. Then, click Download.
34. Contact O&M personnel and provide the collected logs.

Alarm Clearance

After the fault is rectified, the system automatically clears this alarm.

Reference

None