ALM-45001 Faulty HetuEngine Compute Instances

Alarm Description

The system checks the HetuEngine compute instance status every 60 seconds. This alarm is generated when a HetuEngine compute instance is faulty.

This alarm is cleared when all faulty HetuEngine compute instances are restored.

Alarm Attributes

Alarm Parameters

Impact on the System

SQL tasks submitted to the faulty compute instance of HetuEngine will fail.

Possible Causes

• The HDFS service is abnormal.
• The Yarn service is abnormal.
• Yarn queue resources are insufficient.
• The process of compute instances is faulty.
• The number of Worker progresses in the faulty compute instance is 0.

Handling Procedure

Check the HDFS service status.

1. In the alarm list, check whether the "ALM-14000 HDFS Service Unavailable" alarm is generated.
   • If yes, go to Step 2.
   • If no, go to Step 4.

2. Clear the alarm according to the alarm help in ALM-14000 HDFS Service Unavailable.
3. In the alarm list, check whether the "ALM-45001 Faulty HetuEngine Compute Instances" alarm is cleared.
   • If yes, no further action is required.
   • If no, go to Step 4.

Check the YARN service status.

4. In the alarm list, check whether the "ALM-18000 YARN Service Unavailable" alarm is generated.
   • If yes, go to Step 5.
   • If no, go to Step 7.

5. Clear the alarm according to the alarm help in ALM-18000 YARN Service Unavailable.
6. In the alarm list, check whether the "ALM-45001 Faulty HetuEngine Compute Instances" alarm is cleared.
   • If yes, no further action is required.
   • If no, go to Step 7.

Check the YARN queue resource status.

7. In the alarm list, check whether the "ALM-18022 Insufficient YARN Queue Resources" alarm is generated.
   • If yes, go to 8.
   • If no, go to Step 10.

8. Clear the alarm according to the alarm help in ALM-18022 Insufficient YARN Queue Resources.
9. In the alarm list, check whether the "ALM-45001 Faulty HetuEngine Compute Instances" alarm is cleared.
   • If yes, no further action is required.
   • If no, go to Step 10.

Check the HetuEngine compute instance status.

10. Log in to FusionInsight Manager as an administrator who can access the HetuEngine web UI and choose Cluster > Services > HetuEngine.
11. In the Basic Information area on the Dashboard tab page, click the link next to HSConsole WebUI to access the HSConsole page.
12. On the compute instance page, check whether any compute instances are in the FAULT state.
    • If yes, go to Step 4.
    • If no, go to Step 3.

13. Check whether there are compute instances with no Worker progress.
    • If yes, go to Step 14.
    • If no, go to Step 15.

14. In the Operation column of the target compute instance, click Start and wait until the instance is started.
15. In the alarm list, check whether the "ALM-45001 Faulty HetuEngine Compute Instances" alarm is cleared.
    • If yes, no further action is required.
    • If no, go to Step 16.

Collect fault information.

16. On FusionInsight Manager, choose O&M > Log > Download.
17. Expand the Service drop-down list. In the Services dialog box that is displayed, select HetuEngine under the target cluster name, and click OK.
18. Expand the Hosts drop-down list. In the Select Host dialog box that is displayed, select the hosts to which the role belongs, and click OK.
19. Click in the upper right corner, and set Start Date and End Date for log collection to 30 minutes ahead of and after the alarm generation time respectively. Then, click Download.
20. Send the collected fault logs to O&M personnel for help.

Alarm Clearance

This alarm is automatically cleared after the fault is rectified.

Related Information

None