ALM-45001 Faulty HetuEngine Compute Instances

Alarm Description

The system checks the HetuEngine compute instance status every 60 seconds. This alarm is generated when a HetuEngine compute instance is faulty.

This alarm is cleared when all faulty HetuEngine compute instances are restored.

This section applies only to MRS 3.2.0-LTS or later.

Alarm Attributes

Alarm Parameters

Impact on the System

SQL tasks submitted to the faulty compute instance of HetuEngine will fail.

Possible Causes

• The HDFS service is abnormal.
• The Yarn service is abnormal.
• Yarn queue resources are insufficient.
• The process of compute instances is faulty.
• The number of Worker progresses in the faulty compute instance is 0.
• The compute instance resources are insufficient.

Handling Procedure

Check the HDFS service status.

1. On Manager, choose O&M > Alarm > Alarms. In the alarm list, check whether the "ALM-14000 HDFS Service Unavailable" alarm is reported.

   For details about how to log in to FusionInsight Manager, see Accessing MRS Manager.

   • If yes, go to Step 2.
   • If no, go to Step 4.

2. Clear the alarm according to the alarm help in ALM-14000 HDFS Service Unavailable.
3. In the alarm list, check whether the "ALM-45001 Faulty HetuEngine Compute Instances" alarm is cleared.
   • If yes, no further action is required.
   • If no, go to Step 4.

Check the YARN service status.

4. In the alarm list, check whether the "ALM-18000 YARN Service Unavailable" alarm is generated.
   • If yes, go to Step 5.
   • If no, go to Step 7.

5. Clear the alarm according to the alarm help in ALM-18000 YARN Service Unavailable.
6. In the alarm list, check whether the "ALM-45001 Faulty HetuEngine Compute Instances" alarm is cleared.
   • If yes, no further action is required.
   • If no, go to Step 7.

Check the YARN queue resource status.

7. In the alarm list, check whether the "ALM-18022 Insufficient YARN Queue Resources" alarm is generated.
   • If yes, go to 8.
   • If no, go to Step 10.

8. Clear the alarm according to the alarm help in ALM-18022 Insufficient YARN Queue Resources.
9. In the alarm list, check whether the "ALM-45001 Faulty HetuEngine Compute Instances" alarm is cleared.
   • If yes, no further action is required.
   • If no, go to Step 10.

Check the HetuEngine compute instance status.

10. Log in to Manager as an administrator who can access the HetuEngine web UI and choose Cluster > Services > HetuEngine.
11. In the Basic Information area on the Dashboard page, click the link next to HSConsole WebUI to access the HSConsole page.
12. On the Compute Instance page, check whether any compute instances are in the FAULT state.
    • If yes, go to Step 13.
    • If no, go to Step 15.

13. Select the compute instance in the FAULT state, click Restart, and wait until the instance restarts.
14. In the alarm list, check whether the "ALM-45001 Faulty HetuEngine Compute Instances" alarm is cleared.
    • If yes, no further action is required.
    • If no, go to Step 15.

Check whether the number of Workers in a HetuEngine compute instance is 0.

15. On the Compute Instance page, click next to the tenant name and check whether there are compute instances with no Workers.
    • If yes, go to Step 16.
    • If no, go to Step 18.

16. In the Operation column, click Restart and wait until the instance starts.
17. In the alarm list, check whether the "ALM-45001 Faulty HetuEngine Compute Instances" alarm is cleared.
    • If yes, no further action is required.
    • If no, go to Step 18.

Check whether compute instance resources are properly configured.

18. Log in to Manager as an administrator who can access the HetuEngine web UI.
19. Choose O&M > Alarm > Alarms > Tasks Stacked on HetuEngine Compute Instance, check the Additional Information of the alarm, and view and record the tenant name for which the alarm is generated.
20. Choose Cluster > Services > HetuEngine. In the Basic Information area on the Dashboard page, click the link next to HSConsole Web UI. The HSConsole page is displayed.
21. On the Compute Instance page, click Configure in the Operation column of the tenant to which the compute instance belongs. Check whether the resources configured for the compute instance are proper. (The minimum resources are used by default. You can adjust the configuration based on site requirements.)
    • If yes, go to Step 25.
    • If no, go to Step 22.

22. Return to the compute instance list, click Stop Instances in the Operation column, and stop instances as prompted.
    

    Tasks submitted to the stopped compute instances will be interrupted.

23. Click Configure, add resources to the target compute instance based on site requirements, and click OK. Click Start Instances and start instances as prompted.
24. After 2 minutes, check whether the alarm is cleared.
    • If yes, no further action is required.
    • If no, go to Step 25.

Collect fault information.

25. On Manager, choose O&M > Log > Download.
26. Expand the Service drop-down list. In the Services dialog box that is displayed, select HetuEngine under the target cluster name, and click OK.
27. Expand the Hosts drop-down list. In the Select Host dialog box that is displayed, select the hosts to which the role belongs, and click OK.
28. Click in the upper right corner, and set Start Date and End Date for log collection to 30 minutes ahead of and after the alarm generation time respectively. Then, click Download.
29. Send the collected fault logs to O&M personnel for help.

Alarm Clearance

This alarm is automatically cleared after the fault is rectified.

Related Information

None