ALM-45009 Task Failure Rate of HetuEngine Compute Instances Exceeds the Threshold

Alarm Description

The system checks the task failure rate of HetuEngine compute instances every 30 seconds. This alarm is generated when the task failure rate of a HetuEngine compute instance is greater than or equal to 50%.

This alarm is cleared when the task failure rate is less than 50%.

Alarm Attributes

Alarm Parameters

Impact on the System

If the task failure rate of HetuEngine compute instances is too high, service running is adversely affected. You need to locate and rectify the fault in a timely manner.

Possible Causes

• The compute instance specification is too small.
• Large SQL tasks occupy too many compute resources. No resource is available for other tasks, and the compute instance cannot respond quickly. As a result, tasks are stacked.

Handling Procedure

Check whether compute instance resources are properly configured.

1. Log in to FusionInsight Manager as an administrator who can access the HetuEngine web UI.
2. Choose O&M > Alarm > Alarms. In the right pane, click alarm Task Failure Rate of HetuEngine Compute Instances Exceeds the Threshold, and view and record the tenant name in Additional Information.
3. Choose Cluster > Services > HetuEngine. In the Basic Information area on the Dashboard page, click the link next to HSConsole Web UI. The HSConsole page is displayed.
4. On the Compute Instance page, click Configure in the Operation column of the tenant to which the compute instance belongs. Check whether the resources configured for the compute instance are proper. (The the minimum resources are used by default. You can adjust the configuration based on site requirements.)
   • If yes, go to 8.
   • If no, go to 5.

5. Return to the compute instance list, click Stop Instances in the Operation column, and stop instances as prompted.
   

   Tasks submitted to the stopped compute instances will be interrupted.

6. Click Configure, add resources to the target compute instance based on site requirements, and click OK. Click Start Instances and start instances as prompted.
7. Wait 2 minutes and check whether the alarm is cleared.
   • If yes, no further action is required.
   • If no, go to 8.

Check whether there are large SQL tasks.

8. On the Compute Instances page, expand the instances of the tenant and click LINK in the WebUI column of a compute instance to view the status of all tasks.
9. In the Sort column, select Execution Time to sort the running tasks and check whether there are tasks that have been running for hours.
   • If yes, go to 10.
   • If no, select all failed tasks in the Failed column, collect detailed error information about SQL running in the computing instance, and go to 12.

10. Stop the tasks that have been running for a long time based on service requirement and optimize the service SQL statements.
11. Wait 2 minutes and check whether the alarm is cleared.
    • If yes, no further action is required.
    • If no, select all failed tasks in the Failed column, collect detailed error information about SQL running in the computing instance, and go to 12.

Collect fault information.

12. On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
13. Expand the Service drop-down list, select HetuEngine for the target cluster, and click OK.
14. Expand the Hosts drop-down list. In the Select Host dialog box that is displayed, select the hosts to which the role belongs, and click OK.
15. Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
16. Contact O&M engineers and provide the collected logs.

Alarm Clearance

This alarm is automatically cleared after the fault is rectified.

Related Information

None.