ALM-16002 Hive SQL Execution Success Rate Is Lower Than the Threshold

Updated on 2022-08-12 GMT+08:00

View PDF

Description

The system checks the percentage of the HQL statements that are executed successfully in every 30 seconds. The formula is: Percentage of HQL statements that are executed successfully = Number of HQL statements that are executed successfully by Hive in a specified period/Total number of HQL statements that are executed by Hive. This indicator can be viewed on the Cluster > Name of the desired cluster > Services > Hive > Instance > HiveServer instance . The default threshold of the percentage of HQL statements that are executed successfully is 90%. An alarm is reported when the percentage is lower than the 90%. Users can view the name of the host where an alarm is generated in the location information about the alarm. The IP address of the host is the IP address of the HiveServer node.

Users can modify the threshold by choosing O&M > Alarm > Thresholds > Name of the desired cluster > Hive > Percentage of HQL Statements That Are Executed Successfully by Hive.

This alarm is cleared when the execution success rate is higher than 110% of the threshold.

Attribute

Alarm ID	Alarm Severity	Automatically Cleared
16002	Major	Yes

Parameters

Name	Meaning
Source	Specifies the cluster for which the alarm is generated.
ServiceName	Specifies the service for which the alarm is generated.
RoleName	Specifies the role for which the alarm is generated.
HostName	Specifies the host for which the alarm is generated.
Trigger Condition	Specifies the threshold triggering the alarm. If the current indicator value exceeds this threshold, the alarm is generated.

Impact on the System

The system configuration and performance cannot meet service processing requirements.

Possible Causes

A syntax error occurs in HQL statements.
The HBase service is abnormal when a Hive on HBase task is performed.
The Spark service is abnormal when a Hive on Spark task is performed.
The dependent basic services, such as HDFS, Yarn, and ZooKeeper, are abnormal.

Procedure

Check whether the HQL statements comply with syntax.

On the FusionInsight Manager page, choose O&M > Alarm to view the alarm details and obtain the node where the alarm is generated.
Use the Hive client to log in to the HiveServer node where an alarm is reported. Query the HQL syntax provided by Apache, and check whether the HQL commands are correct. For details, see https://cwiki.apache.org/confluence/display/hive/languagemanual.
- If yes, go to 4.
- If no, go to 3.
NOTE:

To view the user who runs an incorrect statement, you can download the hiveserver audit log file of the HiveServer node where this alarm is generated. Start Data and End Data are 10 minutes before and after the alarm generation time respectively. Open the log file and search for the Result=FAIL keyword to filter the log information about the incorrect statement, and then view the user who runs the incorrect statement according to UserName in the log information.
Enter the correct HQL statements, and check whether the command can be properly executed.
- If yes, go to 12.
- If no, go to 4.

Check whether the HBase service is abnormal.

Check whether an Hive on HBase task is performed with the user who runs the HQL command.
- If yes, go to 5.
- If no, go to 8.
On the FusionInsight Manager page, click Cluster > Name of the desired cluster > Services, check whether the HBase service is normal in the service list.
- If yes, go to 8.
- If no, go to 6.
Choose O&M > Alarm, check the related alarms displayed on the alarm page and clear them according to related alarm help.
Enter the correct HQL statements, and check whether the command can be properly executed.
- If yes, go to 12.
- If no, go to 8.

Check whether the HDFS, Yarn, and ZooKeeper are normal.

On the FusionInsight Manager portal, click Cluster > Name of the desired cluster > Services.
In the service list, check whether the services, such as HDFS, Yarn, and ZooKeeper are normal.
- If yes, go to 12.
- If no, go to 10.
Check the related alarms displayed on the alarm page and clear them according to related alarm help.
Enter the correct HQL statements, and check whether the command can be properly executed.
- If yes, go to 12.
- If no, go to 13.
After 1 minute, check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 13.

Collect fault information.

On the FusionInsight Manager home page, choose O&M > Log > Download.
Select the following nodes in the required cluster from the Service:
- MapReduce
- Hive
Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
Contact the O&M personnel and send the collected logs.