ALM-16008 Non-Heap Memory Usage of the Hive Service Exceeds the Threshold

Alarm Description

The system checks the Hive service status every 30 seconds. The alarm is generated when the non-heap memory usage of a Hive service exceeds the threshold (95% of the maximum memory).

You can choose O&M > Alarm > Thresholds > Name of the desired cluster > Hive to change the threshold.

The alarm is cleared when the non-heap memory usage is less than or equal to the threshold.

Alarm Attributes

Alarm Parameters

Impact on the System

When the non-heap memory usage of Hive is overhigh, the performance of Hive task operation is affected. In addition, a memory overflow may occur so that the Hive service is unavailable.

Possible Causes

The non-heap memory of the Hive instance on the node is overused or the non-heap memory is inappropriately allocated. As a result, the usage exceeds the threshold.

Handling Procedure

Check non-heap memory usage.

1. On the FusionInsight Manager portal, click O&M > Alarm > Alarms and select the alarm whose Alarm ID is 16008. Then check the role name in Location and confirm the IP address of the instance.
   • If the role for which the alarm is generated is HiveServer, go to Step 2.
   • If the role for which the alarm is generated is MetaStore, go to Step 3.

2. On the FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > Hive > Instances and click the HiveServer for which the alarm is generated to go to the Dashboard page. Click the drop-down menu in the Chart area and choose Customize > CPU and Memory, select HiveServer Memory Usage Statistics and click OK, and check whether the used non-heap memory of the HiveServer service reaches the threshold (default value: 95%) of the maximum non-heap memory specified for HiveServer.
   • If yes, go to Step 4.
   • If no, go to Step 7.
   Figure 1 HiveServer Memory Usage Statistics
   

3. On the FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > Hive > Instances and click the MetaStore for which the alarm is generated to go to the Dashboard page. Click the drop-down menu in the Chart area and choose Customize > CPU and Memory, select MetaStore Memory Usage Statistics and click OK, and check whether the used non-heap memory of the MetaStore service reaches the threshold (default value: 95%) of the maximum non-heap memory specified for MetaStore.
   • If yes, go to Step 4.
   • If no, go to Step 7.
   Figure 2 MetaStore Memory Usage Statistics
   

4. On the FusionInsight Manager portal, choose Cluster > Name of the desired cluster > Services > Hive > Configurations > All Configurations. Choose HiveServer/MetaStore > JVM. Adjust the value of -XX:MaxMetaspaceSize in HIVE_GC_OPTS/METASTORE_GC_OPTS as the following rules. Click Save.
   

   1. Suggestions for GC parameter settings for the HiveServer:
      • Set the value of -XX:MaxMetaspaceSize to 1/8 of the value of -Xmx. For example, if -Xmx is set to 2 GB, set -XX:MaxMetaspaceSize to 256 MB. If -Xmx is set to 4 GB, set -XX:MaxMetaspaceSize to 512 MB.
   2. Suggestions for GC parameter settings for the MetaServer:
      • Set the value of -XX:MaxMetaspaceSize to 1/8 of the value of -Xmx. For example, if -Xmx is set to 2 GB, set -XX:MaxMetaspaceSize to 256 MB. If -Xmx is set to 4 GB, set -XX:MaxMetaspaceSize to 512 MB.

5. Click More > Restart Service to restart the service.
   

   During Hive service restart, instances cannot provide services for external systems, and the SQL tasks that are being executed on the instances may fail.

6. Check whether the alarm is cleared.
   • If yes, no further action is required.
   • If no, go to Step 7.

Collect fault information.

7. On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
8. Expand the Service drop-down list, and select Hive for the target cluster.
9. Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes before and after the alarm generation time, respectively. Then, click Download.
10. Send the collected fault logs to O&M personnel for help.

Alarm Clearance

This alarm is automatically cleared after the fault is rectified.

Related Information

None