Help Center/ MapReduce Service/ User Guide/ MRS Cluster O&M/ MRS Cluster Alarm Handling Reference/ ALM-16006 The Direct Memory Usage of the Hive Process Exceeds the Threshold
Updated on 2024-09-23 GMT+08:00
View PDF
Share

ALM-16006 The Direct Memory Usage of the Hive Process Exceeds the Threshold

Description

The system checks the Hive service status every 30 seconds. The alarm is generated when the direct memory usage of an Hive service exceeds the threshold (95% of the maximum memory).

Users can choose O&M > Alarm > Thresholds > Name of the desired cluster > Hive to change the threshold.

The alarm is cleared when the direct memory usage is less than or equal to the threshold.

Attribute

Alarm ID

Alarm Severity

Automatically Cleared

16006

Major

Yes

Parameters

Name

Meaning

Source

Specifies the cluster for which the alarm is generated.

ServiceName

Specifies the service name for which the alarm is generated.

RoleName

Specifies the role name for which the alarm is generated.

HostName

Specifies the object (host ID) for which the alarm is generated.

Trigger Condition

Specifies the threshold triggering the alarm. If the current indicator value exceeds this threshold, the alarm is generated.

Impact on the System

When the direct memory usage of Hive is overhigh, the performance of Hive task operation is affected. In addition, a memory overflow may occur so that the Hive service is unavailable.

Possible Causes

The direct memory of the Hive instance on the node is overused or the direct memory is inappropriately allocated. As a result, the usage exceeds the threshold.

Procedure

Check direct memory usage.

  1. On the FusionInsight Manager portal, click O&M > Alarm > Alarms and select the alarm whose Alarm ID is 16006. Then check the role name in Location and confirm the IP adress of the instance.

    • If the role for which the alarm is generated is HiveServer, go to 2.
    • If the role for which the alarm is generated is MetaStore, go to 3.

  2. On the FusionInsight Manager portal, choose Cluster > Name of the desired cluster > Services > Hive > Instance and click the HiveServer for which the alarm is generated to go to the Dashboard page. Click the drop-down menu in the Chart area and choose Customize > CPU and Memory, and select HiveServer Memory Usage Statistics and click OK, check whether the used direct memory of the HiveServer service reaches the threshold(default value: 95%) of the maximum direct memory specified for HiveServer.

    • If yes, go to 4.
    • If no, go to 7.
    Figure 1 HiveServer Memory Usage Statistics

  3. On the FusionInsight Manager portal, choose Cluster > Name of the desired cluster > Services > Hive > Instance and click the MetaStore for which the alarm is generated to go to the Dashboard page. Click the drop-down menu in the Chart area and choose Customize > CPU and Memory, and select MetaStore Memory Usage Statistics and click OK, check whether the used direct memory of the MetaStore service reaches the threshold(default value: 95%) of the maximum direct memory specified for MetaStore.

    • If yes, go to 4.
    • If no, go to 7.
    Figure 2 MetaStore Memory Usage Statistics

  4. On the FusionInsight Manager portal, choose Cluster >Name of the desired cluster > Services > Hive > Configurations > All Configurations. Choose HiveServer/MetaStore > JVM. Adjust the value of -XX:MaxDirectMemorySize in HIVE_GC_OPTS/METASTORE_GC_OPTS as the following rules. Click Save.

    Suggestions for GC parameter settings for the HiveServer:
    • It is recommended that you set the value of -XX:MaxDirectMemorySize to 1/8 of the value of -Xmx. For example, if -Xmx is set to 8 GB, -XX:MaxDirectMemorySize is set to 1024 MB. If -Xmx is set to 4 GB, -XX:MaxDirectMemorySize is set to 512 MB. It is recommended that the value of -XX:MaxDirectMemorySize be greater than or equal to 512 MB.
    Suggestions for GC parameter settings for the MetaServer:
    • It is recommended that you set the value of -XX:MaxDirectMemorySize to 1/8 of the value of -Xmx. For example, if -Xmx is set to 8 GB, -XX:MaxDirectMemorySize is set to 1024 MB. If -Xmx is set to 4 GB, -XX:MaxDirectMemorySize is set to 512 MB. It is recommended that the value of -XX:MaxDirectMemorySize be greater than or equal to 512 MB.

  5. Click More > Restart Service to restart the service.

    During Hive service restart, instances cannot provide services for external systems, and the SQL tasks that are being executed on the instances may fail.

  6. Check whether the alarm is cleared.

    • If yes, no further action is required.
    • If no, go to 7.

Collect fault information.

  1. On the FusionInsight Manager portal, choose O&M > Log > Download.
  2. Select Hive in the required cluster from the Service.
  3. Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
  4. Contact the O&M personnel and send the collected fault logs.

Alarm Clearing

After the fault is rectified, the system automatically clears this alarm.

Related Information

None