Help Center/ MapReduce Service/ User Guide/ MRS Cluster O&M/ MRS Cluster Alarm Handling Reference/ ALM-18019 Non Heap Memory Usage of JobHistoryServer Exceeds the Threshold
Updated on 2024-09-23 GMT+08:00
View PDF
Share

ALM-18019 Non Heap Memory Usage of JobHistoryServer Exceeds the Threshold

Description

The system checks the Non Heap memory usage of MapReduce JobHistoryServer every 30 seconds and compares the actual usage with the threshold. The alarm is generated when the Non Heap memory usage of MapReduce JobHistoryServer exceeds the threshold (90% of the maximum memory by default).

Users can choose O&M > Alarm > Thresholds > Name of the desired cluster > MapReduce to change the threshold.

The alarm is cleared when the Non Heap memory usage is less than or equal to the threshold.

Attribute

Alarm ID

Alarm Severity

Automatically Cleared

18019

Major

Yes

Parameters

Name

Meaning

Source

Specifies the cluster for which the alarm is generated.

ServiceName

Specifies the service name for which the alarm is generated.

RoleName

Specifies the role name for which the alarm is generated.

HostName

Specifies the object (host ID) for which the alarm is generated.

Trigger Condition

Specifies the threshold triggering the alarm. If the current indicator value exceeds this threshold, the alarm is generated.

Impact on the System

When the Non Heap memory usage of MapReduce JobHistoryServer is overhigh, the performance of MapReduce task submission and operation is affected. In addition, a memory overflow may occur so that the MapReduce service is unavailable.

Possible Causes

The Non Heap memory of the MapReduce JobHistoryServer instance on the node is overused or the Non Heap memory is inappropriately allocated. As a result, the usage exceeds the threshold.

Procedure

Check the Non Heap memory usage.

  1. On the FusionInsight Manager portal, choose O&M > Alarm > Alarms > ALM-18019 Non Heap Memory Usage of MapReduce JobHistoryServer Exceeds the Threshold > Location. Check the HostName of the instance for which the alarm is generated.
  2. On the FusionInsight Manager portal, choose Cluster > Name of the desired cluster > Services > MapReduce > Instance > JobHistoryServer. Click the drop-down menu in the upper right corner of Chart, choose Customize > JobHistoryServer Non Heap memory usage statistics. JobHistoryServer indicates the corresponding HostName of the instance for which the alarm is generated. Check the Non Heap memory usage.
  3. Check whether the used Non Heap memory of JobHistoryServer reaches 90% of the maximum Non Heap memory specified for JobHistoryServer.

    • If yes, go to 4.
    • If no, go to 6.

  4. On the FusionInsight Manager portal, choose Cluster > Name of the desired cluster > Services > MapReduce > Configurations > All Configurations > JobHistoryServer > System. Adjust the GC_OPTS memory parameter of the NodeManager, click Save, and click OK, and restart the role instance.

    • During the restart of JobHistoryServer, the status query of tasks such as Hive is affected, and the query result may be inaccurate.
    • The mapping between the number of historical tasks (10000) and the memory of the JobHistoryServer is as follows:

      -Xms30G -Xmx30G -XX:NewSize=1G -XX:MaxNewSize=2G

  5. Check whether the alarm is cleared.

    • If yes, no further action is required.
    • If no, go to 6.

Collect fault information.

  1. On the FusionInsight Manager portal, choose O&M > Log > Download.
  2. Select the following node in the required cluster from the Service.

    • NodeAgent
    • MapReduce

  3. Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
  4. Contact the O&M personnel and send the collected logs.

Alarm Clearing

After the fault is rectified, the system automatically clears this alarm.

Related Information

None