Help Center/ MapReduce Service/ User Guide/ MRS Cluster O&M/ MRS Cluster Alarm Handling Reference/ ALM-26054 Nimbus Heap Memory Usage Exceeds the Threshold
Updated on 2024-09-23 GMT+08:00
View PDF
Share

ALM-26054 Nimbus Heap Memory Usage Exceeds the Threshold

Description

The system checks the heap memory usage of Storm Nimbus every 30 seconds and compares the actual usage with the threshold. The alarm is generated when the heap memory usage of Storm Nimbus exceeds the threshold (80% of the maximum memory by default) for 5 consecutive times.

Users can choose O&M > Alarm > Thresholds > Name of the desired cluster > Storm > Nimbus to change the threshold.

The alarm is cleared when the heap memory usage is less than or equal to the threshold.

Attribute

Alarm ID

Alarm Severity

Automatically Cleared

26054

Major

Yes

Parameters

Name

Meaning

Source

Specifies the cluster for which the alarm is generated.

ServiceName

Specifies the service name for which the alarm is generated.

RoleName

Specifies the role name for which the alarm is generated.

HostName

Specifies the object (host ID) for which the alarm is generated.

Trigger Condition

Specifies the threshold triggering the alarm. If the current indicator value exceeds this threshold, the alarm is generated.

Impact on the System

When the heap memory usage of Storm Nimbus is overhigh, frequent GCs occur. In addition, a memory overflow may occur so that the Yarn service is unavailable.

Possible Causes

The heap memory of the Storm Nimbus instance on the node is overused or the heap memory is inappropriately allocated. As a result, the usage exceeds the threshold.

Procedure

Check the heap memory usage.

  1. On the FusionInsight Manager portal, choose O&M > Alarm > Alarms > Heap Memory Usage of Storm Nimbus Exceeds the Threshold > Location. Check the host name of the instance for which the alarm is generated.
  2. On the FusionInsight Manager portal, choose Cluster > Name of the desired cluster > Services > Storm > Instance. Click the instance for which the alarm is generated to go to the page for the instance. Click the drop-down menu in the chart area and choose Customize > Nimbus > Heap Memory Usage of Nimbus. Click OK.
  3. Check whether the used heap memory of Nimbus reaches the threshold (The default value is 80% of the maximum heap memory) specified for Nimbus.

    • If yes, go to 4.
    • If no, go to 6.

  4. On the FusionInsight Manager portal, choose Cluster > Name of the desired cluster > Services > Storm > Configurations > All Configurations > Nimbus > System. Change the value of -Xmx in NIMBUS_GC_OPTS based on site requirements, and click Save. Click OK.

    • You are advised to set -Xms and -Xmx to the same value to prevent adverse impact on performance when JVM dynamically adjusts the heap memory size.
    • The number of Workers grows as the Storm cluster scale increases. You can increase the value of GC_OPTS for Nimbus. The recommended value is as follows: If the number of Workers is 20, set -Xmx to a value greater than or equal to 1 GB. If the number of Workers exceeds 100, set -Xmx to a value greater than or equal to 5 GB.

  5. Restart the affected services or instances and check whether the alarm is cleared.

    • If yes, no further action is required.
    • If no, go to 6.

      During the restart of the service or instance, services are interrupted. After the service or instance is restarted, services are restored.

Collect fault information.

  1. On the FusionInsight Manager portal, choose O&M > Log > Download.
  2. Select the following node in the required cluster from the Service drop-down list.

    • NodeAgent
    • Storm

  3. Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
  4. Contact the O&M personnel and send the collected logs.

Alarm Clearing

After the fault is rectified, the system automatically clears this alarm.

Related Information

None