On this page

Show all

Help Center/ MapReduce Service/ User Guide (Kuala Lumpur Region)/ FusionInsight Manager Operation Guide (Applicable to 3.x)/ Alarm Reference (Applicable to MRS 3.x)/ ALM-18002 NodeManager Heartbeat Lost

ALM-18002 NodeManager Heartbeat Lost

Updated on 2022-08-12 GMT+08:00

View PDF

Description

The system checks the number of lost NodeManager nodes every 30 seconds, and compares the number with the threshold. The Number of Lost Nodes indicator has a default threshold. The alarm is generated when the value of Number of Lost Nodes exceeds the threshold.

To change the threshold, on FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > Yarn. On the displayed page, choose Configurations > All Configurations, and change the value of yarn.nodemanager.lost.alarm.threshold. You do not need to restart Yarn to make the change take effect.

The default threshold is 0. The alarm is generated when the number of lost nodes exceeds the threshold, and is cleared when the number of lost nodes is less than the threshold.

Attribute

Alarm ID	Alarm Severity	Automatically Cleared
18002	Major	Yes

Parameters

Name	Meaning
Source	Specifies the cluster for which the alarm is generated.
ServiceName	Specifies the service for which the alarm is generated.
RoleName	Specifies the role for which the alarm is generated.
HostName	Specifies the host for which the alarm is generated.
Lost Host	Specifies the list of hosts with lost nodes.

Impact on the System

The lost NodeManager node cannot provide the Yarn service.
The number of containers decreases, so the cluster performance deteriorates.

Possible Causes

NodeManager is forcibly deleted without decommission.
All the NodeManager instances are stopped or the NodeManager process is faulty.
The host where the NodeManager node resides is faulty.
The network between the NodeManager and ResourceManager is disconnected or busy.

Procedure

Check the NodeManager status.

On the FusionInsight Manager, and choose O&M > Alarm > Alarms. Click before the alarm and obtain lost nodes in Additional Information.
Check whether the lost nodes are hosts that have been manually deleted without decommission.
- If yes, go to 3.
- If no, go to 5.
After the setting, Choose Cluster > Name of the desired cluster > Services > Yarn. On the displayed page, choose Configurations > All Configurations. Search for yarn.nodemanager.lost.alarm.threshold and change its value to the number of hosts that are not out of service and proactively deleted. After the setting, check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 4.
Manually clear the alarm. Note that decommission must be performed before deleting hosts.
On the FusionInsight Manager portal, choose Cluster > Hosts, and check whether the nodes obtained in 1 are healthy.
- If yes, go to 7.
- If no, go to 6.
Rectify the node fault based on ALM-12006 Node Fault and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 7.

Check the process status.

On the FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > Yarn > Instance, and check whether there are NodeManager instances whose status is not Good.
- If yes, go to 10.
- If no, go to 8.
Check whether the NodeManager instance is deleted.
- If yes, go to 9.
- If no, go to 11.
Restart the active and standby ResourceManager instances, and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 13.

Check the instance status.

Select NodeManager instances which running state is not Normal and restart them. Check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 11.

Check the network status.

Log in to the management node, ping the IP address of the lost NodeManager node to check whether the network is disconnected or busy.
- If yes, go to 12.
- If no, go to 13.
Rectify the network, and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 13.

Collect fault information.

On the FusionInsight Manager in the active cluster, choose O&M > Log > Download.
Select Yarn in the required cluster from the Service.
Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
Contact the O&M personnel and send the collected logs.

Alarm Clearing

After the fault is rectified, the system automatically clears this alarm.

Related Information

None

Parent topic: Alarm Reference (Applicable to MRS 3.x)

Previous topic: ALM-18000 Yarn Service Unavailable

Next topic: ALM-18003 NodeManager Unhealthy

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot