Help Center/ MapReduce Service/ User Guide (Kuala Lumpur Region)/ FusionInsight Manager Operation Guide (Applicable to 3.x)/ Alarm Reference (Applicable to MRS 3.x)/ ALM-12089 Inter-Node Network Is Abnormal
Updated on 2022-08-12 GMT+08:00
View PDF
Share

ALM-12089 Inter-Node Network Is Abnormal

Description

The alarm module checks the network health status of nodes in the cluster every 10 seconds. This alarm is generated when the network between two nodes is unreachable or the network status is unstable.

Attribute

Alarm ID

Alarm Severity

Auto Clear

12089

Major

Yes

Parameters

Name

Meaning

Source

Specifies the cluster or system for which the alarm is generated.

ServiceName

Specifies the service for which the alarm is generated.

RoleName

Specifies the role for which the alarm is generated.

HostName

Specifies the host for which the alarm is generated.

Impact on the System

Functions of some components, such as HDFS and ZooKeeper, are affected.

Possible Causes

  • The node breaks down.
  • The network is faulty.

Procedure

Check the network health status.

  1. In the alarm list on FusionInsight Manager, click the drop-down button of the alarm and view Additional Information. Record the source IP address and destination IP address of the node for which the alarm is reported.
  2. Log in to the node for which the alarm is reported. On the node, ping the target node to check whether the network between the two nodes is normal.

    • If yes, go to 6.
    • If no, go to 3.

Check the node status.

  1. On FusionInsight Manager, click Host and check whether the host list contains the faulty node to determine whether the faulty node has been removed from the cluster.

    • If yes, go to 5.
    • If no, go to 4.

  2. Check whether the faulty node is powered off.

    • If yes, start the faulty node and go to 2.
    • If no, contact related personnel to find root cause, if need to remove the faulty nodes from the cluster and go to 5, otherwise go to 6.

  3. Remove the file $NODE_AGENT_HOME/etc/agent/hosts.ini of all nodes in the cluster, and clean up the file /var/log/Bigdata/unreachable/unreachable_ip_info.log, and then manually clear the alarm.
  4. Wait for 30 seconds and checking if the alarm was been cleared.

    • If yes, no further action is required.
    • If no, go to 7.

Collect fault information.

  1. On the FusionInsight Manager portal, choose O&M > Log > Download.
  2. Select OmmAgent from the Service and click OK.
  3. Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
  4. Contact the O&M personnel and send the collected log information.

Alarm Clearing

After the fault is rectified, the system automatically clears this alarm.

Related Information

None