Help Center > > User Guide> Managing Active Clusters> Alarm Reference> ALM-14009 Number of Dead DataNodes Exceeds the Threshold

ALM-14009 Number of Dead DataNodes Exceeds the Threshold

Updated at: Mar 31, 2020 GMT+08:00

Description

The system checks the number of faulty DataNodes in the HDFS cluster every 30 seconds and compares it with the threshold. This alarm is generated when the number of faulty DataNodes in the HDFS cluster exceeds the threshold and is cleared when the number is less than or equal to the threshold.

Attribute

Alarm ID

Alarm Severity

Automatically Cleared

14009

Major

Yes

Parameters

Parameter

Description

ServiceName

Specifies the service for which the alarm is generated.

RoleName

Specifies the role for which the alarm is generated.

HostName

Specifies the host for which the alarm is generated.

Trigger condition

Generates an alarm when the actual indicator value exceeds the specified threshold.

Impact on the System

Faulty DataNodes cannot provide HDFS services.

Possible Causes

  • DataNodes are faulty or overloaded.
  • The network between the NameNode and the DataNode is disconnected or busy.
  • NameNodes are overloaded.

Procedure

  1. Check whether DataNodes are faulty.

    1. Use the client on the cluster node and run the hdfs dfsadmin -report command to check whether DataNodes are faulty.
      • If yes, go to 1.b.
      • If no, go to 2.a.
    2. On the MRS cluster details page, choose Services > HDFS > Instance to check whether any DataNode is stopped.

      For MRS 1.8.10 or earlier, log in to MRS Manager and choose Services > HDFS > Instance to check whether any DataNode is stopped.

      • If yes, go to 1.c.
      • If no, go to 2.a.
    3. Select the DataNode instance, and choose More > Restart Instance to restart it. Wait 5 minutes and check whether the alarm is cleared.
      • If yes, no further action is required.
      • If no, go to 2.a.

  2. Check the status of the network between the NameNode and the DataNode.

    1. Log in to the faulty DataNode using its service IP address. Run the ping IP address of the NameNode command to check whether the network between the DataNode and the NameNode is abnormal.
      • If yes, go to 2.b.
      • If no, go to 3.a.
    2. Rectify the network fault. Wait 5 minutes and check whether the alarm is cleared.
      • If yes, no further action is required.
      • If no, go to 3.a.

  3. Check whether the DataNode is overloaded.

    1. On the MRS cluster details page, click Alarms and check whether alarm ALM-14008 HDFS DataNode Memory Usage Exceeds the Threshold exists.

      For MRS 1.8.10 or earlier, log in to MRS Manager to view alarm information.

      • If yes, go to 3.b.
      • If no, go to 4.a.
    2. Follow the procedures in ALM-14008 HDFS DataNode Memory Usage Exceeds the Threshold to handle the alarm and check whether the alarm is cleared.
      • If yes, go to 3.c.
      • If no, go to 4.a.
    3. Wait 5 minutes and check whether the alarm is cleared.
      • If yes, no further action is required.
      • If no, go to 4.a.

  4. Check whether the NameNode is overloaded.

    1. On the MRS cluster details page, click Alarms and check whether alarm ALM-14007 HDFS NameNode Memory Usage Exceeds the Threshold exists.

      For MRS 1.8.10 or earlier, log in to MRS Manager to view alarm information.

      • If yes, go to 4.b.
      • If no, go to 5.
    2. Follow the procedures in ALM-14007 HDFS NameNode Memory Usage Exceeds the Threshold to handle the alarm and check whether the alarm is cleared.
      • If yes, go to 4.c.
      • If no, go to 5.
    3. Wait 5 minutes and check whether the alarm is cleared.
      • If yes, no further action is required.
      • If no, go to 5.

  5. Collect fault information.

    1. On MRS Manager, choose System > Export Log.
    2. Contact the O&M personnel and send the collected log information.

Related Information

N/A

Did you find this page helpful?

Submit successfully!

Thank you for your feedback. Your feedback helps make our documentation better.

Failed to submit the feedback. Please try again later.

Which of the following issues have you encountered?







Please complete at least one feedback item.

Content most length 200 character

Content is empty.

OK Cancel