Help Center/ MapReduce Service/ User Guide/ MRS Cluster O&M/ MRS Cluster Alarm Handling Reference/ ALM-14002 DataNode Disk Usage Exceeds the Threshold
Updated on 2024-09-23 GMT+08:00
View PDF
Share

ALM-14002 DataNode Disk Usage Exceeds the Threshold

Alarm Description

The system checks the DataNode disk usage every 30 seconds and compares the actual disk usage with the threshold. A default threshold range is provided for the DataNode disk usage. This alarm is generated when the DataNode disk usage exceeds the threshold.

To change the threshold, choose O&M > Alarm > Thresholds > Name of the desired cluster > HDFS.

If Trigger Count is 1, this alarm is cleared when the DataNode disk usage is less than or equal to the threshold. If Trigger Count is greater than 1, this alarm is cleared when the DataNode disk usage is less than or equal to 80% of the threshold.

Alarm Attributes

Alarm ID

Alarm Severity

Auto Cleared

14002

Major

Yes

Alarm Parameters

Parameter

Description

Source

Specifies the cluster for which the alarm was generated.

ServiceName

Specifies the service for which the alarm was generated.

RoleName

Specifies the role for which the alarm was generated.

HostName

Specifies the host for which the alarm was generated.

Trigger Condition

Specifies the threshold for triggering the alarm.

Impact on the System

Insufficient disk space will impact data write to HDFS.

Possible Causes

  • The disk space configured for the HDFS cluster is insufficient.
  • Data skew occurs among DataNodes.

Handling Procedure

Check whether the cluster disk capacity is insufficient.

  1. On FusionInsight Manager, choose O&M > Alarm > Alarms, and check whether the ALM-14001 HDFS Disk Usage Exceeds the Threshold alarm exists.

    • If yes, go to 2.
    • If no, go to 4.

  2. Handle the alarm by following the instructions in ALM-14001 HDFS Disk Usage Exceeds the Threshold and check whether the alarm is cleared.

    • If yes, go to 3.
    • If no, go to 11.

  3. Choose O&M > Alarm > Alarms and check whether the alarm is cleared.

    • If yes, no further action is required.
    • If no, go to 4.

Check the balance status of DataNodes.

  1. On FusionInsight Manager, choose Hosts. Check whether the number of DataNodes on each rack is almost the same. If the difference is large, adjust the racks to which DataNodes belong to ensure that the number of DataNodes on each rack is almost the same.Restart the HDFS service for the settings to take effect.

    The service will be unavailable during the restart. In addition, upper-layer services that depend on the service are affected.

  2. Choose Cluster > Name of the desired cluster > Services > HDFS.
  3. In the Basic Information area, click NameNode(Active). The HDFS web UI is displayed.

    By default, the admin user does not have the permissions to manage other components. If the page cannot be opened or the displayed content is incomplete when you access the native UI of a component due to insufficient permissions, you can manually create a user with the permissions to manage that component.

  4. In the Summary area of the HDFS web UI, check whether the value of Max is 10% greater than that of Median in DataNodes usages.

    • If yes, go to 8.
    • If no, go to 11.

  5. Balance skewed data in the cluster. Log in to the MRS client as user root. If the cluster is in normal mode, run the su - omm command to switch to user omm. Run the cd command to go to the client installation directory and run the source bigdata_env command. If the cluster uses the security mode, perform security authentication. Run kinit hdfs and enter the password as prompted. Obtain the password from the MRS cluster administrator.
  6. Run the following command to balance data distribution:

    hdfs balancer -threshold 10

  7. Wait several minutes and check whether the alarm is cleared.

    • If yes, no further action is required.
    • If no, go to 11.

Collect the fault information.

  1. On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
  2. Expand the drop-down list next to the Service field. In the Services dialog box that is displayed, select HDFS for the target cluster.
  3. Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
  4. Contact O&M personnel and provide the collected logs.

Alarm Clearance

This alarm is automatically cleared after the fault is rectified.

Related Information

None