Help Center/ MapReduce Service/ User Guide/ MRS Cluster O&M/ MRS Cluster Alarm Handling Reference/ ALM-14001 HDFS Disk Usage Exceeds the Threshold (For MRS 2.x or Earlier)
Updated on 2024-09-23 GMT+08:00
View PDF
Share

ALM-14001 HDFS Disk Usage Exceeds the Threshold (For MRS 2.x or Earlier)

Description

The system checks the disk usage of the HDFS cluster every 30 seconds and compares the actual disk usage with the threshold. The HDFS cluster disk usage indicator has a default threshold. This alarm is generated when the HDFS disk usage exceeds the threshold.

This alarm is cleared when the disk usage of the HDFS cluster is less than or equal to the threshold.

Attribute

Alarm ID

Alarm Severity

Auto Clear

14001

Major

Yes

Parameters

Parameter

Description

ServiceName

Specifies the service for which the alarm is generated.

RoleName

Specifies the role for which the alarm is generated.

HostName

Specifies the host for which the alarm is generated.

NSName

Specifies the NameService service for which the alarm is generated.

Trigger condition

Generates an alarm when the actual indicator value exceeds the specified threshold.

Impact on the System

The performance of writing data to HDFS is affected.

Possible Causes

The disk space configured for the HDFS cluster is insufficient.

Procedure

  1. Check the disk capacity and delete unnecessary files.

    1. On the MRS cluster details page, choose Components > HDFS. The Service Status page is displayed.
    2. In the Charts area, view the value of the monitoring indicator Percentage of HDFS Capacity to check whether the HDFS disk usage exceeds the threshold (80% by default).
      • If yes, go to 1.c.
      • If no, go to 3.
    3. Use the client on the cluster node and run the hdfs dfsadmin -report command to check whether the value of DFS Used% is less than 100% minus the threshold.
      • If yes, go to 1.e.
      • If no, go to 3.
    4. Use the client on the cluster node and run the hdfs dfs -rm -r file or directory path command to delete unnecessary files.
    5. Wait 5 minutes and check whether the alarm is cleared.
      • If yes, no further action is required.
      • If no, go to 2.a.

  2. Expand the system.

    1. Expand the disk capacity.
    2. Wait 5 minutes and check whether the alarm is cleared.
      • If yes, no further action is required.
      • If no, go to 3.

  3. Collect fault information.

    1. On MRS Manager, choose System > Export Log.
    2. Contact the O&M engineers and send the collected logs.

Reference

None