ALM-14002 DataNode Disk Usage Exceeds the Threshold (For MRS 2.x or Earlier)
Description
The system checks the DataNode disk usage every 30 seconds and compares the actual disk usage with the threshold. The Percentage of DataNode Capacity indicator has a default threshold. This alarm is generated when the value of the Percentage of DataNode Capacity indicator exceeds the threshold.
This alarm is cleared when the value of the Percentage of DataNode Capacity indicator is less than or equal to the threshold.
Attribute
Alarm ID |
Alarm Severity |
Auto Clear |
---|---|---|
14002 |
Major |
Yes |
Parameters
Parameter |
Description |
---|---|
ServiceName |
Specifies the service for which the alarm is generated. |
RoleName |
Specifies the role for which the alarm is generated. |
HostName |
Specifies the host for which the alarm is generated. |
Trigger condition |
Generates an alarm when the actual indicator value exceeds the specified threshold. |
Impact on the System
Insufficient disk space will impact read/write to HDFS.
Possible Causes
- The disk space configured for the HDFS cluster is insufficient.
- Data skew occurs among DataNodes.
Procedure
- Check the cluster disk capacity.
- Go to the MRS cluster details page. On the Alarms page, check whether the ALM-14001 HDFS Disk Usage Exceeds the Threshold alarm exists.
- Handle the alarm by following the instructions in ALM-14001 HDFS Disk Usage Exceeds the Threshold and check whether the alarm is cleared.
- Wait 5 minutes and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 2.a.
- Check the balance status of DataNodes.
- Use the client on the cluster node, run the hdfs dfsadmin -report command to view the value of DFS Used% on the DataNode for which the alarm is generated, and compare the value with those on other DataNodes. Check whether the difference between the values is larger than 10.
- If data skew occurs, use the client on the cluster node and run the hdfs balancer -threshold 10 command.
- Wait 5 minutes and check whether the alarm is cleared.
- If yes, no further action is required.
- If no, go to 3.
- Collect fault information.
- On MRS Manager, choose .
- Contact the O&M engineers and send the collected logs.
Reference
None
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.