Help Center/ MapReduce Service/ User Guide/ MRS Cluster O&M/ MRS Cluster Alarm Handling Reference/ ALM-14003 Number of Lost HDFS Blocks Exceeds the Threshold (For MRS 2.x or Earlier)
Updated on 2024-09-23 GMT+08:00
View PDF
Share

ALM-14003 Number of Lost HDFS Blocks Exceeds the Threshold (For MRS 2.x or Earlier)

Description

The system checks the number of lost blocks every 30 seconds and compares the number of lost blocks with the threshold. The lost blocks indicator has a default threshold. This alarm is generated when the number of lost blocks exceeds the threshold.

This alarm is cleared when the number of lost blocks is less than or equal to the threshold.

Attribute

Alarm ID

Alarm Severity

Auto Clear

14003

Major

Yes

Parameters

Parameter

Description

ServiceName

Specifies the service for which the alarm is generated.

RoleName

Specifies the role for which the alarm is generated.

HostName

Specifies the host for which the alarm is generated.

NSName

Specifies the NameService service for which the alarm is generated.

Trigger condition

Generates an alarm when the actual indicator value exceeds the specified threshold.

Impact on the System

Data stored in HDFS is lost. HDFS may enter the safe mode and cannot provide write services. Lost block data cannot be restored.

Possible Causes

  • The DataNode instance is abnormal.
  • Data is deleted.

Procedure

  1. Check the DataNode instance.

    1. On the MRS cluster details page, choose Components > HDFS > Instances.
    2. Check whether the status of all DataNode instances is Good.
      • If yes, go to 3.
      • If no, go to 1.c.
    3. Restart the DataNode instance and check whether the restart is successful.
      • If yes, go to 2.b.
      • If no, go to 2.a.

  2. Delete the damaged file.

    1. Use the client on the cluster node. Run the hdfs fsck / -delete command to delete the lost file. Then rewrite the file and recover the data.
    2. Wait 5 minutes and check whether the alarm is cleared.
      • If yes, no further action is required.
      • If no, go to 3.

  3. Collect fault information.

    1. On MRS Manager, choose System > Export Log.
    2. Contact the O&M engineers and send the collected logs.

Reference

None