Help Center > > User Guide> FusionInsight Manager Operation Guide> Alarm Reference (Applicable to MRS 3.x)> ALM-14003 Number of Lost HDFS Blocks Exceeds the Threshold

ALM-14003 Number of Lost HDFS Blocks Exceeds the Threshold

Updated at: Mar 25, 2021 GMT+08:00

Description

The system checks the lost blocks every 30 seconds and compares the actual lost blocks with the threshold. The lost HDFS blocks indicator has a default threshold, this alarm is generated when the value of the lost blocks indicator exceeds the threshold.

To change the threshold, choose O&M > Alarm > Thresholds > Name of the desired cluster > HDFS.

When the Trigger Count is 1, this alarm is cleared when the value of lost HDFS blocks is less than or equal to the threshold. When the Trigger Count is greater than 1, this alarm is cleared when the value of lost HDFS blocks is less than or equal to 90% of the threshold.

Attribute

Alarm ID

Alarm Severity

Automatically Cleared

14003

Major

Yes

Parameters

Name

Meaning

Source

Specifies the cluster for which the alarm is generated.

ServiceName

Specifies the service for which the alarm is generated.

RoleName

Specifies the role for which the alarm is generated.

HostName

Specifies the host for which the alarm is generated.

NameServiceName

Specifies the NameService for which the alarm is generated.

Trigger Condition

Specifies the threshold triggering the alarm. If the current indicator value exceeds this threshold, the alarm is generated.

Impact on the System

The Hadoop distributed file system (HDFS) enters the safe mode and cannot write data. Data on the lost blocks cannot be restored.

Possible Causes

  • The DataNode instance is abnormal.
  • Data is deleted.

Procedure

Check the DataNode instance.

  1. On the FusionInsight Manager portal, choose Cluster > Name of the desired cluster > Services > HDFS > Instance.
  2. Check whether the Running Status of all DataNode instance is Normal.

    • If yes, go to 10.
    • If no, go to 3.

  3. Restart the DataNode instance and check whether the DataNode instance restarts successfully.

    • If yes, go to 4.
    • If no, go to 5.

  4. Choose O&M > Alarm > Alarms to check whether the alarm is cleared.

    • If yes, no further action is required.
    • If no, run 5.

Delete the damaged file.

  1. On the FusionInsight Manager portal, Choose Cluster > Name of the desired cluster > Services > HDFS > NameNode(Active). On the WebUI page of the HDFS, view the information about lost blocks.

    • If a block is lost, a line in red is displayed on the WebUI.
    • By default, the admin user does not have the permissions to manage other components. If the page cannot be opened or the displayed content is incomplete when you access the native UI of a component due to insufficient permissions, you can manually create a user with the permissions to manage that component.

  2. The user checks whether the file containing the lost data block is useful.

    Files generated in directories /mr-history, /tmp/hadoop-yarn, and /tmp/logs during MapReduce task execution are unnecessary.

    • If yes, go to 7.
    • If no, go to 8.

  3. The user checks whether the file containing the lost data block is backed up.

    • If yes, go to 8.
    • If no, go to 10.

  4. On the node client, run hdfs fsck / -delete to delete the lost file. If the file where the lost block is located is a useful file, you need to write the file again to restore the data.
  5. Choose O&M > Alarm > Alarms to check whether the alarm is cleared.

    • If yes, no further action is required.
    • If no, run 10.

Collect fault information.

  1. On the FusionInsight Manager portal, choose O&M > Log > Download.
  2. Select HDFS in the required cluster from the Service.
  3. Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
  4. Contact the O&M personnel and send the collected logs.

Alarm Clearing

After the fault is rectified, the system automatically clears this alarm.

Related Information

None

Did you find this page helpful?

Submit successfully!

Thank you for your feedback. Your feedback helps make our documentation better.

Failed to submit the feedback. Please try again later.

Which of the following issues have you encountered?







Please complete at least one feedback item.

Content most length 200 character

Content is empty.

OK Cancel