ALM-14027 DataNode Disk Fault

Alarm Description

The system checks the disk status on DataNodes every 60 seconds. This alarm is generated when a disk is faulty.

After all faulty disks on the DataNode are recovered, you need to manually clear the alarm and restart the DataNode.

Alarm Attributes

Alarm Parameters

Impact on the System

If this alarm is reported, there are abnormal disk partitions on the DataNode. This may cause the loss of written files.

Possible Causes

• The hard disk is faulty.
• The disk permissions are configured improperly.

Handling Procedure

Check whether a disk alarm is generated.

1. On FusionInsight Manager, choose O&M > Alarm > Alarms and check whether ALM-12014 Partition Lost or ALM-12033 Slow Disk Fault exists.
   • If yes, go to 2.
   • If no, go to 4.

2. Rectify the fault by referring to the handling procedure of ALM-12014 Partition Lost or ALM-12033 Slow Disk Fault. Then, check whether the alarm is cleared.
   • If yes, go to 3.
   • If no, go to 4.

3. Wait 5 minutes and check whether the alarm is cleared.
   • If yes, no further action is required.
   • If no, go to 4.

Modify disk permissions.

4. Choose O&M > Alarm > Alarms and view Location and Additional Information of the alarm to obtain the location of the faulty disk.
5. Log in to the node for which the alarm is generated as user root. Go to the directory where the faulty disk is located, and run the ll command to check whether the permission of the faulty disk is 711 and whether the user is omm.
   • If yes, go to 8.
   • If no, go to 6.

6. Modify the permission of the faulty disk. For example, if the faulty disk is data1, run the following commands:

   chown omm:wheel data1

   chmod 711 data1

7. In the alarm list on Manager, click Clear in the Operation column of the alarm to manually clear the alarm. Choose Cluster > Services > HDFS > Instance, select the DataNode, choose More > Restart Instance, wait for 5 minutes, and check whether a new alarm is reported.
   • If no, no further action is required.
   • If yes, go to 8.

Collect the fault information.

8. On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
9. Expand the Service drop-down list, and select HDFS and OMS for the target cluster.
10. Click in the upper right corner, and set Start Date and End Date for log collection to 20 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
11. Contact O&M engineers and provide the collected logs.

Alarm Clearance

After the fault is rectified, the system does not automatically clear this alarm and you need to manually clear the alarm.

Related Information

None.