Help Center/ MapReduce Service/ User Guide/ MRS Cluster O&M/ MRS Cluster Alarm Handling Reference/ ALM-14027 DataNode Disk Fault
Updated on 2024-09-23 GMT+08:00
View PDF
Share

ALM-14027 DataNode Disk Fault

Alarm Description

The system checks the disk status on DataNodes every 60 seconds. This alarm is generated when a disk is faulty.

After all faulty disks on the DataNode are recovered, you need to manually clear the alarm and restart the DataNode.

Alarm Attributes

Alarm ID

Alarm Severity

Auto Cleared

14027

Major

No

Alarm Parameters

Parameter

Description

Source

Specifies the cluster for which the alarm was generated.

ServiceName

Specifies the service for which the alarm was generated.

RoleName

Specifies the role for which the alarm was generated.

HostName

Specifies the host for which the alarm was generated.

Failed Volumes

Specifies the list of faulty disks.

Impact on the System

If this alarm is reported, there are abnormal disk partitions on the DataNode. This may cause the loss of written files.

Possible Causes

  • The hard disk is faulty.
  • The disk permissions are configured improperly.

Handling Procedure

Check whether a disk alarm is generated.

  1. On FusionInsight Manager, choose O&M > Alarm > Alarms and check whether ALM-12014 Partition Lost or ALM-12033 Slow Disk Fault exists.

    • If yes, go to 2.
    • If no, go to 4.

  2. Rectify the fault by referring to the handling procedure of ALM-12014 Partition Lost or ALM-12033 Slow Disk Fault. Then, check whether the alarm is cleared.

    • If yes, go to 3.
    • If no, go to 4.

  3. Wait 5 minutes and check whether the alarm is cleared.

    • If yes, no further action is required.
    • If no, go to 4.

Modify disk permissions.

  1. Choose O&M > Alarm > Alarms and view Location and Additional Information of the alarm to obtain the location of the faulty disk.
  2. Log in to the node for which the alarm is generated as user root. Go to the directory where the faulty disk is located, and run the ll command to check whether the permission of the faulty disk is 711 and whether the user is omm.

    • If yes, go to 8.
    • If no, go to 6.

  3. Modify the permission of the faulty disk. For example, if the faulty disk is data1, run the following commands:

    chown omm:wheel data1

    chmod 711 data1

  4. In the alarm list on Manager, click Clear in the Operation column of the alarm to manually clear the alarm.Choose Cluster > Services > HDFS > Instance, select the DataNode, choose More > Restart Instance, wait for 5 minutes, and check whether a new alarm is reported.

    Services may be affected or interrupted during the restart. You are advised to perform this operation during off-peak hours.

    • If no, no further action is required.
    • If yes, go to 8.

Collect the fault information.

  1. On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
  2. Expand the Service drop-down list, and select HDFS and OMS for the target cluster.
  3. Click in the upper right corner, and set Start Date and End Date for log collection to 20 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
  4. Contact O&M personnel and provide the collected logs.

Alarm Clearance

After the fault is rectified, the system does not automatically clear this alarm and you need to manually clear the alarm.

Related Information

None