ALM-12063 Unavailable Disk

Alarm Description

For versions earlier than MRS 3.6.0.1:

The system checks whether the data disk of the current host is available every five minutes (on the hour in versions earlier than MRS 3.1.5). Operations such as file creation, file writing, and file deletion are performed in the mount directory of the disk. If the operations are successful, the disk is available and a clear alarm is reported. If the operations fail, a fault alarm is reported.

For MRS 3.6.0.1 or later:

The system checks whether the data disk and installation directory disk of the current host are available every minute. The check methods are as follows:

• Method 1: Check whether the disk is readable or writable.

  Create, write, and delete files in the disk's mount directory. If any of these operations fail, an alarm indicating a disk fault is generated. If all operations succeed, the disk is considered available and the alarm is cleared.

• Method 2: Check whether the disk file system is functioning normally.

  Check whether the file system of the virtual disk device partition corresponding to the disk mount directory is functioning normally. For XFS disks, run the xfs_info command. For EXT4 disks, run the tune2fs command. If the file system is abnormal, an alarm indicating a disk fault is generated. If the file system is normal, the alarm is cleared.

• Method 3: Check whether the disk has bad sectors.

  Check whether the virtual disk partition for the disk mount directory is normal by reading the message logs of the operating system. If the device partition is abnormal, an alarm indicating the disk fault is generated. If the device partition is normal, the alarm is cleared.

In MRS 3.6.0.1 or later, the alarm name is changed from "Unavailable Disk" to "Disk Fault".

Alarm Attributes

Parameters

Impact on the System

Versions earlier than MRS 3.6.0.1: If you need to modify or use data on a disk that is unwritable or unreadable, the job may fail.

MRS 3.6.0.1 or later: If you need to modify or use data on a disk that is unwritable or unreadable or whose file system is abnormal, the job may fail to read data or an error may occur.

Possible Causes

The permission on the disk mount directory is abnormal, the disk file system is damaged, or the disk partition is abnormal.

Handling Procedure (for Versions Earlier Than MRS 3.6.0.1)

Check whether the permission of the disk mount directory is normal.

1. Log in to MRS Manager. In the alarm list, click in the row that contains the alarm to view the IP address of the host and disk name for which the alarm is generated.

   For details about how to log in to FusionInsight Manager, see Accessing MRS Manager.

2. Log in to the host for which the alarm is generated as user root.

3. Run the following command to obtain the mount point and check whether the permission of the mount directory is unwritable or unreadable:

   df -h |grep Disk name

   If the permission of the mount directory is 000 or the owner is root, the mount directory is unreadable and unwritable.

   • If it is, go to Step 4.
   • If it is not, go to Step 8.

4. Modify the directory permission.
5. One hour later, check whether this alarm is cleared.
   • If it is, no further action is required.
   • If it is not, go to Step 6.

6. Contact hardware engineers to rectify the disk.

7. One hour later, check whether this alarm is cleared.
   • If it is, no further action is required.
   • If it is not, go to Step 8.

Collect fault information.

8. On MRS Manager, choose O&M > Log > Download.
9. Select NodeAgent from the Service and click OK.
10. Click in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
11. Contact the O&M personnel and send the collected log information.

Handling Procedure (for MRS 3.6.0.1 or Later)

1. Log in to MRS Manager. In the alarm list, click the alarm to view the alarm details and obtain IP address of the host and disk name for which the alarm is generated.

   For details about how to log in to FusionInsight Manager, see Accessing MRS Manager.

2. Check the details in the additional information of the alarm.
   • If the alarm details contain Disk not readable or writable, go to Step 3.
   • If the alarm details contain Disk file system exception, go to Step 7.
   • If the alarm details contain Disk sector exception, go to Step 12.

Check whether the permission of the disk mount directory is normal.

3. Log in to the host for which the alarm is generated as user omm.
4. Run the following command to obtain the mount point and check whether the permission of the mount directory is unwritable or unreadable:

   df -h |grep Disk name

   If the permission of the mount directory is 000 or the owner is root, the mount directory is unreadable and unwritable.

   • If yes, go to Step 5.
   • If no, go to Step 14.

5. Modify the directory permission.
6. Wait about 2 minutes and check whether the alarm is cleared.
   • If yes, no further action is required.
   • If no, go to Step 14.

Check whether the disk file system is functioning normally.

7. Log in to the host for which the alarm is generated as user root.
8. In the alarm's additional information, check whether the file system type of the disk for which the alarm is generated is XFS.
   • If the file system type is XFS, go to Step 9.
   • If the file system type is EXT4, go to Step 10.
   • If the file system is of another type, go to Step 14.

9. Run the following commands to switch to user omm and check whether the returned value shows any exceptions:

   su - omm

   xfs_info ${Virtual disk device partition name} | grep "error"

   • If yes, contact hardware personnel for maintenance. After the maintenance is complete, go to Step 11.
   • If no, go to Step 14.

   For example, the following command output indicates that no exceptions occur:

   

10. Run the following command to check whether the returned value shows any exceptions:

    tune2fs -l /dev/vdb2 | grep "Filesystem state" | grep "errors"

    • If yes, contact hardware personnel for maintenance. After the maintenance is complete, go to Step 11.
    • If no, go to Step 14.

    For example, the following command output indicates that no exceptions occur:

    

11. Wait about 2 minutes and check whether the alarm is cleared.
    • If yes, no further action is required.
    • If no, go to Step 14.

Check whether the disk partitions are normal.

12. Log in to the node for which the alarm was generated as user root and run the following command based on the mount point information in the alarm location information to check whether the system logs contain exceptions:

    grep -rn "blk_update_request:" /var/log/messages | grep "error" | grep "${Mount point}" | grep "sector"

    • If yes, contact hardware personnel for maintenance. After the maintenance is complete, go to Step 13.
    • If no, go to Step 14.

13. Manually clear the content queried in Step 12 from the /var/log/message file and check whether the alarm is cleared 2 minutes later.
    • If yes, no further action is required.
    • If no, go to Step 14.

Collect fault information.

14. On FusionInsight Manager, choose O&M > Log > Download.
15. Expand the Service drop-down list, select NodeAgent for the target cluster, and click OK.
16. Click the edit icon in the upper right corner and select a time span starting 10 minutes before and ending 10 minutes after when the alarm was generated. Then, click Download to collect the logs.
17. Send the collected fault logs to O&M personnel for help.

Alarm Clearing

After the fault is rectified, the system automatically clears this alarm.

Related Information

None