ALM-12180 Suspended Disk I/O

Alarm Description

• For HDDs, the alarm is triggered when any of the following conditions is met:
  • By default, the system collects data every 3 seconds. The svctm latency reaches 6 seconds within 30 seconds in at least seven collection periods.
  • By default, the system collects data every 3 seconds. Disk queue depth (avgqu-sz) > 0 and IOPS = 0, or bandwidth = 0 and ioutil > 99% in at least 10 collection periods within 30 seconds.
  • By default, the system collects data every 3 seconds. At least 50% of detected svctm take no less than 1000 ms within 300 seconds.
• For SSDs, the alarm is triggered when any of the following conditions is met:
  • By default, the system collects data every 3 seconds. The svctm latency reaches 3 seconds within 30 seconds in at least seven collection periods.
  • By default, the system collects data every 3 seconds. Disk queue depth (avgqu-sz) > 0 and IOPS = 0, or bandwidth = 0 and ioutil > 99% in at least 10 collection periods within 30 seconds.
  • By default, the system collects data every 3 seconds. At least 50% of detected svctm take no less than 500 ms within 300 seconds.

The collection period is 3 seconds, and the detection period is 30 or 300 seconds. This alarm is automatically cleared when none of the conditions are met for three consecutive detection periods (30 or 300 seconds).

• Run the following command in the OS to collect data:

  iostat -x -t 1 1

  

  Parameters in the command output are as follows:

  avgqu-sz indicates the disk queue depth.

  The sum of r/s and w/s is the IOPS.

  The sum of rkB/s and wkB/s is the bandwidth.

  %util is the ioutil value.

• Calculate svctm as follows:

  svctm = (tot_ticks_new - tot_ticks_old) / (rd_ios_new + wr_ios_new - rd_ios_old - wr_ios_old)

  When the detection period is 30 seconds, if rd_ios_new + wr_ios_new - rd_ios_old - wr_ios_old = 0, then svctm = 0.

  When the detection period is 300 seconds and rd_ios_new + wr_ios_new - rd_ios_old - wr_ios_old = 0, if tot_ticks_new - tot_ticks_old = 0, then svctm = 0; otherwise, the value of svctm is infinite.

  The parameters can be obtained as follows:

  The system runs the cat /proc/diskstats command every 3 seconds to collect data.

  

  In the data collected for the first time, the number in the fourth column is the rd_ios_old value, the number in the eighth column is the wr_ios_old value, and the number in the thirteenth column is the tot_ticks_old value.

  In the data collected for the second time, the number in the fourth column is the rd_ios_new value, the number in the eighth column is the wr_ios_new value, and the number in the thirteenth column is the tot_ticks_new value.

  In this case, the value of svctm is as follows:

  (19571460 - 19569526) / (1101553 + 28747977 - 1101553 - 28744856) = 0.6197

Alarm Attributes

Alarm Parameters

Impact on the System

If the I/O usage keeps increasing, operations will be affected and services will be interrupted. The possible impacts are as follows:

• The system I/O performance deteriorates, which means slow response and low throughput. For example, job submission is slow, page responds slowly, interface response times out, and the system is in error or even crash.
• Customer services may be interrupted. The system may break down and the key information stored on the faulty disk may be lost.

Possible Causes

The disk is aged.

Handling Procedure

Replace the disk.

1. Log in to FusionInsight Manager and choose O&M > Alarm > Alarms.
2. View the detailed information about the alarm. Check the values of HostName and DiskName in the location information to obtain the information about the faulty disk for which the alarm is reported.
3. Check whether the host for which the alarm is generated is the active OMS node or the active node of the instance in active/standby mode.
   • If yes, go to 4.
   • If no, go to 6.

4. Log in to the node for which the alarm is generated as the root user and run the following command to check the mount point of the faulty disk:

   df -h | grep "Name of the faulty disk"

   Check whether the mount point partition of the faulty disk is the cluster software installation directory (${BIGDATA_HOME}) or data disk directory (${BIGDATA_DATA_HOME} by default).

   • If yes, go to 5.
   • If no, go to 6.

5. Trigger an active/standby switchover to rectify the fault.
   • Active OMS node

     If O&M operations cannot be performed due to slow disk faults, such as system freezing, delayed page refreshing, or slow API response, and the alarm is generated for the active OMS node, perform the following operations to trigger an active/standby switchover to restore services:

     1. Log in to the active OMS node as user omm.
     2. Run the following command to perform an active/standby switchover:
        • For the IPv4 network: ${OMS_RUN_PATH}/workspace/ha/module/hacom/tools/ha_client_tool --ip=127.0.0.1 --port=20013 --switchover --name=product
        • For the IPv6 network: ${OMS_RUN_PATH}/workspace/ha/module/hacom/tools/ha_client_tool --ip=::1 --port=20013 --switchover --name=product
     3. After the active/standby switchover is successful, the system recovers. Perform 6 to replace the faulty disk.
   • Active node of an active/standby instance

     If the alarm is generated for the active node of an instance in active/standby mode and the slow disk fault affects the running of the instance, trigger an active/standby switchover on FusionInsight Manager to restore services.

     1. Log in to FusionInsight Manager and choose Cluster > Services > Name of the desired service.
     2. On the service details page, expand the More drop-down list and select Perform xxx Switchover.
     3. In the displayed dialog box, enter the password of the current login user and click OK.
     4. In the displayed dialog box, click OK to perform active/standby switchover.
     5. After the active/standby switchover is successful, the system recovers. Perform 6 to replace the faulty disk.

6. Replace the hard disk.
7. Check whether the alarm is cleared.
   • If yes, no further action is required.
   • If no, go to 8.

Collect fault information.

8. On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
9. Select OMS for Service and click OK.
10. Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
11. Contact O&M engineers and provide the collected logs.

Alarm Clearance

This alarm is automatically cleared after the fault is rectified.

Related Information

None.