ALM-12180 Disk Card I/O (For MRS 2.x or Earlier)

Updated on 2024-09-23 GMT+08:00

View PDF

Description

For MRS 2.x or earlier:

For HDDs, the alarm is triggered when any of the following conditions is met:
- The system collects data every 3 seconds, and detects that the svctm value exceeds 6s for 10 consecutive periods within 30 seconds.
- The system collects data every 3 seconds, and detects that the avgqu-sz value is greater than 0, the IOPS or bandwidth is 0, and the ioutil value is greater than 99% for 10 consecutive periods within 30 seconds.
For SSDs, the alarm is triggered when any of the following conditions is met:
- The system collects data every 3 seconds, and detects that the svctm value exceeds 2s for 10 consecutive periods within 30 seconds.
- The system collects data every 3 seconds, and detects that the avgqu-sz value is greater than 0, the IOPS or bandwidth is 0, and the ioutil value is greater than 99% for 10 consecutive periods within 30 seconds.

This alarm is automatically cleared when none of the conditions are met for 90 seconds.

For MRS 1.9.3.10 or later:

For HDDs, the alarm is triggered when any of the following conditions is met:
- By default, the system collects data every 3 seconds. The svctm latency reaches 6 seconds within 30 seconds in at least seven collection periods.
- The system collects data every 3 seconds by default, and within 30 seconds, at least 10 collection cycles have a disk queue depth (avgqu-sz) > 0, IOPS = 0, or bandwidth = 0, and ioutil > 99%.
- By default, the system collects data every 3 seconds. At least 50% of detected svctm take no less than 1000 ms within 300 seconds.
For SSDs, the alarm is triggered when any of the following conditions is met:
- By default, the system collects data every 3 seconds. The svctm latency reaches 3 seconds within 30 seconds in at least seven collection periods.
- By default, the system collects data every 3 seconds. Disk queue depth (avgqu-sz) > 0 and IOPS = 0, or bandwidth = 0 and ioutil > 99% in at least 10 collection periods within 30 seconds.
- By default, the system collects data every 3 seconds. At least 50% of detected svctm take no less than 500 ms within 300 seconds.

The collection period is 3 seconds, and the detection period is 30 or 300 seconds. This alarm is automatically cleared when none of the conditions are met for three consecutive detection periods (30 or 300 seconds).

NOTE:

For details about how to obtain related parameters, see Related Information.

Attribute

Alarm ID	Alarm Severity	Auto Clear
12180	Major	Yes

Parameters

Parameter	Description
Source	Specifies the cluster or system for which the alarm is generated.
ServiceName	Specifies the service for which the alarm is generated.
RoleName	Specifies the role for which the alarm is generated.
HostName	Specifies the host for which the alarm is generated.
DiskName	Specifies the disk for which the alarm is generated.

Impact on the System

A continuously high I/O usage may adversely affect service operations and result in service loss.

Possible Causes

The disk is aged.

Procedure

Replace the disk.

Log in to FusionInsight Manager and choose O&M > Alarm > Alarms.
View the detailed information about the alarm. Check the values of HostName and DiskName in the location information to obtain the information about the faulty disk for which the alarm is reported.
Replace the faulty disk.
Check whether the alarm is cleared.
- If yes, no further action is required.
- If no, perform 5.

Collect the fault information.

On MRS Manager, choose System > Export Log.
Contact O&M engineers and send the collected logs.

Alarm Clearing

This alarm is automatically cleared after the fault is rectified.

Related Information

To obtain the related parameters, perform the following steps:

Run the following command in the OS to collect data:
iostat -x -t 1 1

The command parameters are as follows:

avgqu-sz indicates the disk queue depth.

The sum of r/s and w/s is the IOPS.

The sum of rkB/s and wkB/s is the bandwidth.

%util is the value of ioutil.
The value of svctm is calculated as follows:
svctm = (tot_ticks_new - tot_ticks_old) / (rd_ios_new + wr_ios_new - rd_ios_old - wr_ios_old)

For MRS 2.x or earlier:

If rd_ios_new + wr_ios_new - rd_ios_old - wr_ios_old is 0, then svctm is 0.

For MRS 1.9.3.10 or later:

When the detection period is 30 seconds, if rd_ios_new + wr_ios_new - rd_ios_old - wr_ios_old = 0, then svctm = 0.

When the detection period is 300 seconds and rd_ios_new + wr_ios_new - rd_ios_old - wr_ios_old = 0, if tot_ticks_new - tot_ticks_old = 0, then svctm = 0; otherwise, the value of svctm is infinite.

The parameters in the preceding expression can be obtained as follows:

Obtain the parameter values from the data collected via the cat /proc/diskstats command run by the system every 3 seconds. The following shows an example.

In the data collected for the first time, the number in the fourth column is the value of rd_ios_old, the number in the eighth column is the value of wr_ios_old, and the number in the thirteenth column is the value of tot_ticks_old.

In the data collected for the second time, the number in the fourth column is the value of rd_ios_new, the number in the eighth column is the value of wr_ios_new, and the number in the thirteenth column is the value of tot_ticks_new.

In this case, the value of svctm is as follows:

(19571460 - 19569526) / (1101553 + 28747977 - 1101553 - 28744856) = 0.6197