Symptom

When SCSI EVS disks are used and containers are created and deleted on a CentOS node, the disks are frequently mounted and unmounted. The read/write rate of the system disk may instantaneously surge. As a result, the system is suspended, affecting the normal node running.

When this problem occurs, the following information is displayed in the dmesg log:

Attached SCSI disk
task jdb2/xxx blocked for more than 120 seconds.

Example:

Possible Cause

After a PCI device is hot added to BUS 0, the Linux OS kernel will traverse all the PCI bridges mounted to BUS 0 for multiple times, and these PCI bridges cannot work properly during this period. During this period, if the PCI bridge used by the device is updated, due to a kernel defect, the device considers that the PCI bridge is abnormal, and the device enters a fault mode and cannot work normally. If the front end is writing data into the PCI configuration space for the back end to process disk I/Os, the write operation may be deleted. As a result, the back end cannot receive notifications to process new requests on the I/O ring. Finally, the front-end I/O suspension occurs.

This problem is caused by a Linux kernel defect. For details, see the defects in Linux distributions.

Impact

CentOS Linux kernels of versions earlier than 3.10.0-1127.el7 are affected.

Solution

Install the patch.

CCE cluster versions that support the patch are as follows:

• v1.15.6-r1
• v1.15.11-r1
• v.1.17.9-r0

For existing nodes

1. Log in to the worker node as user root.
2. Run the following command to install the patch:

   cd /root; curl http://{obs_package_bucket}/cluster-versions/CCE-Kernel-Patch-1127.18.2.1.tgz -1 -O; tar -zxf CCE-Kernel-Patch-1127.18.2.1.tgz; bash update.sh; script_path=`cat /etc/crontab | grep 'run/update.sh' | awk -F " " '{print $8}'`; if [ $script_path"x" != "x" ];then sed -i "/grub2-mkconfig/d" $script_path; fi

   obs_package_bucket is the address resolved from the domain name of the OBS bucket.

3. Restart the OS after the patch is installed.

For newly created nodes

1. Log in to the CCE console. In the navigation pane, choose Resource Management > Clusters. In the card view of the cluster to which you will add nodes, click Buy Node.
2. Enter the following command in the Post-installation Script text box:

   cd /root; curl http://{obs_package_bucket}/cluster-versions/CCE-Kernel-Patch-1127.18.2.1.tgz -1 -O; tar -zxf CCE-Kernel-Patch-1127.18.2.1.tgz; bash update.sh; script_path=`cat /etc/crontab | grep 'run/update.sh' | awk -F " " '{print $8}'`; if [ $script_path"x" != "x" ];then sed -i "/grub2-mkconfig/d" $script_path; fi

   obs_package_bucket is the address resolved from the domain name of the OBS bucket.

3. After the node is created, log in to the node and restart the OS for the kernel patch to take effect. Before restarting the node, you are advised to evict the workloads on the node to avoid service interruption.

Appendix