What Can I Do If the Performance of a Linux ECS Deteriorates After the Kernel Is Updated?

Symptom

For some types of Linux ECSs, after the kernel is upgraded or the OS version is updated, the overall performance or the performance of specific applications may deteriorate.

The following instance flavors are affected:

• General computing-plus: aC9s, aC9, aC8h, aC8, and aC7
• Memory-optimized: aM9, aM8, and aM7

Possible Causes

The root cause of this performance change is the security mitigation introduced by the Linux kernel community to fix a specific CPU vulnerability known as Speculative Return Stack Overflow (SRSO), also referred to as CVE-2023-20569.

• SRSO is a speculative execution side-channel vulnerability that may allow attackers to exploit the speculative execution mechanism of the CPU to access unauthorized memory data.
• To fix the SRSO vulnerability, the Linux kernel has incorporated a software mitigation patch called SafeRET by default in its latest versions.
• The SafeRET mechanism prevents attacks by adding extra security checks and instruction sequences. These operations introduce additional CPU overhead under specific workloads. This may cause instance performance deterioration. According to the official assessment of the system, the actual security risk level of the SRSO vulnerability is relatively low in typical server production environments.

You can choose whether to disable the SafeRET mitigation based on your service scenario's security and performance priorities.

Solution 1: Retaining the Current Status (Default Security Configuration)

If you think the current ECS performance is acceptable or your services are more sensitive to potential security risks (even though they are low), you do not need to perform any operations. The instance will continue to run in a secure state with the SRSO vulnerability fixed.

Procedure for verifying the current status:

grep . /sys/devices/system/cpu/vulnerabilities/*

In the command output, search for spec_rstack_overflow. If "Mitigation: SafeRET" or similar information indicating that mitigation is enabled is displayed, SafeRET has been enabled.

Solution 2: Disabling the SafeRET Mitigation Measure (to Improve Performance)

If the performance deterioration significantly affects your services and you are willing to accept the potential security risks related to the SRSO vulnerability after evaluation, you can add mitigations=off to the kernel boot parameters to disable SafeRET.

Disabling the SafeRET mitigation measure will reduce the instance's ability to protect against the SRSO vulnerability. Perform this operation only after you fully understand the risks and confirm that your services can tolerate these risks. You are strongly advised to disable the mitigation measure only in scenarios where performance requirements are extremely strict and the security environment is relatively controllable.

The following steps require root permissions. Misoperations may cause the system to fail to start. Back up important data in advance. You are advised to verify the operations in a test environment first.

1. Modify the GRUB configuration.
   1. Run the following command to open the /etc/default/grub file using a text editor (such as vim or nano):

      sudo vi /etc/default/grub

   2. Locate the line starting with GRUB_CMDLINE_LINUX=. This line usually contains existing kernel boot parameters within quotation marks (" ").
   3. Add mitigations=off to the end of the parameter list within the quotation marks in this line to disable SafeRET.

      Ensure that you add it after the existing parameters and before the closing quotation mark, with a space separating it from the preceding parameter. For example:

      GRUB_CMDLINE_LINUX="...{existing-parameters}... mitigations=off"

   4. Save the file and exit the editor.
2. Update the GRUB configuration.

   For systems that use GRUB2 (most modern distributions such as CentOS 7/8/9, Ubuntu 18.04/20.04/22.04/24.04, Debian 10/11/12, AlmaLinux, and Rocky Linux), run the following command:

   sudo grub2-mkconfig -o /boot/grub2/grub.cfg  # Generally applicable to RHEL and CentOS
   #Alternative:
   sudo update-grub  # Generally applicable to Debian and Ubuntu

   For older systems that use GRUB, run the following command:

   sudo update-grub

3. Run the following command to restart the instance:

   sudo reboot

4. After the instance is restarted, run the following command to check whether the disabling takes effect:

   grep . /sys/devices/system/cpu/vulnerabilities/*

   Check spec_rstack_overflow. If "Vulnerable", "Not affected" (in some kernel versions), or a clear indication that mitigation has been disabled is displayed, SafeRET has been disabled.