Upgrading a Node OS
CVE vulnerabilities in a node OS can lead to issues like cluster data leakage and service interruptions, posing risks to cluster stability, security, and compliance.
For OS-level security vulnerabilities, CCE provides timely notices and releases corresponding patches and remediation guides. When a new OS version becomes available, CCE includes fixes for known security vulnerabilities. If a new version has not yet been released, you can enable the cluster security service to scan nodes for vulnerabilities, obtain recommended remediation suggestions and solutions, and quickly fix the issues directly on the console.
CVE Vulnerability Fixing for a Node OS
- New OS images are released according to the CCE version schedule (typically one version per quarter). You can check the latest kernel information for node OSs in maintained clusters by referring to Node OSs. If the latest kernel version is later than the version currently running on your nodes, you can upgrade the node OS to the latest version to address known vulnerabilities by resetting the node. If the cluster version has reached EOS, upgrade the cluster version first.
- If there are still vulnerabilities that need to be fixed after the OS is upgraded to the latest version, you can enable the cluster security service and fix the vulnerabilities on the console.
- For high-risk vulnerabilities, CCE issues dedicated vulnerability notices and releases patches within the SLA timeframe. For details, see Vulnerability Fixing Policies.
Scenarios Where a Kernel Can Be Upgraded
| Kernel Software Type | Container Tunnel Network (CCE Standard Cluster) | VPC Network (CCE Standard Cluster) | Cloud Native 2.0 Network (CCE Turbo Cluster) |
|---|---|---|---|
| Kernels starting with kernel (linux in Ubuntu) | Do not upgrade the kernels manually. Doing so may cause CCE functions to become abnormal. You can upgrade the cluster to the latest version and then reset the nodes. If the cluster version is already the latest one and vulnerabilities still persist after the node reset, CCE will address the issue in subsequent versions in accordance with the Huawei Cloud vulnerability SLO. For details, see Vulnerability Fixing Policies. | An upgrade must meet the following requirements:
| Direct upgrades supported |
| Kernels starting with openvswitch and ovs | Do not upgrade the kernels manually. Doing so may cause CCE functions to become abnormal. You can upgrade the cluster to the latest version and then reset the nodes. If the cluster version is already the latest one and vulnerabilities still persist after the node reset, CCE will address the issue in subsequent versions in accordance with the Huawei Cloud vulnerability SLO. For details, see Vulnerability Fixing Policies. | Direct upgrades supported | Direct upgrades supported |
| Kernels starting with ipvlan | Direct upgrades supported | An upgrade must meet the following requirements:
| Direct upgrades supported |
| Kernels starting with libnvidia-container and nvidia-container-toolkit | Direct upgrades supported on non-GPU nodes only | Direct upgrades supported on non-GPU nodes only | Direct upgrades supported on non-GPU nodes only |
| Kernels starting with s3fs, obsfs, kata, docker, containerd, and runc | Do not upgrade the kernels manually. Doing so may cause CCE functions to become abnormal. You can upgrade the cluster to the latest version and then reset the nodes. If the cluster version is already the latest one and vulnerabilities still persist after the node reset, CCE will address the issue in subsequent versions in accordance with the Huawei Cloud vulnerability SLO. For details, see Vulnerability Fixing Policies. | ||
Precautions
- Vulnerabilities in Huawei-developed CCE components, such as Docker, containerd, runC, Kubernetes, Open vSwitch, and IPvlan, cannot be fixed through manual upgrades. You must upgrade the cluster to address these vulnerabilities.
- After upgrading GPU or NPU nodes, you need to verify the driver compatibility with the updated OSs. In some cases, the driver may also need to be upgraded.
- It is advised to upgrade an OS during off-peak hours. Before an upgrade, drain the nodes and evict service pods to other nodes.
- After upgrading an OS kernel, you must restart the node for the upgrade to take effect. Do not reset the node. Otherwise, the OS will be rolled back to the most recent version released by CCE.
Upgrade Process
You can upgrade a node OS to the latest version released by CCE, thereby fixing vulnerabilities by either resetting the node or performing a rolling upgrade.
- Node OSs in the default node pool are upgraded by resetting the nodes.
- Node OSs in a custom node pool are upgraded by performing a node pool rolling upgrade.
- In scenarios involving GPUs or NPUs, where driver and OS compatibility is required, you must verify the driver and OS mapping provided by the virtualization domain. A node OS can be upgraded only after compatibility testing is complete. If additional dependencies exist, CCE will release a new cluster version to resolve them. For example, if the driver is incompatible with the latest OS version, you must upgrade the cluster to its minor version to enable the new OS.
Enabling the Security Service to Fix Vulnerabilities
You can enable the security service either during cluster creation or after the cluster is already running. Once enabled, cluster management permissions are granted to the HSS agency, and CCE will automatically install the agent on the nodes in the cluster.
After vulnerabilities are detected, you can view details on the HSS page and fix the vulnerabilities with one click.
Manually Fixing Vulnerabilities
- Before upgrading a kernel, ensure that an image source is available. Alternatively, bind an EIP to the ECS.
- It is strongly advised to verify the upgrade process in a testing environment before applying it to a production environment.
Huawei Cloud EulerOS 2.0 Upgrade Procedure
- Log in to a node and check the kernel version.
uname -a
- If an upgrade is needed, drain the node to safely migrate service pods to other nodes to prevent service loss. This is recommended because the kernel upgrade requires a node restart, during which pods on the node will be temporarily unavailable.
- Install the required packages. These packages are necessary to ensure the kernel upgrade completes successfully.
sudo yum install -y grub2-pc-modules grub2-tools-extra grub2-pc
- Upgrade the system kernel.
sudo yum update -y kernel # Restart the node after the installation is complete. reboot
After the restart, the kernel has been upgraded to the latest version.
- If the cluster uses a tunnel network, recompile Open vSwitch. Skip this step for clusters that do not use the tunnel networks.
cd /opt/cloud/cce/kubernetes/canal/openvswitch bash can_ovs.sh uninstall bash can_ovs.sh install
- Enable node scheduling and verify the function.
EulerOS 2.9 Upgrade Procedure
- Log in to a node and check the kernel version.
uname -a
- If an upgrade is needed, drain the node to safely migrate service pods to other nodes to prevent service loss. This is recommended because the kernel upgrade requires a node restart, during which pods on the node will be temporarily unavailable.
- Upgrade the system kernel.
sudo yum update -y kernel # Restart the node after the installation is complete. reboot
After the restart, the kernel has been upgraded to the latest version.
- If the cluster uses a tunnel network, recompile Open vSwitch. Skip this step for clusters that do not use the tunnel networks.
cd /opt/cloud/cce/kubernetes/canal/openvswitch bash can_ovs.sh uninstall bash can_ovs.sh install
- Enable node scheduling and verify the function.
Ubuntu 22.04 Upgrade Procedure
- Log in to a node and check the kernel version.
uname -a
- If an upgrade is needed, drain the node to safely migrate service pods to other nodes to prevent service loss. This is recommended because the kernel upgrade requires a node restart, during which pods on the node will be temporarily unavailable.
- Obtain the software package information.
sudo apt update
- Upgrade the system kernel.
sudo apt install -y linux-image-generic # Restart the node after the installation is complete. reboot
After the restart, the kernel has been upgraded to the latest version.
- If the cluster uses a VPC network, recompile IPvlan. Skip this step for clusters that do not use the VPC networks.
cd /opt/cloud/cce/kubernetes/yangtse/ipvlan/ bash ipvlan.sh uninstall bash ipvlan.sh install reboot
- Enable node scheduling and verify the function.
Performing Post-Upgrade Verification
Create a pod on the upgraded node and check whether the node runs properly. For details, see New Pods.
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot
