Help Center/ Cloud Container Engine/ User Guide/ Nodes/ Node O&M/ Differences in kubelet and Runtime Component Configurations Between CCE and the Native Community

Updated on 2024-09-30 GMT+08:00

View PDF

Differences in kubelet and Runtime Component Configurations Between CCE and the Native Community

To maintain the stability of nodes, CCE stores Kubernetes and container runtime components on separate data disks. Kubernetes uses the /mnt/paas/kubernetes directory, and the container runtime uses the /mnt/paas/runtime directory. The paths to these directories are the same as the default paths used in the community through soft links.

Differences in Default Paths to kubelet and Runtime

Path Name	Kubernetes Native Path	Path on CCE Nodes
kubelet root-dir	/var/lib/kubelet	/mnt/paas/kubernetes/kubelet
kubelet	/var/lib/kubelet	/mnt/paas/kubernetes/kubelet The soft link from /var/lib/kubelet to /mnt/paas/kubernetes/kubelet is also available.
Container runtime (Docker)	/var/lib/docker	Data Disk Space Allocation set to shared: /mnt/paas/runtime The soft link from /var/lib/docker to /mnt/paas/runtime is also available. Data Disk Space Allocation set to specific: /var/lib/docker, which is identical to the Kubernetes native path
Container runtime (containerd)	/var/lib/containerd	Data Disk Space Allocation set to shared: /mnt/paas/runtime The soft link from /var/lib/containerd to /mnt/paas/runtime is also available. Data Disk Space Allocation set to specific: /var/lib/containerd, which is identical to the Kubernetes native path
containerd logs	/var/log/pods	/var/lib/containerd/container_logs The soft link from /var/log/pods to /var/lib/containerd/container_logs is also available.

The differences in default paths to kubelet and runtime between CCE and the native community may have the following impacts:

If a soft link file is mounted into a container, the real path that the soft link file points to cannot be accessed.
For example, if /var/log is mounted to /mnt/log in a container through a hostPath, /mnt/log/pods is an abnormal soft link file in the container, and the actual data in /var/log/pods cannot be accessed.

Mount the actual file path to the container to prevent a file read failure caused by soft links.
The kubelet root-dir path (/mnt/paas/kubernetes/kubelet) is inconsistent with the community path (/var/lib/kubelet). If the container mount path of a third-party CSI add-on is the community path, the file mounting does not take effect.
For example, when Vault (an open-source third-party add-on) uses secrets-store-csi-driver to mount a secret, if the root-dir path to the add-on is inconsistent with the CCE configuration path (the default value of the add-on is the same as the community path /var/lib/kubelet), the Vault secret cannot be obtained in the container.

This is because CSI uses mountPropagation to mount volumes to target containers. kubelet includes the mount path (/mnt/paas/kubernetes/kubelet/pods/{podID}/volume/...) related to kubelet root-dir into the CSI request for mounting a volume. When the CSI container mounts a volume to the mount path requested by kubelet CSI, if the mount path is irrelevant to the hostPath, the mount propagation will be invalid. For example, the CSI container mounts host /var/lib/kubelet to container /var/lib/kubelet, and /mnt/paas/kubernetes/kubelet/pods/{podID}/volume/... in the CSI container is irrelevant to the hostPath.

Update the root-dir path in CSI to match the kubelet root-dir path on the current CCE node.

Changing the Soft Link Mode of CCE kubelet and Runtime Components to Mounting

To ensure compatibility with third-party open-source software, you can change the soft link mode of the kubelet and runtime components in CCE clusters of v1.23.18-r0, v1.25.13-r0, v1.27-10-r0, v1.28.8-r0, v1.29.4-r0, or later versions by configuring the cluster or node pool. The node pool configuration takes priority over the cluster configuration.

This operation can only be performed through APIs.

Example of creating a cluster:

POST /api/v3/projects/{project_id}/clusters
{
       "kind": "Cluster",
       "apiVersion": "v3",
       "metadata": {
              "name": "xxxxxxxxxx",
              ...
       },
       "spec": {
              ...
              "configurationsOverride": [
                     {
                            "name": "node-config",
                            "configurations": [
                                   {
                                          "name": "enable-mount-bind",
                                          "value": true
                                   }
                            ]
                     }
              ],
              ...
       }
}

Example of creating a node pool:

{
       "kind": "NodePool",
       "apiVersion": "v3",
       "metadata": {
              "name": "xxxxxxxxx",
              ...
       },
       "spec": {
              ...
              "nodeTemplate": {
                     "configurationsOverride": [{
                            "name": "node-config",
                            "configurations": [{
                                   "name": "enable-mount-bind",
                                   "value": true
                            }]
                     }],
                     ...
              }
       }
}

Changing the soft link mode of CCE kubelet and runtime components to mounting may have the following impacts:

After the change, the source directory remains unchanged, but the target soft link file is replaced with the mounted directory. This ensures that the container that was initially mounted to the soft link can still be accessed correctly without any further adjustments needed.
After the soft link mode is changed to mounting, the source and target directories will have two identical path structures. If you search for a file in the public parent directory, you may find the same file in different paths.
For example, a file may be found in both /var/lib/kubelet and /mnt/paas/kubernetes/kubelet.
There will be two identical mounting records in the source and destination directories. If your service code relies on the mounting records, evaluate how your services will be affected.
Example