Container Engines

Introduction to Container Engines

Container engines, one of the most important components of Kubernetes, manage the lifecycle of images and containers. The kubelet interacts with a container runtime through the Container Runtime Interface (CRI).

CCE supports containerd and Docker. containerd is recommended for its shorter traces, fewer components, higher stability, and less consumption of node resources.

Kubernetes has removed dockershim from v1.24 and does not support Docker by default. For details, see Kubernetes is Moving on From Dockershim: Commitments and Next Steps. CCE is going to stop the support for Docker. Change the node container engine from Docker to containerd. For details, see Migrating Nodes from Docker to containerd.

**Table 1** Comparison between container engines
Item	containerd	Docker
Tracing	kubelet --> CRI plugin (in the containerd process) --> containerd	Docker (Kubernetes v1.23 and earlier versions): kubelet --> dockershim (in the kubelet process) --> docker --> containerd Docker (community solution for Kubernetes v1.24 or later): kubelet --> cri-dockerd (kubelet uses CRI to connect to cri-dockerd) --> docker--> containerd
Command	crictl/ctr	docker
Kubernetes CRI	Native support	Support through dockershim or cri-dockerd
Pod delayed startup	Minor	High
kubelet CPU/memory usage	Minor	High
Runtime's CPU/memory usage	Minor	High

Mapping Between Node OSs and Container Engines

VPC network clusters of v1.23 or later versions support containerd. Tunnel network clusters of v1.23.2-r0 or later versions support containerd.

**Table 2** Node OSs and container engines in CCE clusters
OS	Kernel Version	Container Engine	Container Storage Rootfs	Container Runtime
CentOS 7.6	3.x	Docker Clusters of v1.23 and later support containerd.	Clusters of v1.19.16 and earlier use Device Mapper. Clusters of v1.19.16 and later use OverlayFS.	runC
EulerOS 2.3	3.x	Docker	Device Mapper	runC
EulerOS 2.5	3.x	Docker	Device Mapper	runC
EulerOS 2.9	4.x	Docker Clusters of v1.23 and later support containerd.	OverlayFS	runC
Ubuntu 18.04	4.x	Docker Clusters of v1.23 and later support containerd.	OverlayFS	runC
Ubuntu 22.04	4.x	Docker Clusters of v1.23 and later support containerd.	OverlayFS	runC
Huawei Cloud EulerOS 1.1	3.x	Docker containerd	OverlayFS	runC
Huawei Cloud EulerOS 2.0	5.x	Docker containerd	OverlayFS	runC

**Table 3** Node OSs and container engines in CCE Turbo clusters
Node Type	OS	Kernel Version	Container Engine	Container Storage Rootfs	Container Runtime
ECS (VM)	CentOS 7.6	3.x	Docker containerd	OverlayFS	runC
	Ubuntu 18.04	4.x
	EulerOS 2.9	4.x
	Huawei Cloud EulerOS 1.1	3.x
	Huawei Cloud EulerOS 2.0	5.x
ECS (PM)	EulerOS 2.9	4.x	containerd	Device Mapper	Kata

**Table 4** Kunpeng Node OSs and container engines in CCE
OS	Kernel Version	Container Engine	Container Storage Rootfs	Container Runtime
Huawei Cloud EulerOS 2.0	5.x	Docker containerd	OverlayFS	runC
EulerOS 2.9	4.x	Docker containerd	OverlayFS	runC
EulerOS 2.8	4.x	Docker	OverlayFS	runC

Common Commands of containerd and Docker

containerd does not support Docker APIs and Docker CLI, but you can run crictl commands to implement similar functions.

**Table 5** Image-related commands
Operation	Docker Command	containerd Command
Operation	docker	crictl	ctr
List local images.	docker images	crictl images	ctr -n k8s.io i ls
Pull images.	docker pull	crictl pull	ctr -n k8s.io i pull
Push images.	docker push	None	ctr -n k8s.io i push
Delete a local image.	docker rmi	crictl rmi	ctr -n k8s.io i rm
Check images.	docker inspect	crictl inspecti	None

**Table 6** Container-related commands
Operation	Docker Command	containerd Command
Operation	docker	crictl	ctr
List containers.	docker ps	crictl ps	ctr -n k8s.io c ls
Create a container.	docker create	crictl create	ctr -n k8s.io c create
Start a container.	docker start	crictl start	ctr -n k8s.io run
Stop a container.	docker stop	crictl stop	None
Delete a container.	docker rm	crictl rm	ctr -n k8s.io c del
Connect to a container.	docker attach	crictl attach	None
Access the container.	docker exec	crictl exec	None
Query container details.	docker inspect	crictl inspect	ctr -n k8s.io c info
View container logs.	docker logs	crictl logs	None
Check the resource usage of the container.	docker stats	crictl stats	None
Update container resource limits.	docker update	crictl update	None

**Table 7** Pod-related commands
Operation	Docker Command	containerd Command
Operation	docker	crictl	ctr
List pods.	None	crictl pods	None
View pod details.	None	crictl inspectp	None
Start a pod.	None	crictl start	None
Run a pod.	None	crictl runp	None
Stop a pod.	None	crictl stopp	None
Delete a pod.	None	crictl rmp	None

Containers created and started by containerd are immediately deleted by kubelet. containerd does not support suspending, resuming, restarting, renaming, and waiting for containers, nor Docker image build, import, export, comparison, push, search, and labeling. containerd does not support file copy. You can log in to the image repository by modifying the configuration file of containerd.

Differences in Tracing

Docker (Kubernetes v1.23 and earlier versions):
kubelet --> dockershim (in the kubelet process) --> docker --> containerd
Docker (community solution for Kubernetes v1.24 or later):
kubelet --> cri-dockerd (kubelet uses CRI to connect to cri-dockerd) --> docker--> containerd
containerd:
kubelet --> CRI plugin (in the containerd process) --> containerd

Although Docker has added functions such as swarm cluster, docker build, and Docker APIs, it also introduces bugs. Compared with containerd, Docker has one more layer of calling. Therefore, containerd is more resource-saving and secure.

Container Engine Versions

Docker
- EulerOS/CentOS: docker-engine 18.9.0, a Docker version customized for CCE. Security vulnerabilities will be fixed promptly.
- Ubuntu: docker-ce: 24.0.9 (community version). You are advised to use the containerd engine for Ubuntu nodes.
  
  The open source docker-ce of Ubuntu may trigger bugs when concurrent exec operations are performed (for example, multiple exec probes are configured). You are advised to use HTTP/TCP probes.
containerd: 1.6.14

If the cluster version is v1.28.8-r0, v1.29.4-r0, v1.30.1-r0, or later, upgrade containerd to 1.7.16.