Why Are Pods Evicted by kubelet Due to Abnormal cgroup Statistics?

Symptom

On an Arm node, pods are evicted by kubelet due to the abnormal cgroup statistics. As a result, the node runs abnormally.

kubelet keeps evicting pods. After all containers are killed, kubelet still considers that the memory is insufficient.

The resource usage is normal.

The value of usage_in_bytes of cgroup in the /sys/fs/cgroup/memory directory is abnormal.

# cd /sys/fs/cgroup/memory
# cat memory.usage_in_bytes
17618837504

Possible Causes

On an Arm node, the kernel of EulerOS 2.8 and 2.9 has a bug, which causes kubelet to evict pods and results in service unavailability.

This issue has been resolved in the following versions:

• EulerOS 2.8: kernel-4.19.36-vhulk1907.1.0.h1252.eulerosv2r8.aarch64
• EulerOS 2.9: kernel-4.19.90-vhulk2103.1.0.h819.eulerosv2r9.aarch64

Solution

• If your cluster version is 1.19.16-r0, 1.21.7-r0, 1.23.5-r0, 1.25.1-r0, or later, reset the OS of the node to the latest version.
• If your cluster version does not meet the requirements, upgrade the cluster to the specified version and then reset the node OS to the latest version.