Why Are Pods Evicted by kubelet Due to Abnormal cgroup Statistics?
Symptom
On an Arm node, pods are evicted by kubelet due to the abnormal cgroup statistics. As a result, the node runs abnormally.
kubelet keeps evicting pods. After all containers are killed, kubelet still considers that the memory is insufficient.
The resource usage is normal.
The value of usage_in_bytes of cgroup in the /sys/fs/cgroup/memory directory is abnormal.
# cd /sys/fs/cgroup/memory # cat memory.usage_in_bytes 17618837504
Possible Causes
On an Arm node, the kernel of EulerOS 2.8 and 2.9 has a bug, which causes kubelet to evict pods and results in service unavailability.
This issue has been resolved in the following versions:
- EulerOS 2.8: kernel-4.19.36-vhulk1907.1.0.h1252.eulerosv2r8.aarch64
- EulerOS 2.9: kernel-4.19.90-vhulk2103.1.0.h819.eulerosv2r9.aarch64
Solution
- If your cluster version is 1.19.16-r0, 1.21.7-r0, 1.23.5-r0, 1.25.1-r0, or later, reset the OS of the node to the latest version.
- If your cluster version does not meet the requirements, upgrade the cluster to the specified version and then reset the node OS to the latest version.
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot