Why Are Pods Evicted by kubelet Due to Abnormal cgroup Statistics?
Symptom
On an Arm node, pods are evicted by kubelet due to the abnormal cgroup statistics. As a result, the node runs abnormally.
kubelet keeps evicting pods. After all containers are killed, kubelet still considers that the memory is insufficient.
The resource usage is normal.
The value of usage_in_bytes of cgroup in the /sys/fs/cgroup/memory directory is abnormal.
# cd /sys/fs/cgroup/memory # cat memory.usage_in_bytes 17618837504
Possible Causes
On an Arm node, the kernel of EulerOS 2.8 and 2.9 has a bug, which causes kubelet to evict pods and results in service unavailability.
![](https://support.huaweicloud.com/intl/en-us/cce_faq/public_sys-resources/note_3.0-en-us.png)
This issue has been resolved in the following versions:
- EulerOS 2.8: kernel-4.19.36-vhulk1907.1.0.h1252.eulerosv2r8.aarch64
- EulerOS 2.9: kernel-4.19.90-vhulk2103.1.0.h819.eulerosv2r9.aarch64
Solution
- If your cluster version is 1.19.16-r0, 1.21.7-r0, 1.23.5-r0, 1.25.1-r0, or later, reset the OS of the node to the latest version.
- If your cluster version does not meet the requirements, upgrade the cluster to the specified version and then reset the node OS to the latest version.
OSs FAQs
- What Can I Do If cgroup kmem Leakage Occasionally Occurs When an Application Is Repeatedly Created or Deleted on a Node Running CentOS with an Earlier Kernel Version?
- What Should I Do If There Is a Service Access Failure After a Backend Service Upgrade or a 1-Second Latency When a Service Accesses a CCE Cluster?
- Why Are Pods Evicted by kubelet Due to Abnormal cgroup Statistics?
- When Container OOM Occurs on the CentOS Node with an Earlier Kernel Version, the Ext4 File System Is Occasionally Suspended
- What Should I Do If a DNS Resolution Failure Occurs Due to a Defect in IPVS?
- What Should I Do If the Number of ARP Entries Exceeds the Upper Limit?
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbotmore