What Do I Do If Zombie memcgs Cause My System to Run Out of Memory?

Residual memory cgroups (memcgs) are a common problem in Kubernetes clusters. Too many zombie memcgs can cause a node to run out of memory, or even crash.

This problem is caused by a design defect in the kernel's memcg subsystem. When a user deletes a memcg, the memcg structure is still retained in the kernel. As a result, the system may encounter insufficient memory errors or, after running for a period of time, the system can become unresponsive.

To solve this problem, you can use the background reclamation thread memcg_recycle, which is enabled by default. You can take the following actions and adjust system parameters as needed:

1. Disable background recycling. (It is enabled by default.)

   echo 0 > /proc/sys/vm/memcg_recycle_enable

2. Enable background reclamation.

   echo 1 > /proc/sys/vm/memcg_recycle_enable

3. Check the threshold. (When the number of zombie memcgs exceeds the threshold, reclamation starts. The default value is 512.)

   cat /proc/sys/vm/memcg_recycle_threshold  

4. Change the threshold as needed. Example:

   echo 1024 > /proc/sys/vm/memcg_recycle_threshold

5. Check the reclamation speed. (You can set how many pages can be reclaimed in each period. The default value is 32,768.)

   cat /proc/sys/vm/memcg_recycle_pages_scan 

6. Change the reclamation speed as needed. Example:

   echo 65536 > /proc/sys/vm/memcg_recycle_pages_scan

1. Background reclamation of zombie memcgs consumes a certain amount of CPU. You can enable or disable background reclamation and adjust the zombie memcg threshold and reclamation speed as you need.

2. The page cache of a memory file system such as tmpfs cannot be reclaimed. So, the zombie memcgs of these file systems cannot be reclaimed.