How Do I Prevent Zombie Processes in a Container?

Symptom

When a child process in a container is terminated, it is not reclaimed. Over time, a large number of zombie processes can accumulate and drive up resource usage.

Figure 1 Using df -m to check the disk space

Possible Cause

In a container, zombie processes are usually generated because the parent process is not correctly handling the termination signal of child processes (the parent process is not waiting for child processes to terminate). In a container environment, there is one main process (PID 1), which functions as the parent process of all other processes. This parent process needs to reclaim terminated child processes to avoid too many zombie processes from being generated. However, some service processes are not able to reclaim zombie processes. If such a process is used as PID 1 in a container, child processes in the container become zombie processes after they terminate.

This problem does not occur in some earlier Linux kernel versions because of an out-of-bounds namespace bug in these versions. A shim process outside a container is used as the parent process for the container. The shim process is able to reclaim child processes. This bug has been fixed in Linux v4.11 and later versions.

Solution

Use a process that is able to reclaim child processes (for example, bash) as the PID 1 process in a container. Alternatively, modify the service code to properly handle the termination signal of child processes in the service process.