What Should I Do If "Failed to reclaim image" Is Displayed in the Node Events?

Symptom

In the event of a node, the alarm "Failed to reclaim image" is repeatedly generated. The following shows an example:

wanted to free xx bytes, but freed xx bytes space with errors in image deletion: rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference "imageName:tag" (must force) - container 966fce58d9b8 is using its referenced image 50a7aa6fa56a

In this event, the container with ID 966fce58d9b8 was stopped but not completely deleted.

Possible Causes

kubelet periodically reclaims images that are not in use based on the imageGCHighThresholdPercent and imageGCLowThresholdPercent parameters. If you run the docker or crictl command on a node to start a container, the container will be in an exit state but is not fully deleted after being stopped. This means that the container still needs the image. However, kubelet cannot detect whether the image is being used by the container if it does not belong to any pods on the node. If kubelet attempts to delete the container image, the container runtime will stop it because the container still needs the image. As a result, kubelet is unable to regularly reclaim the container image.

Solution

Log in to the node, get the container for which the alarm is generated, and check whether the container is exited. Replace {containerId} with the container ID in the alarm.

• To get the container on a node using Docker, run the following command:

  docker ps -a | grep {containerId}

• To get the container on a node using containerd, run the following command:

  crictl ps -a | grep {containerId}

If the container is no longer used, delete this container. Replace {containerId} with the container ID in the alarm.

• To delete the container on a node using Docker, run the following command:

  docker rm {containerId} 

• To delete the container on a node using containerd, run the following command:

  crictl rm {containerId}

After the faulty container is deleted, kubelet can reclaim images normally.