Why Volume Mounting Failure Is Reported When a Job Pod Completes Its Task?

Symptom

When a job pod completes its task, there are still events showing that a volume failed to be mounted to the pod.

Possible Cause

When a pod of a Deployment, StatefulSet, job, or cron job is started on a node:

• The kubelet creates a podWorker (an independent goroutine) to check volume mounting into the pod at a regular interval of 0.3 seconds. If the volume mounting is not complete within 0.3 seconds or the timeout duration (123 seconds) expires, the event "Unable to mount volumes for pod ..." is reported.
• The VolumeManager (an independent goroutine) in the kubelet is responsible for mounting volumes into the pod.

For a long running pod of a Deployment or StatefulSet: If the pod is started without experiencing failures to pull images, mount volumes, assign container CIDR block, or provide node CPU or memory resources that match the pod's needs, the volume mounting will eventually be successful after a few check cycles.

For a short-lived pod of a job or cron job: The pod is very likely to exit gracefully soon after it is started. For example, the GCS Demo job in question runs only a few echo and ls commands and exits in less than 1 second. If the pod exits gracefully during the interval between two consecutive volume mounting checks, a false event notification will be generated but services are not affected. In practice, pod jobs always run longer than 1 second.

The kubelet's capability to check volume mounting is already considered in the volume mounting framework provided by the community.

Solution

A pod of a job or cron job usually runs for a short period of time. If the pod completes its task much earlier than the timeout duration of volume mounting, the volume mounting timeout event is a false event notification and can be ignored. The event has no impact on the service run by the pod.