Why Is Pod Creation or Deletion Suspended on a Node Where File Storage Is Mounted?

Symptom

On the node to which SFS or SFS Turbo volumes are mounted, pod deletion tasks stay in the Stopping state, and pod creation tasks remain Creating.

Possible Causes

• The backend file storage is deleted. As a result, the mount point cannot be accessed.
• The network between the node and the file storage is abnormal. As a result, the mount point cannot be accessed.

Solution

1. Log in to the node to which the file storage is mounted and run the following command to find the mount path of the file storage:

   findmnt

   Example mount path: /mnt/paas/kubernetes/kubelet/pods/7b88feaf-71d6-4e6f-8965-f5f0766d9f35/volumes/kubernetes.io~csi/sfs-turbo-ls/mount

2. Run the following command to access the file storage folder:

   cd /mnt/paas/kubernetes/kubelet/pods/7b88feaf-71d6-4e6f-8965-f5f0766d9f35/volumes/kubernetes.io~csi/sfs-turbo-ls/mount

   If the access fails, the file storage is deleted or the network between the file storage and the node is abnormal.

3. Run the umount -l command to unmount the file storage.

   umount -l /mnt/paas/kubernetes/kubelet/pods/7b88feaf-71d6-4e6f-8965-f5f0766d9f35/volumes/kubernetes.io~csi/sfs-turbo-ls/mount

4. Restart kubelet.

   systemctl restart kubelet

Root Cause

This problem usually occurs when the hard mounts are used for file storage. In this mode, all processes that access the mount point are hung until the access is successful. You can use soft mounts to avoid this issue. For details, see Setting Mount Options.