Why Is Pod Creation or Deletion Suspended on a Node Where File Storage Is Mounted?
Symptom
On the node to which SFS or SFS Turbo volumes are mounted, pod deletion tasks stay in the Stopping state, and pod creation tasks remain Creating.
Possible Causes
- The backend file storage is deleted. As a result, the mount point cannot be accessed.
- The network between the node and the file storage is abnormal. As a result, the mount point cannot be accessed.
Solution
- Log in to the node to which the file storage is mounted and run the following command to find the mount path of the file storage:
findmnt
Example mount path: /mnt/paas/kubernetes/kubelet/pods/7b88feaf-71d6-4e6f-8965-f5f0766d9f35/volumes/kubernetes.io~csi/sfs-turbo-ls/mount
- Run the following command to access the file storage folder:
cd /mnt/paas/kubernetes/kubelet/pods/7b88feaf-71d6-4e6f-8965-f5f0766d9f35/volumes/kubernetes.io~csi/sfs-turbo-ls/mount
If the access fails, the file storage is deleted or the network between the file storage and the node is abnormal.
- Run the umount -l command to unmount the file storage.
umount -l /mnt/paas/kubernetes/kubelet/pods/7b88feaf-71d6-4e6f-8965-f5f0766d9f35/volumes/kubernetes.io~csi/sfs-turbo-ls/mount
- Restart kubelet.
systemctl restart kubelet
Root Cause
This problem usually occurs when the hard mounts are used for file storage. In this mode, all processes that access the mount point are hung until the access is successful. You can use soft mounts to avoid this issue. For details, see Setting Mount Options.
Workload Abnormalities FAQs
- How Do I Use Events to Fix Abnormal Workloads?
- What Should I Do If Pod Scheduling Fails?
- What Should I Do If a Pod Fails to Pull the Image?
- What Should I Do If Container Startup Fails?
- What Should I Do If a Pod Fails to Be Evicted?
- What Should I Do If a Storage Volume Cannot Be Mounted or the Mounting Times Out?
- What Should I Do If a Workload Remains in the Creating State?
- What Should I Do If Pods in the Terminating State Cannot Be Deleted?
- What Should I Do If a Workload Is Stopped Caused by Pod Deletion?
- What Should I Do If an Error Occurs When Deploying a Service on the GPU Node?
- Why Does Pod Fail to Write Data?
- Why Is Pod Creation or Deletion Suspended on a Node Where File Storage Is Mounted?
- How to Locate Faults Using an Exit Code?
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbotmore