Node Mount Points

Check Items

Check whether there are inaccessible mount points on the node.

Solution

Scenario: There are inaccessible mount points on the node.

If a node has an inaccessible mount point, the possible cause is that the node or a pod on the node uses network storage via NFS (common NFS types include OBSFS and SFS), and the node is disconnected from the remote NFS server. As a result, the mount point becomes invalid, and all processes that access the mount point are suspended in the D state.

1. Log in to the node.
2. Create a script file, for example, /tmp/check_hang_mount.sh on the node. The content of the script file is as follows:

   for mount_path in `cat /proc/self/mountinfo | awk '{print $5}' | grep -v netns`
   do  
       timeout 10 sh -c "cd $mount_path"
       if [ $? == 124 ];then
           echo "$mount_path hang mount"
       fi
   done

3. Run the saved script and check the output.

   

   The mount points of the /root/foo and /root/bar folders are incorrect.

4. Run the following command to check the suspended mount points:

   mount -n | grep /root/foo

   

   When a mount point is suspended, it typically indicates that services are not using it anymore. After confirming that the mount point is no longer required, run the following command to unmount it and then re-execute the previously mentioned script:

   umount -l -f localhost:/tmp/nfs

   

   After the execution, perform the check again on the pre-upgrade check page.