What Should I Do When Node Metrics Are Abnormal and the Disk Status Is Either Read-Only or Shows Insufficient Space?

Symptom

One or more of the following symptoms may occur:

• The node metrics appeared abnormal, such as a read-only disk or insufficient disk space. The node was tainted by kubelet, which restricts scheduling.
• File or directory creation in the node disk directory failed.
• The workload mounted to the node disk subdirectory failed to run.
• Existing pods on the node were evicted, and their status changed to ContainerStatusUnknown.

Possible Cause

In Linux, disk resource limitations typically fall into:

• Physical disk space: refers to the actual storage used by files. When the physical disk space of a node is used up, the system cannot create files or write data. As a result, the node may be abnormal. Kubernetes detects this condition as disk pressure and applies a taint to the node, which restricts pod scheduling and affects service availability.
• inodes: store metadata for file system objects such as files, directories, device files, sockets, and pipes. They include information like file size, permissions, timestamps, and data block pointers but exclude the data content or file name. If a node directory contains a large number of small files or subdirectories, inodes may be used up, causing certain disk errors.

Solution

1. Log in to the node and run the following command to check whether the physical disk space is full.

   df -h

   If the value of Used% of the corresponding mount point is 100%, the physical disk space is used up. In this case, you need to expand the disk capacity. For details, see Expanding the Storage Space. If the value is below 100%, some physical disk space is still available.

   If information similar to that shown in the figure below is displayed, some physical disk space is available.

   

2. If there is some available physical disk space, run the following command to check the usage of available inodes.

   df -i

   Information similar to that shown in the figure below is displayed.

   

   If the value of IUse% of the corresponding mount point is 100%, inodes are used up. You can delete unnecessary files to release inodes. If there are a large number of subdirectories in the /var/paas/everest/subpathvolumes directory, you can directly delete all these subdirectories. These subdirectories are temporary during the storage mounting. Deleting them will not affect the workload availability.