Handling Insufficient Disk Space on a Linux ECS

Scenarios

Applications running on Linux servers continuously generate and accumulate data, occupying disk space. As the disk capacity on a server is limited, after the server runs for a long time, storage resources may be used up. If the disk space is insufficient or used up, new data cannot be written to applications. This may affect application running or even cause service interruption or system exception. Therefore, you need to monitor the disk usage and take measures to clear disk space in a timely manner.

This section describes how to check the disk usage and address the space insufficiency problems. The method varies depending on the in-use server OS. This section uses CentOS as the example OS to describe how to resolve issues related to insufficient storage space. For other Linux OSs, see the corresponding OS product documents.

Symptom

If error message "No space left on device" is returned when you create a file or run an application on a Linux ECS, the disk space on the ECS is used up. In this case, you need to identify the cause of the insufficient disk space and take measures to resolve the issue.

Possible Causes

There are several possible causes of the insufficient disk space:

• 100% disk partition usage
• 100% partition inode usage
• There are zombie processes
  

  There are unreleased spaces used by deleted files that are still held open.

• Overwritten mount points
  

  If the original file system has a directory with a large number of files inside, and you mount a new device on this directory, the original mount point is overwritten. When applications in your system continue reading and writing the original file system space, they may receive the space insufficiency message. However, this cannot be reflected when you run the df or du command to check the directory usage of the disk. Because df or du only collects statistics for the newly mounted device.

100% Disk Partition Usage

If the usage of a disk partition reaches 100%, error message "No space left on device" may be reported. You are advised to check the disk usage, identify files that occupy a large amount of space, and delete unnecessary files or expand the disk capacity.

Refer to the following commands to check the disk usage:

1. Check the disk usage.

   df -h

   [test@ecs-test-0001 ~]# df -h
   Filesystem      Size  Used  Avail Use% Mounted on
   devtmpfs       485M    0   485M   0% /dev
   tmpfs          496M    0   496M   0% /dev/shm
   tmpfs          496M  6.8M  489M   2% /run
   tmpfs          496M    0   496M   0% /sys/fs/cgroup
   /dev/vda1      9.8G  2.9G  6.4G  31% /
   tmpfs          108M    0   108M   0% /run/user/0
   /dev/vdb1      9.8G   9.8G  0   100% /data-test

   In this example, the usage of the /dev/vdb1 partition reaches 100%.

2. Go to the root directory and check which directory is using the most space.

   sudo du -sh /* | sort -rh | head -n 10

   [test@ecs-test-0001 ~]# sudo du -sh /* | sort -rh | head -n 18
   du: cannot access '/proc/1781B/task/1781B/?d/4': No such file or directory
   du: cannot access '/proc/1781B/task/1781B/?dinfo/4': No such file or directory
   du: cannot access '/proc/1781B/?d/4': No such file or directory
   du: cannot access '/proc/1781B/?dinfo/4': No such file or directory
   6G     /home
   2.1G    /usr
   413M    /var
   225M    /CloudrResetPwdAgent
   137M    /boot
   36M     /etc
   6.8M    /run
   944K    /tmp
   144K    /root
   28K     /data-test

   In this example, the /home directory occupies the largest space. Consider deleting files in this directory.

3. Go to the /home directory and check which files or directories in the /home directory are using the most space. Find large files that are consuming the most disk space.

   sudo du -sh /home/* | sort -rh | head -n 10

100% Partition Inode Usage

Inodes are data structures used by many file systems to store file metadata. Each file or directory has an inode that stores metadata such as the file permissions, owner, file size, timestamps (created and modified), and location of data blocks, but does not store the file name. File names are stored in directory entries, which work as labels pointing to inodes. The number of inodes is fixed at the time of disk formatting. When they are used up, new files cannot be created even if there is still space available on the disk. So, when the inode usage approaches or already reaches 100%, delete unnecessary files to release inodes.

1. Analyze the number of files in each level-2 directory under the root directory.
   for i in /*; do echo $i; sudo find $i | wc -l; done

   [test@ecs-test-0001 ~]# for i in /*; do echo $i; sudo find $i | wc -l; done    
   /bin
   1
   /boot
   331
   /CloudrResetPwdAgent
   198
   /data-test
   2
   /dev
   346
   /etc
   2433
   /home
   5
   /lib
   1
   /lib64
   1
   /lost+found
   1

   In this example, the directories with the most files have the highest inode usage. For example, the /etc directory has 2433 files. You can delete unnecessary files.

2. Check the number of files in the subdirectories and delete files as required.

   for i in /etc/*; do echo $i; sudo find $i | wc -l; done

There Are Zombie Processes

If a file is deleted but held open by one or more active processes, the file continues to occupy the disk space. Although the file may no longer appear in the file system's directory structure, its data remains on disk as long as any running process continues to reference it through an open file descriptor. The data is only released once all such processes have closed the file or terminated.

1. Install lsof.

   sudo yum install -y lsof

2. Check files that are deleted but still occupied by processes.

   sudo lsof | grep delete | sort -k7 -rn | more

   [test@ecs-test-0001 ~]# lsof | grep delete | sort -k7 -rn | more
   tail 619544 root 3r  REG  253,1 500000000 136507 /home/test_file (deleted)

   In this example, the file size is 500000000Byte. Determine whether zombie processes occupy too much space based on the actual condition.

   You can release space in either of the following ways:

   • Restart the server to stop all processes, thereby releasing the disk space. Restarting the server may affect services. Exercise caution when performing this operation.
   • Run the kill command to clear the process. For details, see 3.

3. Run the kill command to stop the process.
   1. Obtain the process ID.

      lsof | grep delete

   2. Run the kill command.

      kill <process-ID>

      

      If the server is running services, the services may be affected. Exercise caution when performing this operation.

Overwritten Mount Points

If all prior causes have been eliminated and the issue of insufficient space continues, it is possible that the directory's mount point has been overwritten by another file system. For example, when a new file system is mounted on top of a directory that already holds data, the existing contents become hidden, yet they continue to consume disk space. This makes it appear as if the disk space has mysteriously vanished. You can perform the following steps to check whether this problem occurs.

1. Check the mount information.

   mount

   [test@ecs-test-0001 ~]# mount
   sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
   proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
   devtmpfs on /dev type devtmpfs (rw,nosuid,size=496616k,nr_inodes=124154,mode=755)
   securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
   debugfs on /sys/kernel/debug type debugfs (rw,relatime)
   tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=101412k,mode=700)
   binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)
   /dev/vdb1 on /data-test type ext4 (rw,relatime,data=ordered)
   /dev/vdc on /data-test type ext4 (rw,relatime,data=ordered)

   In the example, two devices are mounted on /data-test, which can lead to the mount point being overwritten by the most recent mount operation.

2. Check the mount information of the current partition.

   df -h

   [test@ecs-test-0001 ~]# df -h
   Filesystem      Size  Used Avail Use% Mounted on
   devtmpfs        485M     0  485M   0% /dev
   tmpfs           496M     0  496M   0% /dev/shm
   tmpfs           496M   13M  483M   3% /run
   tmpfs           496M     0  496M   0% /sys/fs/cgroup
   /dev/vda1       9.8G  3.2G  6.1G  34% /
   tmpfs           100M    0  100M   0% /run/user/0
   /dev/vdb1       9.8G   37M  9.2G   1% /data-test

3. Run umount to unmount a partition.

   In this example, run sudo umount /mnt/vdc to unmount /dev/vdc.

   

   Unmounting a file system may interrupt your application services. Perform the unmount operation at an appropriate time.