Why Is the Space Not Released After I Delete a Large File on a Linux ECS?

Symptom

The space usage of the root directory on a Linux ECS is too high, for example, 96%, as shown in Figure 1.

Figure 1 Too high space usage of the root directory

You find that the access_log file generated by Apache occupies about 42 GB in the root directory. Then you perform the following operations attempting to release space:

1. Delete access_log.

   rm /tmp/access_log

2. Check the file system disk space usage.

   df -h

However, the output tells you that the usage is still 96%.

Possible Causes

The file you deleted is locked by another running process that keeps writing data to the file.

The following are basics about file storage in Linux, which can help you better understand the root cause.

In Linux file systems, a file is organized into two parts:

• File pointer: The pointer is stored in the file system metadata. When the file is deleted, the file pointer will be deleted from the metadata.
• File data: The file data is stored on the disk.

Generally, after the file pointer is deleted from the metadata, the disk space occupied by the file data will be marked as available. However, if the file is still being used by another running process when it is deleted, the file pointer will fail to be deleted from the metadata, and the system will determine that the file has not been deleted and does not reclaim the space occupied by the file data.

Solution

1. Run the lsof command to check whether any process keeps writing data to the access_log file.

   # lsof -n |grep delete

   Figure 2 Checking the process that locks the file
   

   As shown in the command output, the access_log file is still being used by the httpd process which keeps writing log data into the file. Value deleted in the brackets indicates that the log file has been deleted. But the space is not released because httpd keeps writing data to the file.

2. Stop or restart the httpd process, or restart the system. You are advised to empty the content of access_log, rather than deleting the file.

   Run the following command to empty access_log:

   # echo "">/access_log

   In this way, the disk space will be released immediately and the process can continue writing logs to the file. Then, run the df -h command to check the space usage again.

   Figure 3 Checking the space usage of the root directory