Failed to Write Files Because the HDFS Directory Quota Is Insufficient

Updated on 2022-12-08 GMT+08:00

View PDF

Symptom

After quota is set for a directory, writing files to the directory fails. The "The DiskSpace quota of /tmp/tquota2 is exceeded" error message is displayed.

[omm@189-39-150-115 client]$ hdfs dfs -put switchuser.py  /tmp/tquota2
put: The DiskSpace quota of /tmp/tquota2 is exceeded: quota = 157286400 B = 150 MB but diskspace consumed = 402653184 B = 384 MB

Possible Causes

The remaining space configured for the directory is less than the space required for writing files.

Cause Analysis

The HDFS supports setting the quota for a specific directory, that is, the maximum space occupied by files in a directory can be set. For example, the following command is used to set a maximum of 150 MB files to be written to the /tmp/tquota directory. (Space = Block size x Number of copies)
hadoop dfsadmin -setSpaceQuota 150M /tmp/tquota2
Run the following command to check the configured quota for the directory. SPACE_QUOTA is the configured space quota, and REM_SPACE_QUOTA is the remaining space.
hdfs dfs -count -q -h -v /tmp/tquota2

Figure 1 Viewing the quota set for a directory
Analyze logs. The following log indicates that writing the file requires 384 MB space, but the current space quota is only 150 MB. Therefore, the space is insufficient. Before a file is written, the required remaining space is as follows: Block size x Number of copies. 128 MB x 3 copies = 384 MB.
```
[omm@189-39-150-115 client]$ 
[omm@189-39-150-115 client]$ hdfs dfs -put switchuser.py  /tmp/tquota2
put: The DiskSpace quota of /tmp/tquota2 is exceeded: quota = 157286400 B = 150 MB but diskspace consumed = 402653184 B = 384 MB
```

Solution

Set a proper quota for the directory.

hadoop dfsadmin -setSpaceQuota 150G /directory name
Run the following command to clear the quota:

hdfs dfsadmin -clrSpaceQuota /directory name