Failed to Write Files Because the HDFS Directory Quota Is Insufficient

Symptom

After quota is set for a directory, writing files to the directory fails. The "The DiskSpace quota of /tmp/tquota2 is exceeded" error message is displayed.

[omm@189-39-150-115 client]$ hdfs dfs -put switchuser.py  /tmp/tquota2
put: The DiskSpace quota of /tmp/tquota2 is exceeded: quota = 157286400 B = 150 MB but diskspace consumed = 402653184 B = 384 MB

Possible Causes

The remaining space configured for the directory is less than the space required for writing files.

Cause Analysis

1. The HDFS supports setting the quota for a specific directory, that is, the maximum space occupied by files in a directory can be set. For example, the following command is used to set a maximum of 150 MB files to be written to the /tmp/tquota directory. (Space = Block size x Number of copies)

   hadoop dfsadmin -setSpaceQuota 150M /tmp/tquota2

2. Run the following command to check the configured quota for the directory. SPACE_QUOTA is the configured space quota, and REM_SPACE_QUOTA is the remaining space.

   hdfs dfs -count -q -h -v /tmp/tquota2

   Figure 1 Viewing the quota set for a directory
   
3. Analyze logs. The following log indicates that writing the file requires 384 MB space, but the current space quota is only 150 MB. Therefore, the space is insufficient. Before a file is written, the required remaining space is as follows: Block size x Number of copies. 128 MB x 3 copies = 384 MB.

   [omm@189-39-150-115 client]$ 
   [omm@189-39-150-115 client]$ hdfs dfs -put switchuser.py  /tmp/tquota2
   put: The DiskSpace quota of /tmp/tquota2 is exceeded: quota = 157286400 B = 150 MB but diskspace consumed = 402653184 B = 384 MB

Solution

1. Set a proper quota for the directory.

   hadoop dfsadmin -setSpaceQuota 150G /directory name

2. Run the following command to clear the quota:

   hdfs dfsadmin -clrSpaceQuota /directory name