Failed to Write Files to HDFS, and Error Message "item limit of xxx is exceeded" Is Displayed

Symptom

The client or upper-layer component logs indicate that a file fails to be written to a directory on HDFS. The error information is as follows:

The directory item limit of /tmp is exceeded: limit=5 items=5.

Cause Analysis

1. The run log file /var/log/Bigdata/hdfs/nn/hadoop-omm-namenode-XXX.log of the client or NameNode contains error information "The directory item limit of /tmp is exceeded:." The error message indicates that the number of files in the /tmp directory exceeds 1048576.

   2018-03-14 11:18:21,625 | WARN  | IPC Server handler 62 on 25000 | DIR* NameSystem.startFile: /tmp/test.txt The directory item limit of /tmp is exceeded: limit=1048576 items=1048577 | FSNamesystem.java:2334
   

2. The dfs.namenode.fs-limits.max-directory-items parameter specifies the maximum number of directories or files that are not in recursion relationship in a single directory. The default value is 1048576. The value ranges from 1 to 6400000.

Solution

1. Check whether it is normal that the directory contains more than one million files that are not in recursion relationship. If it is normal, increase the value of the HDFS parameter dfs.namenode.fs-limits.max-directory-items and restart the HDFS NameNode for the modification to take effect.
2. If it is abnormal, delete unnecessary files.