HDFS File Fails to Be Read, and Error Message "FileNotFoundException" Is Displayed

Symptom

In MapReduce tasks, all Map tasks are successfully executed, but Reduce tasks fail. The error message "FileNotFoundException...No lease on...File does not exist" is displayed in the logs.

Error: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): No lease on /user/sparkhive/warehouse/daas/dsp/output/_temporary/1/_temporary/attempt_1479799053892_17075_r_000007_0/part-r-00007 (inode 6501287): File does not exist. Holder DFSClient_attempt_1479799053892_17075_r_000007_0_-1463597952_1 does not have any open files.
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:3350)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.java:3442)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:3409)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.complete(NameNodeRpcServer.java:789)

Cause Analysis

"FileNotFoundException...No lease on...File does not exist" indicates that the file is deleted during the operation.

1. Search for the file name in the NameNode audit log of HDFS (/var/log/Bigdata/audit/hdfs/nn/hdfs-audit-namenode.log of the active NameNode) to confirm the creation time of the file.
2. Search the NameNode audit logs that are generated within the time range from the file creation to the time of exception occurrence and determine whether the file is deleted or moved to another directory.
3. If the file is not deleted or moved, the parent directory of the file may be deleted or moved. You need to search the upper-layer directory. In this example, the parent directory of the file is deleted.

   2017-05-31 02:04:08,286 | INFO  | IPC Server handler 30 on 25000 | allowed=true      ugi=appUser@HADOOP.COM (auth:TOKEN) ip=/192.168.1.22     cmd=delete      src=/user/sparkhive/warehouse/daas/dsp/output/_temporary      dst=null    perm=null proto=rpc | FSNamesystem.java:8189

   

   • The preceding log indicates that the appUser user of the 192.168.1.22 node deletes /user/sparkhive/warehouse/daas/dsp/output/_temporary.
   • Run the zgrep "file name" *.zip command to search for the contents of the .zip package.

Solution

1. Check the service to find out why the file or the parent directory of the file is deleted.