HDFS客户端无法删除超长目录

问题背景与现象

执行hadoop fs -rm -r -f obs://<obs_path>命令，删除OBS超长目录出现如下报错：

2022-02-28 17:12:45,605 INFO internal.RestStorageService: OkHttp cost 19 ms to apply http request
2022-02-28 17:12:45,606 WARN internal.RestStorageService: Request failed, Response code: 400; Request ID: 0000017F3F9A8545401491602FC8CAD9; Request path: http://wordcount01-fcq.obs.xxx.xxxx.xxx.com/user%2Froot%2F.Trash%2FCurrent%2Ftest1%2F12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456
2022-02-28 17:12:45,606 WARN services.AbstractClient: Storage|1|HTTP+XML|getObjectMetadata||||2022-02-28 17:12:45|2022-02-28 17:12:45|||400|
2022-02-28 17:12:45,607 INFO log.AccessLogger: 2022-02-28 17:12:45 605|com.obs.services.internal.RestStorageService|executeRequest|560|OkHttp cost 19 ms to apply http request
2022-02-28 17:12:45 606|com.obs.services.internal.RestStorageService|handleThrowable|221|Request failed, Response code: 400; Request ID: 0000017F3F9A8545401491602FC8CAD9; Request path: http://wordcount01-fcq.obs.xxx.xxxx.xxx.com/user%2Froot%2F.Trash%2FCurrent%2Ftest1%2F12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456
2022-02-28 17:12:45 606|com.obs.services.AbstractClient|doActionWithResult|404|Storage|1|HTTP+XML|getObjectMetadata||||2022-02-28 17:12:45|2022-02-28 17:12:45|||400|

原因分析

使用rm命令从HDFS中删除某些内容时，文件或目录不会立即被清除，它们将被移动到回收站Current目录（/user/${username}/.Trash/current）中。

解决办法

使用skipTrash命令可以跳过HDFS回收站，直接删除。使用前先设置HDFS客户端配置项“dfs.client.skipTrash.enabled=true”。

1. 以root用户登录集群任一Master节点。
2. 执行如下命令编辑HDFS用到的hdfs-site.xml文件。

   vim 客户单安装目录/HDFS/hadoop/etc/hadoop/hdfs-site.xml

3. 在hdfs-site.xml文件中增加如下内容。

   <property>
   <name>dfs.client.skipTrash.enabled</name>
   <value>true</value>
   </property>

4. 执行以下命令直接删除OBS超长目录。

   hadoop fs -rm -r -f –skipTrash obs://<obs_path>

5. 登录集群其他Master节点，执行2~4，直到集群所有Master节点操作完成。