Configuring Reserved Percentage of Disk Usage on DataNodes

Scenario

When the YARN local directory and DataNode directory are on the same disk, the larger disk can handle more tasks, storing more intermediate data in the YARN local directory.

Currently, you can set dfs.datanode.du.reserved to configure the reserved disk space on DataNodes. A small value cannot meet the requirements of a disk with large capacity. However, configuring a large value for a disk with same capacity wastes a significant amount of disk space.

To avoid this problem, a new parameter dfs.datanode.du.reserved.percentage is introduced to configure the reserved percentage of the disk space.

• If dfs.datanode.du.reserved.percentage and dfs.datanode.du.reserved are configured at the same time, the larger value of the reserved disk space calculated using the two parameters is used as the reserved space of the data nodes.
• You are advised to set dfs.datanode.du.reserved or dfs.datanode.du.reserved.percentage based on the actual disk space.

Configuration Description

Go to the All Configurations page of HDFS and enter a parameter name in the search box by referring to Modifying Cluster Service Configuration Parameters.