Configuring HDFS Disk Balancing

Scenario

DiskBalancer is an online disk balancer that balances disk data on running DataNodes based on various indicators. It works in the similar way of the HDFS Balancer. The difference is that HDFS Balancer balances data between DataNodes, while HDFS DiskBalancer balances data among disks on a single DataNode.

Data among disks may be unevenly distributed if a large number of files have been deleted from a cluster running for a long time, or disk capacity expansion is performed on a node in the cluster. Uneven data distribution may deteriorate the concurrent read/write performance of the HDFS, or cause service failure due to inappropriate HDFS write policies. In this case, the data density among disks on a node needs to be balanced to prevent heterogeneous small disks from becoming the performance bottleneck of the node.

This section applies to MRS 3.x or later.

Configuration Description

Go to the All Configurations page of HDFS and enter a parameter name in the search box by referring to Modifying Cluster Service Configuration Parameters.

**Table 1** Parameter description
Parameter	Description	Default Value
dfs.disk.balancer.auto.enabled	Indicates whether to enable the HDFS DiskBalancer function. The default value is false, indicating that this function is disabled.	false
dfs.disk.balancer.auto.cron.expression	CRON expression of the HDFS disk balancing operation, which is used to control the start time of the balancing operation. This parameter is valid only when dfs.disk.balancer.auto.enabled is set to true. The default value is *0 1 * 6**, indicating that tasks are executed at 01:00 every Saturday. For details about cron expression, see Table 2.	0 1 * * 6
dfs.disk.balancer.max.disk.throughputInMBperSec	Specifies the maximum disk bandwidth that can be used for disk data balancing. The unit is MB/s, and the default value is 10. Set this parameter based on the actual disk conditions of the cluster.	10
dfs.disk.balancer.max.disk.errors	Specifies the maximum number of errors that are allowed in a specified movement process. If the value exceeds this threshold, the movement fails.	5
dfs.disk.balancer.block.tolerance.percent	Specifies the difference threshold between the data storage capacity and optimal status of each disk during data balancing among disks. For example, the ideal data storage capacity of each disk is 1 TB, and this parameter is set to 10. When the data storage capacity of the target disk reaches 900 GB, the storage status of the disk is considered as perfect. Value range: 1 to 100.	10
dfs.disk.balancer.plan.threshold.percent	Specifies the data density difference that is allowed between two disks during disk data balancing. If the absolute value of the data density difference between any two disks exceeds the threshold, data balancing is required. Value range: 1 to 100.	10
dfs.disk.balancer.top.nodes.number	Specifies the top N nodes whose disk data needs to be balanced in the cluster.	5

Table 2 lists the CRON expressions used for HDFS disk balancing. To use this function, set dfs.disk.balancer.auto.enabled to true. Set other parameters based on the cluster status.

**Table 2** CRON expressions
Column	Description
1	Minute. The value ranges from 0 to 59.
2	Hour. The value ranges from 0 to 23.
3	Date. The value ranges from 1 to 31.
4	Month. The value ranges from 1 to 12.
5	Week. The value ranges from 0 to 6. 0 indicates Sunday.

Use Restrictions

Data can only be moved between disks of the same type. For example, data can only be moved between SSDs or between DISKs.
Enabling this function occupies disk I/O resources and network bandwidth resources of involved nodes. Enable this function in off-peak hours.
The DataNodes specified by the dfs.disk.balancer.top.nodes.number parameter are frequently calculated. Therefore, set the parameter to a small value.

Commands for using the DiskBalancer function on the HDFS client are as follows:

**Table 3** DiskBalancer commands
Syntax	Description
hdfs diskbalancer -report -top <N>	Set N to an integer greater than 0. This command can be used to query the top N nodes that require disk data balancing in the cluster.
hdfs diskbalancer -plan <Hostname\| IP Address>	This command can be used to generate a JSON file based on the DataNode. The file contains information about the source disk, target disk, and blocks to be moved. In addition, this command can be used to specify other parameters such as the network bandwidth.
hdfs diskbalancer -query <Hostname:$dfs.datanode.ipc.port>	The default port number of the cluster is 9867. This command is used to query the running status of the DiskBalancer task on the current node.
hdfs diskbalancer -execute <planfile>	In this command, planfile indicates the JSON file generated in the second command. Use the absolute path.
hdfs diskbalancer -cancel <planfile>	This command is used to cancel the running planfile. Use the absolute path.

The supergroup permission is required for running this command on the client. You can run the command as the system user hdfs who has permission to use the HDFS service. Alternatively, you can create a user with the supergroup permission in the cluster and then run the command.
Only formats and usage of commands are provided in Table 3. For more parameters to be configured for each command, run the hdfs diskbalancer -help <command> command to view detailed information.
To troubleshoot performance issues, check the cluster event information for HDFS disk balancing events. If such events occurred, check whether DiskBalancer is enabled in the cluster.
After the automatic DiskBalancer function is enabled, the ongoing task stops only after the current data balancing is complete. The task cannot be canceled during the balancing.
You can manually specify certain nodes for data balancing on the client.