Balancing Fails, and "Source and target differ in block-size" Is Displayed

Updated on 2022-12-14 GMT+08:00

View PDF

Symptom

When the distcp command is executed to copy files across clusters, the message "Source and target differ in block-size." is displayed, indicating that some files fail to be copied. Use -pb to preserve block-sizes during copy. "

 Caused by: java.io.IOException: Check-sum mismatch between hdfs://10.180.144.7:25000/kylin/kylin_default_instance_prod/parquet/f2e72874-f01c-45ff-b219-207f3a5b3fcb/c769cd2d-575a-4459-837b-a19dd7b20c27/339114721280/0.parquettar and hdfs://10.180.180.194:25000/kylin/kylin_default_instance_prod/parquet/f2e72874-f01c-45ff-b219-207f3a5b3fcb/.distcp.tmp.attempt_1523424430246_0004_m_000019_2. Source and target differ in block-size. Use -pb to preserve block-sizes during copy. Alternatively, skip checksum-checks altogether, using -skipCrc. (NOTE: By skipping checksums, one runs the risk of masking data-corruption during file-transfer.)        at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.compareCheckSums(RetriableFileCopyCommand.java:214)

Possible Causes

This is not a version-related problem. When you run the distcp command to copy files, the block size of the source file is not recorded by default. As a result, the verification fails when the block size of the source file is not 128 MB. In this case, you need to add parameter -pb to the distcp command.

Cause Analysis

The block size is set when data is written to HDFS. The default block size is 128 MB. The size of files written by some components or service programs may not be 128 MB, for example, 8 MB.
```
<name>dfs.blocksize</name>
<value>134217728</value>
```
Figure 1 Size of files written by some components or service programs
DistCp reads the file from a source cluster and writes it to a destination cluster. By default, the value of dfs.blocksize in the MapReduce task is used as the block size, whose default value is 128 MB.
After DistCp finishes writing a file, the system performs verification based on the physical size of the block. Because the block size of the file in the source cluster is different from that of the file in the destination cluster, the splitting sizes are different. As a result, the verification fails.
For example, in the preceding file, there are three blocks (17.9/8 MB = 3 blocks) in the old cluster and one block (17.9/128 MB = 1 block) in the new cluster. Therefore, the verification fails because the physical size of the disk is divided.

Solution

Add parameter -pb in the distcp command. This parameter is used to reserve the block size when distcp is used to ensure that the block size of the new cluster is the same as that of the old cluster.

Figure 2 Size of the reserved block during distcp command execution
Click to enlarge