Using DistCp to Copy HDFS Data Across Clusters

Scenarios

DistCp is a tool used to perform large-amount data replication between clusters or in a cluster. It uses MapReduce tasks to implement distributed copy of a large amount of data.

Notes and Constraints

• When you use DistCp to copy files to OBS, the number of nested folders (levels in the hierarchy) cannot exceed 45.
• When using DistCp to copy files from HDFS to OBS or from OBS to HDFS, you need to add -skipcrccheck. Otherwise, the DistCp command execution fails.
• During data replication between clusters, if one cluster is configured with transparent data encryption and the other is not, the following constraints apply:
  • Data within an encryption zone cannot be replicated.
  • To replicate data in a non-encryption zone, add the dfs.client.ignore.namenode.default.kms.uri=true parameter to the Client installation path/Yarn/config/core-site.xml file in the cluster where DistCp commands are executed.

Prerequisites

• You have installed the YARN client. For example, the installation directory is /opt/client.
• Service users for each component have been created by the MRS cluster administrator based on business requirements. If Kerberos authentication is enabled for the cluster (in security mode), the keytab file must be downloaded for machine-to-machine users. A human-to-machine user must change their password upon their first login. (This does not apply to normal mode.)
• For clusters with Kerberos authentication enabled (security mode), mutual trust between clusters needs to be configured. For details, see Configuring Mutual Trust Between MRS Clusters.
• If data needs to be copied between clusters, the data copy function must be enabled for both the clusters. For details, see Enabling MRS Inter-Cluster Replication.

Procedure

1. Log in to the node where the client is installed.
2. Run the following command to go to the client installation directory:

   cd /opt/client

3. Run the following command to configure environment variables:

   source bigdata_env

4. If the cluster is in security mode, the user group to which the user executing the DistCp command belongs must be supergroup and the user run the following command to perform user authentication. If Kerberos authentication is disabled for the cluster (in normal mode), user authentication is not required.

   kinit Component service user

5. Run the DistCp command. The following provides an example:

   hadoop distcp hdfs://hacluster/source hdfs://hacluster/target

Common Usage of DistCp

• The following provides an example of DistCp command. For details about related parameters, see Table 1.

  hadoop distcp -numListstatusThreads 40 -update -delete -prbugpaxtq hdfs://cluster1/source hdfs://cluster2/target

  In the preceding command:

  • -numListstatusThreads specifies the number of threads for creating the list of 40 copied files.
  • -update -delete synchronizes files from the source path to the target path and deletes unnecessary files from the target path. To copy incremental files, delete -delete from the command.
  • -prbugpaxtq, which works with -update, updates the status information of the copied files.
  • hdfs://cluster1/source indicates the source path, and hdfs://cluster2/target indicates the target path.

  Table 1 DistCp command parameters

  Parameter	Description
  -p[rbugpcaxtq]	When this parameter is used together with -update, the status information of a copied file is updated even if the content of the copied file is not updated. r: specifies the number of copies; b: specifies the size of a block; u: specifies the user to which the files belong; g: specifies the user group to which the user belongs; p: specifies the permission; c: specifies the check and the type; a: specifies the access control; t: specifies the timestamp; q: specifies the quota information.
  -i	Ignores the failure during the copying.
  -log <logdir>	Specifies the log path.
  -v	Specifies additional information in the log.
  -m <num_maps>	Specifies the maximum number of concurrent copy tasks.
  -numListstatusThreads	Specifies the number of threads for creating the list of copied files. This option speeds up the DistCp process.
  -overwrite	Overwrites the file at the target path.
  -update	A file at the target path will be updated if it is different in size and checksum from the file at the source path. For details about the -update and -overwrite options, see update and overwrite Parameters.
  -append	When this parameter is used together with the -update, the content of the file at the source path is added to the file at the target path.
  -f <urilist_uri>	Copies the content of the <urilist_uri> files into the list of files to be copied.
  -filters	Specifies a local file that contains multiple regular expressions. If a file to be copied matches a regular expression, the file will not be copied.
  -async	Runs DistCp command asynchronously.
  -atomic {-tmp <tmp_dir>}	Performs an atomic copy. You can add a temporary directory during copying.
  -bandwidth	Specifies the transmission bandwidth of each copy task, in MB/s.
  -delete	Deletes files that exist in the target path but do not exist in the source path. This option is usually used with -update, indicating that files in the source path are synchronized to the target path and redundant files in the target path are deleted.
  -diff <oldSnapshot> <newSnapshot>	Copies the differences between the old and new versions to a file of the old version at the target location.
  -skipcrccheck	Determines whether to skip the cyclic redundancy check (CRC) between the source file and the target file.
  -strategy {dynamic|uniformsize}	Specifies the copy policy. The default policy is uniformsize, that is, each task copies the same number of bytes.

• Data copy between clusters

  For example, to copy data in /foo/bar of cluster1 to /bar/foo of cluster2, run the following command:

  hadoop distcp hdfs://cluster1/foo/bar hdfs://cluster2/bar/foo

  The network between cluster1 and cluster2 must be reachable, and the two clusters must use the same HDFS version or compatible HDFS versions.

• Data copy of multiple source directories

  For example, to copy folders a and b in cluster1 to /bar/foo in cluster2, run the following command:

  hadoop distcp hdfs://cluster1/foo/a \
  hdfs://cluster1/foo/b \
  hdfs://cluster2/bar/foo

  Alternatively, you can run the following command:

  hadoop distcp -f hdfs://cluster1/srclist \
  hdfs://cluster2/bar/foo

  The content of srclist is as follows. Before running the DistCp command, upload the srclist file to HDFS.

  hdfs://cluster1/foo/a 
  hdfs://cluster1/foo/b

update and overwrite Parameters

• -update copies files that only exist in the source path to the target path or updates the copied files in the target path.

  When update is used, if the file to be copied already exists in the target path but the file content is different, the file content in the target path is updated.

• -overwrite overwrites the existing files in the target path.

  When overwrite is used, if the file to be copied already exists in the target path, the file in the target path is still overwritten.

If files with the same name exist in multiple source paths, the DistCp command fails.

If neither update nor overwrite is used and the file to be copied already exists in the target path, the file will be skipped.

The following shows the differences between using no option and using either of the two options:

Assume that the structure of a file in the source path is as follows:

hdfs://cluster1/source/first/1 
hdfs://cluster1/source/first/2 
hdfs://cluster1/source/second/10 
hdfs://cluster1/source/second/20

• Commands without options are as follows:

  hadoop distcp hdfs://cluster1/source/first hdfs://cluster1/source/second  hdfs://cluster2/target

  The preceding command creates the first and second folders in the target path by default. Therefore, the copy result is as follows:

  hdfs://cluster2/target/first/1 
  hdfs://cluster2/target/first/2 
  hdfs://cluster2/target/second/10 
  hdfs://cluster2/target/second/20

• The command with any one of the two options (for example, update) is as follows:

  hadoop distcp -update hdfs://cluster1/source/first hdfs://cluster1/source/second  hdfs://cluster2/target

  The preceding command copies only the content in the source path to the target path. Therefore, the copy result is as follows:

  hdfs://cluster2/target/1 
  hdfs://cluster2/target/2 
  hdfs://cluster2/target/10 
  hdfs://cluster2/target/20

Copying Data in an Encryption Area

This function applies only to MRS 3.2.1 or later.

By default, when DistCp is executed, the checksum provided by the file system is compared to verify that the data is copied to the target. When data is copied from a common directory to an encryption zone, from an encryption zone to a common directory, or between different directories within an encryption zone, the checksum verification fails because the data block content differs.

In this case, you can specify the -skipcrccheck and -update flags to skip the checksum verification. If you use -skipcrccheck, DistCp compares file sizes to check file integrity immediately after each file is copied.

For example, run the following command:

hadoop distcp -skipcrccheck -update /encryptedpath /commonpath

If you use DistCp to copy data from a cluster where transparent encryption is disabled to a cluster where it is enabled, the operation will fail. You must ensure that KMS transparent encryption is enabled on both clusters.

Copying Data in the Original Encryption Zone

This function applies only to MRS 3.2.1 or later.

To retain data within an HDFS encryption zone, HDFS introduces a virtual path prefixed with /.reserved/raw/. This prefix allows cluster administrators to directly access raw data blocks in the file system and copy data without an encryption key. This approach avoids decrypting and re-encrypting the data. As a result, the source data remains identical to the replicated data. However, if the replicated data is encrypted using a new EDEK, an exception may occur.

For example, run the following command:

hadoop distcp -px /.reserved/raw/encrptedpath /.reserved/raw/backpath

• The encryption attributes (such as the EDEK) are stored in the extended attributes -px. To decrypt the file later, you must retain these extended attributes by using the -px flag when you copy the encrypted data to the /.reserved/raw/ directory.
• The current version does not support cross-cluster copying using this method.
• If you run the preceding command to copy files, ensure that both directories are encrypted using the same EZK.

Uploading SFTP Data to HDFS

This function applies only to MRS 3.6.0-LTS.1 and later.

Run the following DistCp command to upload SFTP data to HDFS:

hadoop distcp -libjars ${BIGDATA_CLIENT_HOME}/HDFS/hadoop/share/adapter/lib/hadoop-distcp-sftp-*.jar -Dfs.sftp.user=root  -Dfs.sftp.job.password="{Job password}" -Dfs.sftp.client.password="{Client password}" -Dfs.sftp.impl=com.huawei.hadoop.adapter.sftp.SFTPFileSystem  -Dfs.sftp.impl.disable.cache=true sftp://{SFTP server IP address}:{SFTP server port number}/source/first hdfs://cluster1/target

Parameters in the commands are defined as follows.

curl -k -i -u user name:password -X POST -HContent-type:application/json  -d '{"plainText":"password"}' 'https://x.x.x.x:28443/web/api/v2/tools/encrypt'

java -cp ${BIGDATA_CLIENT_HOME}/HDFS/hadoop/share/adapter/lib/*:${BIGDATA_CLIENT_HOME}/HDFS/hadoop/share/hadoop/common/*:${BIGDATA_CLIENT_HOME}/HDFS/hadoop/share/hadoop/common/lib/* com.huawei.hadoop.adapter.sftp.CryptUtil

source bigdata_env

java -cp ${BIGDATA_CLIENT_HOME}/HDFS/hadoop/share/adapter/lib/*:${BIGDATA_CLIENT_HOME}/HDFS/hadoop/share/hadoop/common/*:${BIGDATA_CLIENT_HOME}/HDFS/hadoop/share/hadoop/common/lib/* com.huawei.hadoop.adapter.sftp.CryptUtil rekey

Helpful Links

• If an error is reported when you use the DistCp command to copy an empty folder, see An Error Is Reported When DistCP Is Used to Copy an Empty Folder.
• If the DistCp command fails to run in a cluster in security mode and an exception occurs, see What Should I Do If an Error Is Reported When I Run DistCp Commands?.
• If some files fail to be copied and the error message "Source and target differ in block-size. Use -pb to preserve block-sizes during copy." is displayed, see Error Message "Source and target differ in block-size" Is Displayed When DistCp Is Used to Copy Files Across Clusters.
• The following lists common problems occurred during DistCp command execution: