Enabling Inter-Cluster Copy to Back Up Data

Scenario

DistCp is used to copy the data stored on HDFS from a cluster to another cluster. DistCp depends on the cross-cluster copy function, which is disabled by default. This function needs to be enabled in both clusters.

Modify parameters on MRS to enable cross-cluster copy.

Impact on the System

Yarn needs to be restarted to enable the cross-cluster copy function and cannot be accessed during the restart.

Prerequisites

The hadoop.rpc.protection parameter of the two HDFS clusters must be set to the same data transmission mode, which can be privacy (encryption enabled) or authentication (encryption disabled).

You can log in to FusionInsight Manager, choose Cluster > Services > HDFS > Configurations, and search for hadoop.rpc.protection.

Procedure

1. Go to the All Configurations page of the Yarn service. For details, see Modifying Cluster Service Configuration Parameters.
   

   If the Components tab is unavailable, complete IAM user synchronization first. (On the Dashboard page, click Synchronize on the right side of IAM User Sync to synchronize IAM users.)

2. In the navigation pane, choose Yarn > Distcp.

   

3. Set haclusterX.remotenn1 of dfs.namenode.rpc-address to the service IP address and RPC port number of one NameNode instance of the peer cluster, and set haclusterX.remotenn2 to the service IP address and RPC port number of the other NameNode instance of the peer cluster. Enter a value in the IP address:port format.

   

   dfs.namenode.rpc-address.haclusterX.remotenn1 and dfs.namenode.rpc-address.haclusterX.remotenn2 do not distinguish active and standby NameNode instances. To obtain the service IP address of the NameNode instance, log in to FusionInsight Manager and choose Cluster > Services > HDFS > Instances. The NameNode RPC port can be obtained by searching for dfs.namenode.rpc.port on the HDFS service configurations page. It cannot be changed on Manager. The port cannot be changed on FusionInsight Manager.

   For example, 10.1.1.1:9820 and 10.1.1.2:9820.

4. Save the configuration. On the Dashboard tab page, and choose More > Restart Service to restart the Yarn service.

   

   Operation succeeded is displayed. Click Finish. The Yarn service is started successfully.

5. Log in to the other cluster and repeat the preceding operations.