Help Center> Cloud Data Migration> Best Practices> IoV Big Data Service Migration to Cloud> Using CDM to Migrate Data of the Last Month
View PDF

Using CDM to Migrate Data of the Last Month

The standby HBase cluster stores about 10 TB data, and the amount of data increased in the last month is about 60 TB. Therefore, the total amount of data is about 70 TB. Company H's 20GE private line supports the cdm.xlarge cluster of CDM. Considering the migration duration, costs, and performance, the cdm.xlarge cluster is used to perform concurrent migrations.

Table 1 CDM cluster specifications

Instance Flavor

vCPUs/Memory

Maximum/Assured Bandwidth

Concurrent Extractors

Application Scenario

cdm.small

2 vCPUs and 4 GB memory

0.8/0.2 Gbit/s

10

Proof of Concept (PoC) verification and development test

cdm.medium

4 vCPUs and 8 GB memory

1.5/0.4 Gbit/s

20

A single table with fewer than 10 million pieces of data, applicable to most migration scenarios and multiple migration tasks

cdm.large

8 vCPUs and 16 GB memory

3/0.8 Gbit/s

30

A single table with 10 million or more than 10 million pieces of data

cdm.xlarge

16 vCPUs and 32 GB memory

10/4 Gbit/s

100

TB-level data migration requiring 10GE bandwidth
NOTE:

You can use multiple CDM clusters to perform migrations concurrently to improve migration efficiency. The MRS HDFS multi-replica policy occupies network bandwidth and affects the migration efficiency.

Creating Links on HUAWEI CLOUD CDM

  1. Log in to the CDM management console and create two CDM clusters.
    • Select the cdm.xlarge flavor.
    • The clusters must reside in the same VPC as MRS and DirectConnect.
    • Configure other parameters as required or retain the default values.
  2. Perform the following operations to create an Apache HDFS link:
    1. In the Operation column, click Job Management. The job management page is displayed.
    2. Choose Link Management > Create Link. On the page that is displayed, select Apache HDFS.

    3. Click Next and configure the link parameters. The URI format is hdfs://NameNode IP address:Port number. If Kerberos authentication is not enabled in the CDH cluster, set Authentication Method to SIMPLE.

    4. Click Test. If a test success message is displayed in the upper right corner, the link works properly. Click Save.
  3. Perform the following operations to create an MRS HDFS link:
    1. Choose Link Management > Create Link. On the page that is displayed, select MRS HDFS.

    2. Click Next and configure the link parameters. Set Authentication Method to SIMPLE and retain the default run mode.

    3. Click Test. If a test success message is displayed in the upper right corner, the link works properly. Click Save.

Creating a Migration Job on HUAWEI CLOUD CDM

  1. On the job management page of the CDM cluster, choose Table/File Migration > Create Job to create jobs. Create a migration job for each table file directory.

    • Source Job Configuration
      • Source Link Name: Select the created Apache HDFS link.
      • Source Directory/File: Select the directory where the HBase table of the CDH cluster resides. For example, /hbase/data/default/table_20180815 indicates that all files in the table_20180815 directory will be migrated.
      • File Format: Select Binary.
    • Destination Job Configuration
      • Destination Link Name: Select the created MRS HDFS link.
      • Write Directory: Select the MRS HBase directory, for example, /hbase/data/default/table_20180815/. The directory must carry a table name (for example, table_20180815). If the directory does not exist, CDM automatically creates it.
      • File Format: Select Binary.
    • Retain the default values of other parameters.
  2. Click Next to configure the task. By default, Concurrent Extractors is 3. You can increase the number of concurrent extractors to improve the migration efficiency. Retain the default values of other parameters.

  3. Repeat the preceding operations to create migration jobs for other directories. The parameter settings are the same. The number of jobs in the two CDM clusters is evenly allocated and executed concurrently.
  4. After a job is executed, you can view the detailed statistics by clicking Historical Record in the Operation column.