Verifying the Consistency of Data Migrated from Alibaba Cloud ApsaraDB for HBase Enhanced Edition to Huawei Cloud MRS HBase

Preparations

• Install the MgC Agent, a tool used for data verification, in the source intranet environment. For details, see Installing the MgC Agent on Linux.
• Add the IP address of the server where the MgC Agent is installed to the whitelist of the HBase Enhanced Edition cluster. For details, see Configuring a Whitelist.
• Applying for a public endpoint for the HBase Enhanced Edition cluster
• Add the mappings between the hostnames and IP addresses of all nodes in the source and target clusters to the /etc/hosts file on the server where the MgC Agent is installed.
  1. On the server where the MgC Agent is installed, open the /etc/hosts file.
  2. In the /etc/hosts file, add a line for each node in the source and target clusters in the following format:

     <IP-address> <hostname>

     For example, if a node uses the IP address 192.168.1.1 and has the hostname source-node-01, add the following information:

     192.168.1.1 source-node-01

  3. After all mappings are added, save and close the /etc/hosts file.
  4. Ping a hostname to check whether it can be resolved successfully. For example:

     ping source-node-01

Procedure

1. Sign in to the MgC console.
2. In the navigation pane on the left, choose Settings.
3. Under Migration Projects, click Create Project.
   Figure 1 Creating a project
   

4. Set Project Type to Big data migration, enter a project name, and click Create.
   Figure 2 Creating a big data migration project
   

5. Connect the MgC Agent to MgC. For more information, see Connecting the MgC Agent to MgC.
6. After the connection is successful, add credentials for accessing the source and target HBase clusters to the MgC Agent. For details, see Adding Resource Credentials.
   • When adding the credential of the source Alibaba Cloud HBase Enhanced Edition cluster, set Resource Type to Big Data - Lindorm and Authentication Method to Username/Password. Then enter the username and password for logging in to the cluster management system. You can view them in the cluster overview.
   • When adding the credential of the target MRS HBase cluster, set Resource Type to Big Data - HBase and Authentication Method to Username/Key. Enter the username for logging in to the HBase client in the Username box and upload the configuration files for Key:
     • For an unsecured cluster (with Kerberos authentication disabled), you need to upload five configuration files: core-site.xml, hdfs-site.xml, yarn-site.xml, mapred-site.xml, and hbase-site.xml. For details about how to obtain the configuration files, see Preparing the Configuration Files of the Running Environment. Review whether Kerberos authentication is enabled for the cluster in the basic cluster information.
     • For a secured cluster (with Kerberos authentication enabled), upload seven files: core-site.xml, hdfs-site.xml, yarn-site.xml, krb5.conf, user.keytab, mapred-site.xml, and hbase-site.xml. The krb5.conf and user.keytab files contain the credentials of the cluster users. You can perform the following steps to obtain the two files. For details about how to obtain the other five configuration files, see Preparing the Configuration Files of the Running Environment.
       1. Log in to FusionInsight Manager, and choose System > Permission > User.
       2. Select developuser and choose More > Download Authentication Credential to download the authentication credential files.
       3. Decompress the downloaded credential file to obtain the user.keytab and krb5.conf files.

7. In the navigation pane, click Big Data Verification. In the navigation pane, under Project, select the project created in step 4.
8. If you are performing a big data verification with MgC for the first time, select your MgC Agent to enable this feature. Click Select MgC Agent. In the displayed dialog box, select the MgC Agent you connected to MgC from the drop-down list.
   

   Ensure that the selected MgC Agent is always Online and Enabled before your verification is complete.

9. In the Features area, click Preparations.
10. Choose Connection Management and click Create Connection.
    Figure 3 Creating a connection
    

11. On the Select Connection Type page, select Lindorm and click Next.
12. Set connection parameters based on Table 1 and click Test. If the test is successful, the connection is set up.

    Table 1 Parameters for creating a Lindorm connection

    Parameter	Configuration
    Connection To	Select Source.
    Connection Name	The default name is Lindorm-4 random characters (including letters and numbers). You can also customize a name.
    MgC Agent	Select the MgC Agent connected to MgC in 5.
    Lindorm Credential	Select the credential you added to the MgC Agent for accessing the source ApsaraDB for HBase Enhanced Edition cluster in 6.
    Database URL	Enter the URL for accessing the source ApsaraDB for HBase Enhanced Edition cluster database. On the ApsaraDB for HBase console, in the cluster list, click the ID of your cluster. In the database connection information, obtain the public IP address.
    Lindorm Version	Select a Lindorm version from the drop-down list. Only version 2.x is available.

13. After the connection test is successful, click Confirm. The cloud service connection is set up.
14. Choose Metadata Management and click Create Metadata Collection Task.
    Figure 4 Create Metadata Collection Task
    

15. Configure the parameters for creating a metadata collection task and click Confirm.

    Table 2 Parameters for configuring a metadata collection task

    Parameter	Configuration
    Task Name	The default name is Metadata-Collection-4 random characters (including letters and numbers). You can also customize a name.
    Metadata Connection	Select the connection created in 12.
    Databases (Optional)	Enter the names of the databases whose metadata needs to be collected. Use commas (,) to separate the database names. If no database name is specified, the metadata of all databases is collected.
    Concurrent Threads	Set the maximum number of threads for executing the collection. The default value is 3. The value ranges from 1 to 10. Configuring more concurrent threads means more efficient collection, but more connection and MgC Agent resources will be consumed.

16. Under Tasks, review the created metadata collection task and its settings. You can modify the task by choosing More > Modify in the Operation column.
    Figure 5 Managing a metadata collection task
    

17. Click Execute Task in the Operation column to run the task. Each time the task is executed, a task execution is generated.
18. Click View Executions in the Operation column. Under Task Executions, you can view the execution records of the task and the status and collection result of each task execution. When a task execution enters a Completed status and the collection results are displayed, you can view the list of databases and tables extracted from collected metadata on the Tables tab.
    Figure 6 Managing task executions
    

19. In the Features area, click Table Management.
20. Under Table Groups, click Create. Configure the parameters for creating a table group and click Confirm.

    Table 3 Parameters for creating a table group

    Parameter	Description
    Table Group	User-defined
    Metadata Connection	Select the connection created in 12. CAUTION: A table group can only contain tables coming from the same metadata source.
    Verification Rule	Select a rule that defines the method for verifying data consistency and the inconsistency tolerance. You can View More to see the details about the verification rules provided by MgC.
    Description (Optional)	Enter a description to identify the table group.

21. On the Table Management page, click the Tables tab, select the data tables to be added to the same table group, and choose Option > Add Tables to Group above the list. In the displayed dialog box, select the desired table group and click Confirm.

    

    

    You can manually import information of incremental data tables to MgC. For details, see Creating a Table Group and Adding Tables to the Group.

22. In the Features area, click Preparations.
23. Choose Connection Management and click Create Connection.
    Figure 7 Creating a connection
    

24. On the Select Connection Type page, select HBase and click Next.
25. Set connection parameters based on Table 4 and click Test. If the test is successful, the connection is set up.

    Table 4 Parameters for creating a connection to HBase

    Parameter	Configuration
    Connection To	Select Target.
    Connection Name	The default name is HBase-4 random characters (including letters and numbers). You can also customize a name.
    MgC Agent	Select the MgC Agent connected to MgC in 5.
    HBase Credential	Select the credential you added to the MgC Agent for accessing the target MRS HBase cluster in 6.
    Secured Cluster	Choose whether the cluster is secured.
    ZooKeeper IP Address	Enter the IP address for connecting to the ZooKeeper node. You can enter the public or private IP address of the ZooKeeper node.
    ZooKeeper Port	Enter the port for connecting to the ZooKeeper node. The default value is 2181.
    HBase Version	Select the HBase version.

26. On the MgC console, create a verification task for Lindorm and execute the task. For details, see Creating and Executing Verification Tasks. During the task creation, select the table group created in 20.
    • On the Select Task Type page, choose Lindorm.
    • Select a verification method. For details about each verification method, see Verification Methods.

27. On the MgC console, create a verification task for HBase and execute the task. For details, see Creating and Executing Verification Tasks. During the task creation, select the table group created in 20.
    • On the Select Task Type page, choose HBase.
    • Select a verification method. For details about each verification method, see Verification Methods.

28. Wait until the task executions enter a Completed status. On the Verification Results page, you can view and export the task execution results. For details, see Viewing and Exporting Verification Results.