Verifying the Consistency of Data Migrated Between MRS Doris Clusters

Preparations

Install the MgC Agent, an MgC tool used for data verification, in the source intranet environment and register an account for using the MgC Agent. For details, see Installing the MgC Agent for Linux.

Procedure

1. Sign in to the MgC console.
2. In the navigation pane on the left, choose Settings.
3. Under Migration Projects, click Create Project.
   Figure 1 Creating a project
   

4. Set Project Type to Complex migration (for big data), enter a project name, and click Create.
   Figure 2 Creating a big data migration project
   

5. Connect the MgC Agent to MgC. For more information, see Connecting the MgC Agent to MgC.
6. After the connection is successful, add the username/password pairs for accessing the source and target MRS Doris clusters to the MgC Agent. For more information, see Adding Resource Credentials.
7. In the navigation pane, choose Migrate > Big Data Verification. In the navigation pane, under Project, select the project created in step 4.
8. If you are performing a big data verification with MgC for the first time, select your MgC Agent to enable this feature. Click Select MgC Agent. In the displayed dialog box, select the MgC Agent you connected to MgC from the drop-down list.
   

   Ensure that the selected MgC Agent is always Online and Enabled before your verification is complete.

9. In the Features area, click Preparations.
10. Choose Connection Management and click Create Connection.
    Figure 3 Creating a connection
    

11. On the Select Connection Type page, select Doris and click Next.

    

12. Configure the parameters for creating a Doris connection, and click Test. If the test is successful, the connection is set up.

    Table 1 Parameters for creating a Doris connection

    Parameter	Configuration
    Connection To	Select Source.
    Connection Name	The default name is Doris-4 random characters (including letters and numbers). You can also customize a name.
    MgC Agent	Select the MgC Agent connected to MgC in step 5.
    Doris Credential	Select the credential you added to the MgC Agent for accessing the source MRS Doris cluster in step 6.
    Database IP Address	Enter the IP address of the Doris FE instance that is connected to the source Doris database. To obtain the IP address of a Doris FE instance, log in to the MRS Manager of the cluster and choose Cluster > Services > Doris > Instances to view the IP address of any FE instance.
    Database Port	Enter the port for connecting to the source Doris database. The database connection port is the query connection port of the Doris FE. To obtain the port, you can log in to the MRS Manager, choose Cluster > Services > Doris > Configurations, and query the value of query_port of the Doris service.
    Database Name	Enter the name of the source Doris database.
    Collect Usage Metrics	This parameter is optional. If this option is enabled, usage metrics for your big data resources will be collected during the execution of tasks created using this connection. The collected information is used to generate reports on the MgC console and for performance optimization. NOTICE: Before using this function, ensure that the Huawei Cloud account you added to the MgC Agent has the read-only permission for MRS and DLI. If the selected credential is the one you currently use to access MgC, you can select This is my MgC credential, and the projects in the region you choose will be listed. Under Region, select the region where the data to be verified is located. Under Project, select the project where the data to be verified is stored. Under Cluster ID, enter the ID of the cluster where the data to be verified is located. If the selected Doris credential is not the one you currently use to access MgC: Under Region ID, enter the ID of the region where the data to be verified is located. For example, if the region is CN South-Guangzhou, enter cn-south-1. Under Project ID, enter the project ID corresponding to the region. Under Cluster ID, enter the ID of the cluster where the data to be verified is located. NOTE: To view the region ID and project ID, choose My Credentials > API Credentials. For details about how to obtain the cluster ID, see Obtaining an MRS Cluster ID.

13. After the connection test is successful, click Confirm. The cloud service connection is set up.
14. Choose Metadata Management and click Create Metadata Collection Task.
    Figure 4 Create Metadata Collection Task
    

15. Configure the parameters for creating a metadata collection task and click Confirm.

    Table 2 Parameters for configuring a metadata collection task

    Parameter	Configuration
    Task Name	The default name is Metadata-Collection-4 random characters (including letters and numbers). You can also specify a name.
    Metadata Connection	Select the connection created in step 12.
    Databases	Enter the names of the databases whose metadata needs to be collected. Use commas (,) to separate the database names. If no database name is specified, the metadata of all databases is collected.
    Concurrent Threads	Set the maximum number of threads for executing the collection. The default value is 3. The value ranges from 1 to 10. Configuring more concurrent threads means more efficient collection, but more connection and MgC Agent resources will be consumed.

16. Under Tasks, you can review the created metadata collection task and its settings. You can modify the task by choosing More > Modify in the Operation column.
    Figure 5 Managing a metadata collection task
    

17. Click Execute Task in the Operation column to run the task. Each time the task is executed, a task execution is generated.
18. Click View Executions in the Operation column. Under Task Executions, you can view the execution records of the task and the status and collection result of each task execution. When a task execution enters a Completed status and the collection results are displayed, you can view the list of databases and tables extracted from collected metadata on the Tables tab.
    Figure 6 Managing task executions
    

19. In the Features area, click Table Management.
20. Under Table Groups, click Create. Configure the parameters for creating a table group and click Confirm.

    

    Table 3 Parameters for creating a table group

    Parameter	Description
    Table Group	Enter a name.
    Metadata Connection	Select the connection created in step 12. CAUTION: A table group can only contain tables coming from the same metadata source.
    Verification Rule	Select a rule that defines the method for verifying data consistency and the inconsistency tolerance. You can View More to see the details about the verification rules provided by MgC.
    Description (Optional)	Enter a description to identify the table group.

21. On the Table Management page, click the Tables tab, select the data tables to be added to the same table group, and choose Option > Add Tables to Group above the list. In the displayed dialog box, select the desired table group and click Confirm.

    

    

    You can manually import information of incremental data tables to MgC. For details, see Creating a Table Group and Adding Tables to the Group.

22. In the Features area, click Preparations.
23. Choose Connection Management and click Create Connection.
    Figure 7 Creating a connection
    

24. On the Select Connection Type page, select Doris and click Next.

    

25. Set connection parameters based on Table 4 and click Test. If the test is successful, the connection is set up.

    Table 4 Parameters for creating a Doris connection

    Parameter	Configuration
    Connection To	Select Target.
    Connection Name	The default name is Doris-4 random characters (including letters and numbers). You can also customize a name.
    MgC Agent	Select the MgC Agent connected to MgC in step 5.
    Doris Credential	Select the credential you added to the MgC Agent for accessing the target MRS HBase cluster in step 6.
    Database IP Address	Enter the IP address of the Doris FE instance that is connected to the target Doris database. To obtain the IP address of a Doris FE instance, log in to the MRS Manager of the cluster and choose Cluster > Services > Doris > Instances to view the IP address of any FE instance.
    Database Port	Enter the port for connecting to the target Doris database. The database connection port is the query connection port of the Doris FE. To obtain the port, you can log in to MRS Manager, choose Cluster > Services > Doris > Configurations, and query the value of query_port of the Doris service.
    Database Name	Enter the name of the MRS Doris database where the data to be verified is located.
    Collect Usage Metrics	This parameter is optional. If this option is enabled, usage metrics for your big data resources will be collected during the execution of tasks created using this connection. The collected information is used to generate reports on the MgC console and for performance optimization. NOTICE: Before using this function, ensure that the Huawei Cloud account you added to the MgC Agent has the read-only permission for MRS and DLI. If the selected credential is the one you currently use to access MgC, you can select This is my MgC credential, and the projects in the region you choose will be listed. Under Region, select the region where the data to be verified is located. Under Project, select the project where the data to be verified is stored. Under Cluster ID, enter the ID of the cluster where the data to be verified is located. If the selected Doris credential is not the one you currently use to access MgC: Under Region ID, enter the ID of the region where the data to be verified is located. For example, if the region is CN South-Guangzhou, enter cn-south-1. Under Project ID, enter the project ID corresponding to the region. Under Cluster ID, enter the ID of the cluster where the data to be verified is located. NOTE: To view the region ID and project ID, choose My Credentials > API Credentials. For details about how to obtain the cluster ID, see Obtaining an MRS Cluster ID.

26. On the MgC console, create a verification task for the source and target clusters, respectively, and execute the tasks. For details, see Creating and Executing Verification Tasks. During the task creation, select the table group created in step 20.
    • On the Select Task Type page, choose Doris.

      

    • Select a verification method. For details about each verification method, see Verification Methods.

27. Wait until the task executions enter a Completed status. You can view and export the task execution results on the Verification Results page. For details, see Viewing and Exporting Verification Results.