Help Center/ Migration Center/ Best Practices/ Verifying Big Data Consistency After Migration/ Verifying the Consistency of Data Migrated Between MRS Doris Clusters
Updated on 2024-12-03 GMT+08:00
View PDF
Share

Verifying the Consistency of Data Migrated Between MRS Doris Clusters

This section describes how to use MgC to verify the consistency of data migrated between different versions of Huawei Cloud MRS Doris clusters.

Preparations

Install Edge, an MgC tool used for data verification, in the source intranet environment and register an account for using Edge. For details, see Installing Edge for Linux.

Procedure

  1. Sign in to the MgC console. On the Overview page, click Create Migration Project.

  2. In the displayed dialog box, enter a project name, select a project type, and click Confirm.

    To review this migration project, choose Settings > Migration Projects in the navigation pane.

  3. Connect Edge to MgC. For more information, see Connecting the Edge Device to MgC.
  4. After the connection is successful, add the username/password pairs for accessing the source and target MRS Doris clusters to Edge. For more information, see Adding Resource Credentials.
  5. On the MgC console, in the navigation pane on the left, choose Research > Data Lineage. In the upper left corner of the page, choose the migration project created in step 2 from the drop-down list.

  6. In the Metadata Collection area, click Create Connection.

  7. On the Select Connection Type page, select Doris and click Next.

  8. Configure the parameters for creating a Doris connection, and click Test. If the test is successful, the connection is set up.

    Table 1 Parameters for creating a Doris connection

    Parameter

    Configuration

    Connection To

    Select Source.

    Connection Name

    The default name is Doris-4 random characters (including letters and numbers). You can also customize a name.

    Edge Device

    Select the Edge device connected to MgC in step 3.

    Doris Credential

    Select the credential used for accessing the source MRS Doris cluster you added to Edge in Step 4.

    Database IP Address

    Enter the IP address of the Doris FE instance that is connected to the source Doris database.

    To obtain the IP address of a Doris FE instance, log in to the MRS Manager of the cluster and choose Cluster > Services > Doris > Instances to view the IP address of any FE instance.

    Database Port

    Enter the port for connecting to the source Doris database.

    The database connection port is the query connection port of the Doris FE. To obtain the port, you can log in to the MRS Manager, choose Cluster > Services > Doris > Configurations, and query the value of query_port of the Doris service.

    Database Name

    Enter the name of the source Doris database.

  9. In the Metadata Collection area, choose Create Task > Metadata Collection.

  10. Configure the parameters for creating a metadata collection task and click Confirm.

    Table 2 Parameters for configuring a metadata collection task

    Parameter

    Configuration

    Task Name

    The default name is Metadata-Collection-4 random characters (including letters and numbers). You can also specify a name.

    Metadata Connection

    Select the connection created in step 8.

    Databases

    Enter the names of the databases whose metadata needs to be collected. Use commas (,) to separate the database names. If no database name is specified, the metadata of all databases is collected.

    Concurrent Threads

    Set the maximum number of threads for executing the collection. The default value is 3. The value ranges from 1 to 10. More concurrent threads means more efficient collection, but more connection and Edge device resources will be consumed.

  11. In the Metadata Collection area, click Collection tasks.

  12. Under Tasks, you can review the created metadata collection task and its settings. Click Execute Task column to run the task. Each time the task is executed, a task execution is generated.
  13. Click View Executions in the Operation column. Under Task Executions, you can view the execution records of the task and the status and collection results of each task execution. When a task execution enters a Completed status and the collection results are displayed, you can view the list of databases and tables extracted from collected metadata on the Tables tab.

  14. In the navigation pane on the left, choose Migrate > Big Data Verification. If this is your first time to visit this page, select an Edge device to enable the verification feature. Click Select Edge Device. In the displayed dialog box, select the Edge device connected to MgC in step 3.

    Ensure that the selected Edge device is always Online and Enabled before your verification is complete.

  15. In the Features area, click Table Management.

  16. Under Table Groups, click Create. Configure the parameters for creating a table group and click Confirm.

    Table 3 Parameters for creating a table group

    Parameter

    Description

    Table Group

    Enter a name.

    Metadata Connection

    Select the connection created in step 8.

    CAUTION:

    A table group can only contain tables coming from the same metadata source.

    Verification Rule

    Select a rule that defines the method for verifying data consistency and the inconsistency tolerance. You can View More to see the details about the verification rules provided by MgC.

    Description (Optional)

    Enter a description to identify the table group.

  17. On the Table Management page, click the Tables tab, select the data tables to be added to the same table group, and choose Option > Add Tables to Group above the list. In the displayed dialog box, select the desired table group and click Confirm.

    You can manually import information of incremental data tables to MgC. For details, see Creating a Table Group and Adding Tables to the Group.

  18. On the Big Data Verification page, in the Features area, click Connection Management.

  19. Click Create Connection in the upper right corner of the page. On the Select Connection Type page, select Doris and click Next.

  20. Set connection parameters based on Table 4 and click Test. If the test is successful, the connection is set up.

    Table 4 Parameters for creating a Doris connection

    Parameter

    Configuration

    Connection To

    Select Target.

    Connection Name

    The default name is Doris-4 random characters (including letters and numbers). You can also customize a name.

    Doris Credential

    Select the credential used for accessing the target MRS HBase cluster you added to Edge in step 4.

    Database IP Address

    Enter the IP address of the Doris FE instance that is connected to the target Doris database.

    To obtain the IP address of a Doris FE instance, log in to the MRS Manager of the cluster and choose Cluster > Services > Doris > Instances to view the IP address of any FE instance.

    Database Port

    Enter the port for connecting to the target Doris database.

    The database connection port is the query connection port of the Doris FE. To obtain the port, you can log in to MRS Manager, choose Cluster > Services > Doris > Configurations, and query the value of query_port of the Doris service.

    Database Name

    Enter the name of the MRS Doris database where the data to be verified is located.

  21. On the MgC console, create a verification task for the source and target Doris clusters, respectively, and execute the tasks. For details, see Creating and Executing Verification Tasks. During the task creation, select the table group created in step 16.

    • On the Select Task Type page, choose Doris.

    • Select a verification method. For details about each verification method, see Verification Methods.

  22. Wait until the task executions enter a Completed status. You can view and export the task execution results on the Verification Results page. For details, see Viewing and Exporting Verification Results.