Verifying the Consistency of Data Migrated from Delta Lake (without Metadata) to MRS Delta Lake

Preparations

Install Edge, an MgC tool used for data verification, in the source intranet environment and register an account for using Edge. For details, see Installing Edge for Linux.

Procedure

1. Sign in to the MgC console. On the Overview page, click Create Migration Project.

   

2. In the displayed dialog box, enter a project name, select a project type, and click Confirm.
   

   To review this migration project, choose Settings > Migration Projects in the navigation pane.

   

3. Connect Edge to MgC. For more information, see Connecting the Edge Device to MgC.
4. After the connection is successful, add the username/password pairs for accessing the source Delta Lake executor and the target MRS Delta Lake executor to Edge. For more information, see Adding Resource Credentials.

   

5. On the MgC console, in the navigation pane on the left, choose Research > Data Lineage. In the upper left corner of the page, choose the migration project created in step 2.

   

6. In the Metadata Collection area, click Create Connection.

   

7. Select Delta Lake (without metadata) and click Next.

   

8. Set connection parameters based on Table 1 and click Test. If the test is successful, the connection is set up.

   Table 1 Parameters for creating a connection to Delta Lake (without metadata)

   Parameter	Configuration
   Connection To	Select Source.
   Connection Name	The default name is Delta-Lake-without-metadata-4 random characters (including letters and numbers). You can also customize a name.
   Edge Device	Select the Edge device connected to MgC in step 3.
   Executor Credential	Select the source Delta Lake executor credential added to Edge in step 4.
   Executor IP Address	Enter the IP address for connecting to the executor.
   Executor Port	Enter the port for connecting to the executor. The default port is 22.
   Spark Client Directory	Enter the absolute path of the bin directory on the Spark client.
   Environment Variable Address	Enter the absolute path of the environment variable file, for example, /opt/bigdata/client/bigdata_env. If this field is not left blank, the environment variable file is automatically sourced before commands are executed.
   SQL File Location	Enter a directory for storing the SQL files generated for consistency verification. You must have the read and write permissions for the folder. NOTICE: After the migration is complete, you need to manually clear the folders generated at this location to release storage space.

9. In the Metadata Collection area, click View Tables or Tables.

   

10. On the displayed page, click Import.

    

11. Click Download to download the import template to the local PC. Open the template, fill in table information, and save the template.
12. In the Import Tables dialog box, click Select File, choose the filled template file, and click Confirm. After the import is complete, you can view the imported tables on Tables tab.

    

13. In the navigation pane on the left, choose Migrate > Big Data Verification. If this is your first time to visit this page, select an Edge device to enable the verification feature. Click Select Edge Device. In the displayed dialog box, select the Edge device connected to MgC in step 3.

    

    

    Ensure that the selected Edge device is always Online and Enabled before your verification is complete.

14. In the Features area, click Table Management.

    

15. Under Table Groups, click Create. Configure the parameters for creating a table group and click Confirm.

    

    Table 2 Parameters for creating a table group

    Parameter	Description
    Table Group	Enter a name.
    Metadata Source	Select the connection created in step 8. CAUTION: A table group can only contain tables coming from the same metadata source.
    Verification Rule	Select a rule that defines the method for verifying data consistency and the inconsistency tolerance. MgC provides multiple verification rules for you to choose. For details about these rules, click View More.
    Description (Optional)	Enter a description to identify the table group.

16. On the Table Management page, click the Tables tab, select the data tables to be added to the same table group, and choose Option > Add Tables to Group above the list. In the displayed dialog box, select the desired table group and click Confirm.

    

    

    You can manually import information of incremental data tables to MgC. For details, see Creating a Table Group and Adding Tables to the Group.

17. Create a connection to the source and target executors separately. For details, see Creating an Executor Connection. Select the source and target executor credentials added to Edge in step 4.
18. Create a data verification task for the source and target Delta Lake clusters, respectively, and execute the tasks. For more information, see Creating and Executing Verification Tasks. During the task creation, select the table group created in step 16.
    • On the Select Task Type page, choose Delta Lake.

      

    • Select a verification method. For details about each verification method, see Verification Methods.

19. Wait until the task executions enter a Completed status. You can view and export the task execution results on the Verification Results page. For details, see Viewing and Exporting Verification Results.