Verifying the Consistency of Data Migrated from MaxCompute to DLI

Preparations

Install Edge, an MgC tool used for data verification, in the source intranet environment and register an account for using Edge. For details, see Installing Edge for Linux.

Procedure

1. Sign in to the MgC console. On the Overview page, click Create Migration Project.

   

2. In the displayed dialog box, enter a project name, select a project type, and click Confirm.
   

   To review this migration project, choose Settings > Migration Projects in the navigation pane.

   

3. Connect Edge to MgC. For more information, see Connecting the Edge Device to MgC.
4. On the Edge console, add your AK/SK pairs required for accessing MaxCompute and DLI. For more information, see Adding Resource Credentials.
   • For details about how to obtain an AK/SK pair for accessing DLI, see How Do I Obtain the AK/SK Pair?
   • For details about how to obtain an AK/SK pair for accessing MaxCompute, see Viewing the Information About AccessKey Pairs of a RAM User.

5. On the MgC console, in the navigation pane on the left, choose Research > Data Lineage. In the upper left corner of the page, choose the migration project created in step 2.

   

6. In the Metadata Collection area, click Create Connection.

   

7. On the Select Connection Type page, select MaxCompute and click Next.

   

8. Configure the parameters for creating a MaxCompute connection, and click Test. If the test is successful, the connection is set up.

   Table 1 Parameters for creating a MaxCompute connection

   Parameter	Configuration
   Connection To	Select Source.
   Connection Name	The default name is MaxCompute-4 random characters (including letters and numbers). You can also customize a name.
   Edge Device	Select the Edge device connected to MgC in step 3.
   Alibaba Cloud Credential	Select the MaxCompute credential added to Edge in step 4.
   MaxCompute Project	Enter the name of your MaxCompute project. You can obtain the project name from the MaxCompute console.
   Endpoint	Enter the endpoint of the region where the MaxCompute project is located. For details about the MaxCompute endpoints in different regions, see MaxCompute Endpoints.

9. In the Metadata Collection area, choose Create Task > Metadata Collection.

   

10. Configure the parameters for creating a metadata collection task and click Confirm.

    

    Table 2 Parameters for configuring a metadata collection task

    Parameter	Configuration
    Task Name	The default name is Metadata-Collection-4 random characters (including letters and numbers). You can also customize a name.
    Metadata Connection	Select the connection created in step 8.
    Databases	Enter the names of the databases whose metadata needs to be collected. Use commas (,) to separate the database names. NOTICE: This parameter is mandatory only if a MaxCompute metadata connection is selected.
    Concurrent Threads	Set the maximum number of threads for executing the collection. The default value is 3. The value ranges from 1 to 10. Configuring more concurrent threads means more efficient collection, but more connection and Edge device resources will be consumed.

11. In the Metadata Collection area, click Collection tasks.

    

12. Under Tasks, you can review the created metadata collection task and its settings. Click Execute Task in the Operation column to run the task. Each time the task is executed, a task execution is generated.

    

13. Click View Executions in the Operation column. Under Task Executions, you can view the execution records of the task and the status and collection results of each task execution. When a task execution enters a Completed status and the collection results are displayed, you can view the list of databases and tables extracted from collected metadata on the Tables tab.

    

14. In the navigation pane on the left, choose Migrate > Big Data Verification. If this is your first time to visit this page, select an Edge device to enable the verification feature. Click Select Edge Device. In the displayed dialog box, select the Edge device connected to MgC in step 3.

    

    

    Ensure that the selected Edge device is always Online and Enabled before your verification is complete.

15. In the Features area, click Table Management.

    

16. Under Table Groups, click Create. Configure the parameters for creating a table group and click Confirm.

    

    Table 3 Parameters for creating a table group

    Parameter	Configuration
    Table Group	Enter a name.
    Metadata Connection	Select the connection created in step 8. CAUTION: A table group can only contain tables coming from the same metadata source.
    Verification Rule	Select a rule that defines the method for verifying data consistency and the inconsistency tolerance. You can View More to see more information about the verification rules provided by MgC.
    Description (Optional)	Enter a description to identify the table group.

17. On the Table Management page, click the Tables tab, select the data tables to be added to the same table group, and choose Option > Add Tables to Group above the list. In the displayed dialog box, select the desired table group and click Confirm.

    

    

    You can manually import information of incremental data tables to MgC. For details, see Creating a Table Group and Adding Tables to the Group.

18. On the Big Data Verification page, in the Features area, click Connection Management.

    

19. Click Create Connection in the upper right corner of the page. On the Select Connection Type page, select Data Lake Insight (DLI) and click Next.

    

20. Configure the parameters listed in Table 4, and click Test. If the test is successful, the connection is set up.

    Table 4 Parameters for creating a DLI connection

    Parameter	Configuration
    Connection To	Select Target.
    Connection Name	The default name is DLI-4 random characters (including letters and numbers). You can also customize a name.
    DLI Credential	Select the DL credential added to Edge in step 4. If the selected credential is the one you currently use to access MgC, you can select This is my MgC credential, and the projects in the region you choose will be listed.
    Region	Enter the code of the target region where the data to be verified is located, for example, ap-southeast-1. For details about region codes, see Endpoints.
    Project	Enter the ID of the project where the data to be verified is stored. For details about how to obtain the project ID, see Obtaining Project Information.
    Queue	Enter the name of the DLI queue used to execute verification. The queue must be a SQL queue.

21. On the MgC console, create a verification task for MaxCompute and execute the task. For details, see Creating and Executing Verification Tasks. During the task creation, select the table group created in step 16.
    • On the Select Task Type page, select MaxCompute for Big Data Component.

      

    • Select a verification method. For details about each verification method, see Verification Methods.

22. On the MgC console, create a verification task for DLI and execute the task. For details, see Creating and Executing Verification Tasks. During the task creation, select the table group created in step 16.
    • On the Select Task Type page, choose Data Lake Insight (DLI).

      

    • Select a verification method. For details about each verification method, see Verification Methods.

23. Wait until the task executions enter a Completed status. On the Verification Results page, you can view and export the task execution results. For details, see Viewing and Exporting Verification Results.