Migrating Full Metadata

Prerequisites

• You have completed all preparations.
• A source connection has been created.
• Target connections have been created.

Procedure

1. Sign in to the MgC console. In the navigation pane, under Project, select your big data migration project from the drop-down list.
2. In the navigation pane on the left, choose Migrate > Big Data Migration.
3. In the upper right corner of the page, click Create Migration Task.

   

4. Select MaxCompute for Source Component, Data Lake Insight (DLI) for Target Component, and Full metadata migration for Task Type, and click Next.

   

5. Set parameters required for creating a full metadata migration task based on Table 1.

   Table 1 Parameters required for creating a full metadata migration task

   Area	Parameter	Configuration
   Basic Settings	Task Name	The default name is Full-metadata-migration-from-MaxCompute-to-DLI-4 random characters (including letters and numbers). You can also customize a name.
   	MgC Agent	Select the MgC Agent you connected to MgC in Preparations.
   Source Settings	Source Connection	Select the source connection you created.
   	MaxCompute Parameters	The parameters are optional and usually left blank. If needed, you can configure the parameters by referring to MaxCompute Documentation.
   Data Scope	By database	Enter the names of databases to be migrated in the Include Databases text box. Click Add to add more entries. A maximum of 10 databases can be added. To exclude certain tables from the migration, download the template in CSV format under Exclude Tables, add information about these tables to the template, and upload the template to MgC. For details, see steps 2 to 5.
   	By table	Download the template in CSV format. Open the downloaded CSV template file with Notepad. CAUTION: Do not use Excel to edit the CSV template file. The template file edited and saved in Excel cannot be identified by MgC. Retain the first line in the CSV template file. From the second line onwards, enter the information about tables to be migrated in the format of {MaxCompute project name},{Table name}. MaxCompute project name refers to the name of the MaxCompute project to be migrated. Table name refers to the data table to be migrated. NOTICE: Use commas (,) to separate the MaxCompute project name and the table name in each line. Do not use spaces or other separators. After adding the information about a table, press Enter to start a new line. After all table information is added, save the changes to the CSV file. Upload the edited and saved CSV file to MgC.
   	Enterprise Project ID (Optional)	Enter the ID of the enterprise project to which the target DLI databases belong. For details about how to obtain an enterprise project ID, see How Do I Obtain an Enterprise Project ID?
   	Convert Internal Table To External	If you set this option to No, the migrated data will be stored in DLI. If you set this option to Yes, the migrated data will be stored in the OBS path you specify.
   Target Settings	Overwriting Policy	Select the policy for processing tables that already exist at the target. Skip: Such tables will be skipped during the migration, and no operation will be performed on them. Rebuild: Such tables and the data in them will be deleted from the target, and new empty tables will be created at the target based on the source table settings.
   	Target Connection	Select the DLI connection with a SQL queue you created. CAUTION: Do not select the connection with a general queue configured.
   	Custom Parameters	Configure the parameters as needed. For details, see Configuration parameter description and Custom Parameters.
   	Table Type	Select an option as required.
   	File Format	Select an option as required.
   	Compression Format	Select an option as required.
   Migration Settings	Concurrency	Set the number of concurrent migration subtasks. The default value is 3. The value ranges from 1 to 10.
   	(Optional) Table Path Mapping	Map storage paths for metadata of external tables between the source and the target. This can help smoothly move the metadata of MaxCompute external tables from Alibaba OSS to Huawei Cloud OBS. The mapped paths in Huawei Cloud OBS must be created in advance. In the paths, the bucket names can be different, and other parts must be the same. Assume that at the source, external tables are stored in oss://oss-cn-hangzhou.aliyuncs.com/ali-test/documents/example_table. In the path, oss-cn-hangzhou.aliyuncs.com is the endpoint of the region where the involved bucket resides, ali-test is the bucket name, documents is the root directory of the bucket, and example_table stores metadata of external tables. We call it table directory below. When you configure a mapped target path, the root directory and the table directory must remain unchanged, for example, obs://hw-test/documents/example_table. Example: Source: oss://oss-cn-hangzhou.aliyuncs.com/ali-test/documents/example_table Target: obs://hw-test/documents/example_table NOTE: The preceding path mapping is only an example.

6. After the configuration is complete, execute the task.
   NOTICE:

   • A migration task can be executed repeatedly. Each time a migration task is executed, a task execution is generated.
   • You can click the task name to modify the task configuration.
   • You can select Run immediately and click Save to create the task and execute it immediately. You can view the created task on the Tasks page.

     

   • You can also click Save to just create the task. You can view the created task on the Tasks page. To execute the task, click Execute in the Operation column.

     

7. After the migration task is executed, click View Executions in the Operation column. On the Task Executions tab, you can view the details of the running task execution and all historical executions.

   

   Click View in the Progress column. On the displayed page, view and export the task execution results.