Help Center/ Migration Center/ User Guide/ Big Data Lineage/ Collecting Metadata/ Creating a Metadata Collection Task
Updated on 2024-12-20 GMT+08:00
View PDF
Share

Creating a Metadata Collection Task

Use the created metadata connection to create a collection task that gathers the details about source databases, tables, and fields to MgC.

You can create metadata collection tasks for:

  • Hive Metastore
  • Doris
  • HBase
  • ClickHouse
  • Hudi (with metadata)
  • Hudi (without metadata)
  • MaxCompute
  • ApsaraDB for ClickHouse

You can also add metadata about tables manually or by importing a file. For details, see Viewing Metadata.

Prerequisites

You have created a metadata connection.

Procedure

  1. Sign in to the MgC console.
  2. In the navigation pane on the left, choose Research > Data Lineage. Select a migration project in the upper left corner of the page.
  3. In the Metadata Collection area, choose Create Task > Metadata Collection.

  4. Set task parameters based on Table 1.

    Table 1 Parameters for configuring a metadata collection task

    Parameter

    Configuration

    Task Name

    The default name is Metadata-Collection-4 random characters (including letters and numbers). You can also specify a name.

    Metadata Connection

    Select the created metadata connection.

    Databases

    Enter the names of the databases whose metadata needs to be collected.

    NOTICE:

    This parameter is mandatory only if a MaxCompute metadata connection is selected. In other cases, it is optional. If it is left blank, the metadata of all databases is collected.

    Concurrent Threads

    Set the number of concurrent threads for executing the collection. The default value is 3. The value ranges from 1 to 10. Configuring Configuring more concurrent threads means more efficient collection, but more connection and Edge device resources will be consumed.

  5. Click Confirm. The metadata collection task is created.
  6. Click Collection tasks. Under Tasks, you can review the created metadata collection task and its settings. You can modify the task by choosing More > Modify in the Operation column.

  7. Click Execute Task in the Operation column to run the task. Each time the task is executed, a task execution is generated.
  8. Click View Executions in the Operation column. Under Task Executions, you can view the execution records of the task and the status and collection result of each task execution. When a task execution enters a Completed status and the collection results are displayed, you can view the list of databases and tables extracted from collected metadata on the Tables tab.

  9. Locate a table and click Collect in the Lineage column to create a lineage collection task.

Parent topic: Collecting Metadata