Help Center> DataArts Studio> User Guide> DataArts Catalog> Tutorials> Developing an Incremental Metadata Collection Task
Updated on 2022-08-05 GMT+08:00
View PDF

Developing an Incremental Metadata Collection Task

Configuring and running a collection task is the prerequisite for building data assets. This section describes how to create different types of metadata collection tasks.

Scenario 1: Adding Metadata Only

Create a collection task to collect new tables only.

For example, if table4 is newly added:
  • Data table metadata before collection: table1, table2, table3
  • Data table metadata after collection: table1, table2, table3, table4

If you only want to collect table 4 in the preceding figure, perform the following steps (on condition that table 1, 2, and 3 are already in DataArts Catalog):

  1. Access the DataArts Catalog module on the DataArts Studio console.
  2. In the navigation pane on the left, choose Collection Tasks.
  3. Click Create.
  4. Configure parameters for the task.

  5. Click Next and set scheduling parameters.

  6. Click Submit to create a collection task.
  7. In the task list, locate the created task and click Run or Start Schedule in the Operation column to go to the Task Monitoring page and view the task status.

Scenario 2: Updating Existing Metadata and Adding New Metadata

Create a collection task to collect all tables, including existing and new ones.

For example, if table4 is newly added:
  • Data table metadata before collection: table1, table2, table3
  • Data table metadata after collection: table1, table2, table3, table4

If you want to collect all tables in the preceding figure, perform the following steps:

  1. Access the DataArts Catalog module on the DataArts Studio console.
  2. In the navigation pane on the left, choose Collection Tasks.
  3. Click Create.
  4. Configure parameters for the task.

  5. Click Next and set scheduling parameters.

  6. Click Submit to create a collection task.
  7. In the task list, locate the created task and click Run or Start Schedule in the Operation column to go to the Task Monitoring page and view the task status.

Scenario 3: Updating Existing Metadata Only

Create a collection task to collect existing tables.

For example, if table4 is newly added:
  • Data table metadata before collection: table1, table2, table3
  • Data table metadata after collection: table1, table2, table3

If you want to collect table 1, 2, and 3 in the preceding figure, perform the following steps:

  1. Access the DataArts Catalog module on the DataArts Studio console.
  2. In the navigation pane on the left, choose Collection Tasks.
  3. Click Create.
  4. Configure parameters for the task.

  5. Click Next and set scheduling parameters.

  6. Click Submit to create a collection task.
  7. In the task list, locate the created task and click Run or Start Schedule in the Operation column to go to the Task Monitoring page and view the task status.

Scenario 4: Updating and Deleting Existing Metadata and Adding New Metadata

Create a collection task to delete existing tables.

For example, if table1 is deleted from the database:
  • Data table metadata before collection: table1, table2, table3
  • Data table metadata after collection: table2, table3

If you want to delete table1, perform the following steps:

  1. Access the DataArts Catalog module on the DataArts Studio console.
  2. In the navigation pane on the left, choose Collection Tasks.
  3. Click Create.
  4. Configure parameters for the task.

  5. Click Next and set scheduling parameters.

  6. Click Submit to create a collection task.
  7. In the task list, locate the created task and click Run or Start Schedule in the Operation column to go to the Task Monitoring page and view the task status.
Parent topic: Tutorials