Help Center/ DataArts Studio/ FAQs/ DataArts Migration/ How Do I Set the Number of Concurrent Extractors for a CDM Migration Job?
Updated on 2024-04-29 GMT+08:00
View PDF
Share

How Do I Set the Number of Concurrent Extractors for a CDM Migration Job?

CDM migrates data through data migration jobs. It works in the following way:
  1. When data migration jobs are submitted, CDM splits each job into multiple tasks based on the Concurrent Extractors parameter in the job configuration.

    Jobs for different data sources may be split based on different dimensions. Some jobs may not be split based on the Concurrent Extractors parameter.

  2. CDM submits the tasks to the running pool in sequence. Tasks (defined by Maximum Concurrent Extractors) run concurrently. Excess tasks are queued.

Changing Concurrent Extractors

  1. The maximum number of concurrent extractors for a cluster varies depending on the CDM cluster flavor. You are advised to set the maximum number of concurrent extractors to twice the number of vCPUs of the CDM cluster.
    Table 1 Maximum number of concurrent extractors for a CDM cluster

    Flavor

    vCPUs/Memory

    Maximum Concurrent Extractors

    cdm.large

    8 vCPUs, 16 GB

    16

    cdm.xlarge

    16 vCPUs, 32 GB

    32

    cdm.4xlarge

    64 vCPUs, 128 GB

    128
    Figure 1 Setting Maximum Concurrent Extractors for a CDM cluster
  2. Configure the number of concurrent extractors based on the following rules:
    1. When data is to be migrated to files, CDM does not support multiple concurrent tasks. In this case, set a single process to extract data.
    2. If each row of the table contains less than or equal to 1 MB data, data can be extracted concurrently. If each row contains more than 1 MB data, it is recommended that data be extracted in a single thread.
    3. Set Concurrent Extractors for a job based on Maximum Concurrent Extractors for the cluster. It is recommended that Concurrent Extractors is less than Maximum Concurrent Extractors.
    4. If the destination is DLI, you are advised to set the number of concurrent extractors to 1. Otherwise, data may fail to be written.
    Figure 2 Setting Concurrent Extractors for a job
Parent topic: DataArts Migration