Help Center/ Cloud Data Migration/ API Reference/ API/ Job Management/ Creating and Executing a Job in a Random Cluster
Updated on 2024-06-06 GMT+08:00

Creating and Executing a Job in a Random Cluster

Function

This API is used to create and execute a job in a random cluster.

Calling Method

For details, see Calling APIs.

URI

POST /v1.1/{project_id}/clusters/job

Table 1 Path Parameters

Parameter

Mandatory

Type

Description

project_id

Yes

String

Project ID. For details about how to obtain the project ID, see Project ID and Account ID.

Request Parameters

Table 2 Request header parameters

Parameter

Mandatory

Type

Description

X-Auth-Token

Yes

String

User token. It can be obtained by calling the IAM API (value of X-Subject-Token in the response header).

X-Language

Yes

String

Request language

Table 3 Request body parameters

Parameter

Mandatory

Type

Description

jobs

Yes

Array of Job objects

Job list. For details, see the descriptions of jobs parameters.

clusters

Yes

Array of strings

IDs of CDM clusters. The system selects a random cluster in running state from the specified clusters and creates and executes a migration job in the cluster.

Table 4 Job

Parameter

Mandatory

Type

Description

job_type

Yes

String

Job type

  • NORMAL_JOB: table/file migration

  • BATCH_JOB: entire DB migration

  • SCENARIO_JOB: scenario migration

Enumeration values:

  • NORMAL_JOB

  • BATCH_JOB

  • SCENARIO_JOB

from-connector-name

Yes

String

Source link type. The available values are as follows: generic-jdbc-connector: link to a relational database obs-connector: link to OBS hdfs-connector: link to HDFS hbase-connector: link to HBase or CloudTable hive-connector: link to Hive ftp-connector/sftp-connector: link to an FTP or SFTP server mongodb-connector: link to MongoDB redis-connector: link to Redis or DCS kafka-connector: link to Kafka dis-connector: link to DIS elasticsearch-connector: link to Elasticsearch or Cloud Search Service (CSS) dli-connector: link to DLI. http-connector: link to an HTTP or HTTPS server (No link parameters are required.) dms-kafka-connector: link to DMS for Kafka

to-config-values

Yes

ConfigValues object

Destination link parameters, which vary depending on the destination. For details, see Destination Job Parameters.

to-link-name

Yes

String

Name of the destination link, that is, the name of the link created through the API used to create a link

driver-config-values

Yes

ConfigValues object

Job parameters, such as Retry upon Failure and Concurrent Extractors. For details, see Job Parameter Description.

from-config-values

Yes

ConfigValues object

Source link parameters, which vary depending on the source. For details, see Source Job Parameters.

to-connector-name

Yes

String

Destination link type. The available values are as follows: generic-jdbc-connector: link to a relational database obs-connector: link to OBS hdfs-connector: link to HDFS hbase-connector: link to HBase or CloudTable hive-connector: link to Hive ftp-connector/sftp-connector: link to an FTP or SFTP server mongodb-connector: link to MongoDB redis-connector: link to Redis or DCS kafka-connector: link to Kafka dis-connector: link to DIS elasticsearch-connector: link to Elasticsearch or Cloud Search Service (CSS) dli-connector: link to DLI. http-connector: link to an HTTP or HTTPS server (No link parameters are required.) dms-kafka-connector: link to DMS for Kafka

name

Yes

String

Job name, which contains 1 to 240 characters

Minimum: 1

Maximum: 240

from-link-name

Yes

String

Name of the source link, that is, the name of the link created through the API used to create a link

creation-user

No

String

User who created the job. The value is generated by the system.

creation-date

No

Long

Time when the job was created, accurate to millisecond. The value is generated by the system.

update-date

No

Long

Time when the job was last updated, accurate to millisecond. The value is generated by the system.

is_incre_job

No

Boolean

Whether the job is an incremental job. This parameter is deprecated.

flag

No

Integer

Whether the job is a scheduled job. If yes, the value is 1. Otherwise, the value is 0. The value is generated by the system based on the scheduled task configuration.

files_read

No

Integer

Number of read files. The value is generated by the system.

update-user

No

String

User who last updated the job. The value is generated by the system.

external_id

No

String

ID of the job to be executed. For a local job, the value is in the format of job_local1202051771_0002 . For a DLI job, the value is the DLI job ID, for example, **"12345"**. The value is generated by the system and does not need to be set.

type

No

String

Job type. The value of this parameter is the same as that of job_type. The options are as follows:

  • NORMAL_JOB: table/file migration job

  • BATCH_JOB: entire DB migration job

  • SCENARIO_JOB: scenario migration job

execute_start_date

No

Long

Time when the last task was started, accurate to millisecond. The value is generated by the system.

delete_rows

No

Integer

Number of rows deleted by an incremental job. This parameter is deprecated.

enabled

No

Boolean

Whether the link is enabled. The value is generated by the system.

bytes_written

No

Long

Number of bytes written by the job. The value is generated by the system.

id

No

Integer

Job ID, which is generated by the system

is_use_sql

No

Boolean

Whether SQL statements are used. The value is generated by the system based on whether SQL statements are used at the source.

update_rows

No

Integer

Number of updated rows in an incremental job. This parameter is deprecated.

group_name

No

String

Group name

bytes_read

No

Long

Number of bytes read by the job. The value is generated by the system.

execute_update_date

No

Long

Time when the last task was updated, accurate to millisecond. The value is generated by the system.

write_rows

No

Integer

Number of rows written by an incremental job. This parameter is deprecated.

rows_written

No

Integer

Number of rows written by the job. The value is generated by the system.

rows_read

No

Long

Number of rows read by the job. The value is generated by the system.

files_written

No

Integer

Number of written files. The value is generated by the system.

is_incrementing

No

Boolean

Whether the job is an incremental job. Similar to parameter is_incre_job, this parameter is deprecated.

execute_create_date

No

Long

Time when the last task was created, accurate to millisecond. The value is generated by the system.

status

No

String

Job execution status

  • BOOTING: The job is starting.

  • RUNNING: The job is running.

  • SUCCEEDED: The job was successfully executed.

  • FAILED: The job execution failed.

  • NEW: The job was not executed.

Table 5 ConfigValues

Parameter

Mandatory

Type

Description

configs

Yes

Array of configs objects

The data structures of source link parameters, destination link parameters, and job parameters are the same. However, the inputs parameter varies. For details, see the descriptions of configs parameters.

extended-configs

No

extended-configs object

Extended configuration. For details, see the descriptions of extended-configs parameters. The extended configuration is not open to external systems. You do not need to set it.

Table 6 configs

Parameter

Mandatory

Type

Description

inputs

Yes

Array of Input objects

Input parameter list. Each element in the list is in name,value format. For details, see the descriptions of inputs parameters. In the from-config-values data structure, the value of this parameter varies with the source link type. For details, see section "Source Job Parameters" in the Cloud Data Migration User Guide. In the to-config-values data structure, the value of this parameter varies with the destination link type. For details, see section "Destination Job Parameters" in the Cloud Data Migration User Guide. For details about the inputs parameter in the driver-config-values data structure, see the job parameter descriptions.

name

Yes

String

Configuration name. The value is fromJobConfig for a source job, toJobConfig for a destination job, and linkConfig for a link.

id

No

Integer

Configuration ID, which is generated by the system. You do not need to set this parameter.

type

No

String

Configuration type, which is generated by the system. You do not need to set this parameter. The value can be LINK (for link management APIs) or JOB (for job management APIs).

Table 7 Input

Parameter

Mandatory

Type

Description

name

Yes

String

Parameter name.

  • For link management APIs, parameter names start with linkConfig.. The parameters vary depending on the link type. For details, see the parameter descriptions of the corresponding link in Link Parameters.

  • For job management APIs, source link parameter names start with fromJobConfig.. For details, see the source job parameters in Source Job Parameters. Destination link parameter names start with toJobConfig.. For details, see Destination Job Parameters. For details about job parameters, see the task parameter descriptions in Job Parameters.

value

Yes

String

Parameter value, which must be a string.

type

No

String

Value type, such as STRING and INTEGER. The value is set by the system.

Table 8 extended-configs

Parameter

Mandatory

Type

Description

name

No

String

Extended configuration name. This parameter is unavailable for external systems and does not need to be set.

value

No

String

Extended configuration value. This parameter is unavailable for external systems and does not need to be set.

Response Parameters

Status code: 200

Table 9 Response body parameters

Parameter

Type

Description

submissions

Array of StartJobSubmission objects

Job running information. For details, see the descriptions of submission parameters.

Table 10 StartJobSubmission

Parameter

Type

Description

isIncrementing

Boolean

Whether the job migrates incremental data

delete_rows

Integer

Number of deleted rows

update_rows

Integer

Number of updated rows

write_rows

Integer

Number of written rows

submission-id

Integer

ID of the submitted job

job-name

String

Job name

creation-user

String

User who created the job

creation-date

Long

Job creation time, accurate to millisecond

execute-date

Long

Job execution time

progress

Float

Job progress. If a job fails, the value is -1. Otherwise, the value ranges from 0 to 100.

status

String

Job status

  • BOOTING: The job is starting.

  • FAILURE_ON_SUBMIT: The job failed to be submitted.

  • RUNNING: The job is running.

  • SUCCEEDED: The job was successfully executed.

  • FAILED: The job execution failed.

  • UNKNOWN: The job status is unknown.

  • NEVER_EXECUTED: The job was not executed.

isStopingIncrement

String

Whether to stop incremental data migration

is-execute-auto

Boolean

Whether to execute the job as scheduled

last-update-date

Long

Time when the job was last updated

last-udpate-user

String

User who last updated the job status

isDeleteJob

Boolean

Whether to delete the job after it is executed

Example Requests

Randomly selecting a CDM cluster and creating a job named es_css to migrate tables from Elasticsearch to DIS

POST /v1.1/1551c7f6c808414d8e9f3c514a170f2e/clusters/job

{
  "jobs" : [ {
    "job_type" : "NORMAL_JOB",
    "from-connector-name" : "elasticsearch-connector",
    "to-config-values" : {
      "configs" : [ {
        "inputs" : [ {
          "name" : "toJobConfig.streamName",
          "value" : "dis-lkGm"
        }, {
          "name" : "toJobConfig.separator",
          "value" : "|"
        }, {
          "name" : "toJobConfig.columnList",
          "value" : "1&2&3"
        } ],
        "name" : "toJobConfig"
      } ]
    },
    "to-link-name" : "dis",
    "driver-config-values" : {
      "configs" : [ {
        "inputs" : [ {
          "name" : "throttlingConfig.numExtractors",
          "value" : "1"
        }, {
          "name" : "throttlingConfig.submitToCluster",
          "value" : "false"
        }, {
          "name" : "throttlingConfig.numLoaders",
          "value" : "1"
        }, {
          "name" : "throttlingConfig.recordDirtyData",
          "value" : "false"
        } ],
        "name" : "throttlingConfig"
      }, {
        "inputs" : [ ],
        "name" : "jarConfig"
      }, {
        "inputs" : [ {
          "name" : "schedulerConfig.isSchedulerJob",
          "value" : "false"
        }, {
          "name" : "schedulerConfig.disposableType",
          "value" : "NONE"
        } ],
        "name" : "schedulerConfig"
      }, {
        "inputs" : [ ],
        "name" : "transformConfig"
      }, {
        "inputs" : [ {
          "name" : "retryJobConfig.retryJobType",
          "value" : "NONE"
        } ],
        "name" : "retryJobConfig"
      } ]
    },
    "from-config-values" : {
      "configs" : [ {
        "inputs" : [ {
          "name" : "fromJobConfig.index",
          "value" : "52est"
        }, {
          "name" : "fromJobConfig.type",
          "value" : "est_array"
        }, {
          "name" : "fromJobConfig.columnList",
          "value" : "array_f1_int:long&array_f2_text:string&array_f3_object:nested"
        }, {
          "name" : "fromJobConfig.splitNestedField",
          "value" : "false"
        } ],
        "name" : "fromJobConfig"
      } ]
    },
    "to-connector-name" : "dis-connector",
    "name" : "es_css",
    "from-link-name" : "css"
  } ],
  "clusters" : [ "b0791496-e111-4e75-b7ca-9277aeab9297", "c2db1191-eb6c-464a-a0d3-b434e6c6df26", "c2db1191-eb6c-464a-a0d3-b434e6c6df26" ]
}

Example Responses

Status code: 200

ok

{
  "submissions" : [ {
    "isIncrementing" : false,
    "job-name" : "obs2obs-03",
    "submission-id" : 13,
    "isStopingIncrement" : "",
    "last-update-date" : 1635909057030,
    "is-execute-auto" : false,
    "delete_rows" : 0,
    "write_rows" : 0,
    "isDeleteJob" : false,
    "creation-user" : "cdmUser",
    "progress" : 0,
    "creation-date" : 1635909057030,
    "update_rows" : 0,
    "status" : "PENDING",
    "execute-date" : 1635909057030
  } ]
}

Status Codes

Status Code

Description

200

ok

Error Codes

See Error Codes.