Help Center/ Cloud Data Migration/ API Reference/ API/ Job Management/ Creating and Executing a Job in a Random Cluster

Updated on 2024-06-06 GMT+08:00

Online Debugging

CLI Examples

View PDF

Creating and Executing a Job in a Random Cluster

Function

This API is used to create and execute a job in a random cluster.

Calling Method

For details, see Calling APIs.

URI

POST /v1.1/{project_id}/clusters/job

**Table 1** Path Parameters
Parameter	Mandatory	Type	Description
project_id	Yes	String	Project ID. For details about how to obtain the project ID, see Project ID and Account ID.

Request Parameters

**Table 2** Request header parameters
Parameter	Mandatory	Type	Description
X-Auth-Token	Yes	String	User token. It can be obtained by calling the IAM API (value of X-Subject-Token in the response header).
X-Language	Yes	String	Request language

**Table 3** Request body parameters
Parameter	Mandatory	Type	Description
jobs	Yes	Array of Job objects	Job list. For details, see the descriptions of jobs parameters.
clusters	Yes	Array of strings	IDs of CDM clusters. The system selects a random cluster in running state from the specified clusters and creates and executes a migration job in the cluster.

**Table 4** Job
Parameter	Mandatory	Type	Description
job_type	Yes	String	Job type NORMAL_JOB: table/file migration BATCH_JOB: entire DB migration SCENARIO_JOB: scenario migration Enumeration values: NORMAL_JOB BATCH_JOB SCENARIO_JOB
from-connector-name	Yes	String	Source link type. The available values are as follows: generic-jdbc-connector: link to a relational database obs-connector: link to OBS hdfs-connector: link to HDFS hbase-connector: link to HBase or CloudTable hive-connector: link to Hive ftp-connector/sftp-connector: link to an FTP or SFTP server mongodb-connector: link to MongoDB redis-connector: link to Redis or DCS kafka-connector: link to Kafka dis-connector: link to DIS elasticsearch-connector: link to Elasticsearch or Cloud Search Service (CSS) dli-connector: link to DLI. http-connector: link to an HTTP or HTTPS server (No link parameters are required.) dms-kafka-connector: link to DMS for Kafka
to-config-values	Yes	ConfigValues object	Destination link parameters, which vary depending on the destination. For details, see Destination Job Parameters.
to-link-name	Yes	String	Name of the destination link, that is, the name of the link created through the API used to create a link
driver-config-values	Yes	ConfigValues object	Job parameters, such as Retry upon Failure and Concurrent Extractors. For details, see Job Parameter Description.
from-config-values	Yes	ConfigValues object	Source link parameters, which vary depending on the source. For details, see Source Job Parameters.
to-connector-name	Yes	String	Destination link type. The available values are as follows: generic-jdbc-connector: link to a relational database obs-connector: link to OBS hdfs-connector: link to HDFS hbase-connector: link to HBase or CloudTable hive-connector: link to Hive ftp-connector/sftp-connector: link to an FTP or SFTP server mongodb-connector: link to MongoDB redis-connector: link to Redis or DCS kafka-connector: link to Kafka dis-connector: link to DIS elasticsearch-connector: link to Elasticsearch or Cloud Search Service (CSS) dli-connector: link to DLI. http-connector: link to an HTTP or HTTPS server (No link parameters are required.) dms-kafka-connector: link to DMS for Kafka
name	Yes	String	Job name, which contains 1 to 240 characters Minimum: 1 Maximum: 240
from-link-name	Yes	String	Name of the source link, that is, the name of the link created through the API used to create a link
creation-user	No	String	User who created the job. The value is generated by the system.
creation-date	No	Long	Time when the job was created, accurate to millisecond. The value is generated by the system.
update-date	No	Long	Time when the job was last updated, accurate to millisecond. The value is generated by the system.
is_incre_job	No	Boolean	Whether the job is an incremental job. This parameter is deprecated.
flag	No	Integer	Whether the job is a scheduled job. If yes, the value is 1. Otherwise, the value is 0. The value is generated by the system based on the scheduled task configuration.
files_read	No	Integer	Number of read files. The value is generated by the system.
update-user	No	String	User who last updated the job. The value is generated by the system.
external_id	No	String	ID of the job to be executed. For a local job, the value is in the format of job_local1202051771_0002 . For a DLI job, the value is the DLI job ID, for example, "12345". The value is generated by the system and does not need to be set.
type	No	String	Job type. The value of this parameter is the same as that of job_type. The options are as follows: NORMAL_JOB: table/file migration job BATCH_JOB: entire DB migration job SCENARIO_JOB: scenario migration job
execute_start_date	No	Long	Time when the last task was started, accurate to millisecond. The value is generated by the system.
delete_rows	No	Integer	Number of rows deleted by an incremental job. This parameter is deprecated.
enabled	No	Boolean	Whether the link is enabled. The value is generated by the system.
bytes_written	No	Long	Number of bytes written by the job. The value is generated by the system.
id	No	Integer	Job ID, which is generated by the system
is_use_sql	No	Boolean	Whether SQL statements are used. The value is generated by the system based on whether SQL statements are used at the source.
update_rows	No	Integer	Number of updated rows in an incremental job. This parameter is deprecated.
group_name	No	String	Group name
bytes_read	No	Long	Number of bytes read by the job. The value is generated by the system.
execute_update_date	No	Long	Time when the last task was updated, accurate to millisecond. The value is generated by the system.
write_rows	No	Integer	Number of rows written by an incremental job. This parameter is deprecated.
rows_written	No	Integer	Number of rows written by the job. The value is generated by the system.
rows_read	No	Long	Number of rows read by the job. The value is generated by the system.
files_written	No	Integer	Number of written files. The value is generated by the system.
is_incrementing	No	Boolean	Whether the job is an incremental job. Similar to parameter is_incre_job, this parameter is deprecated.
execute_create_date	No	Long	Time when the last task was created, accurate to millisecond. The value is generated by the system.
status	No	String	Job execution status BOOTING: The job is starting. RUNNING: The job is running. SUCCEEDED: The job was successfully executed. FAILED: The job execution failed. NEW: The job was not executed.

**Table 5** ConfigValues
Parameter	Mandatory	Type	Description
configs	Yes	Array of configs objects	The data structures of source link parameters, destination link parameters, and job parameters are the same. However, the inputs parameter varies. For details, see the descriptions of configs parameters.
extended-configs	No	extended-configs object	Extended configuration. For details, see the descriptions of extended-configs parameters. The extended configuration is not open to external systems. You do not need to set it.

**Table 6** configs
Parameter	Mandatory	Type	Description
inputs	Yes	Array of Input objects	Input parameter list. Each element in the list is in name,value format. For details, see the descriptions of inputs parameters. In the from-config-values data structure, the value of this parameter varies with the source link type. For details, see section "Source Job Parameters" in the Cloud Data Migration User Guide. In the to-config-values data structure, the value of this parameter varies with the destination link type. For details, see section "Destination Job Parameters" in the Cloud Data Migration User Guide. For details about the inputs parameter in the driver-config-values data structure, see the job parameter descriptions.
name	Yes	String	Configuration name. The value is fromJobConfig for a source job, toJobConfig for a destination job, and linkConfig for a link.
id	No	Integer	Configuration ID, which is generated by the system. You do not need to set this parameter.
type	No	String	Configuration type, which is generated by the system. You do not need to set this parameter. The value can be LINK (for link management APIs) or JOB (for job management APIs).

**Table 7** Input
Parameter	Mandatory	Type	Description
name	Yes	String	Parameter name. For link management APIs, parameter names start with linkConfig.. The parameters vary depending on the link type. For details, see the parameter descriptions of the corresponding link in Link Parameters. For job management APIs, source link parameter names start with fromJobConfig.. For details, see the source job parameters in Source Job Parameters. Destination link parameter names start with toJobConfig.. For details, see Destination Job Parameters. For details about job parameters, see the task parameter descriptions in Job Parameters.
value	Yes	String	Parameter value, which must be a string.
type	No	String	Value type, such as STRING and INTEGER. The value is set by the system.

**Table 8** extended-configs
Parameter	Mandatory	Type	Description
name	No	String	Extended configuration name. This parameter is unavailable for external systems and does not need to be set.
value	No	String	Extended configuration value. This parameter is unavailable for external systems and does not need to be set.

Response Parameters

Status code: 200

**Table 9** Response body parameters
Parameter	Type	Description
submissions	Array of StartJobSubmission objects	Job running information. For details, see the descriptions of submission parameters.

**Table 10** StartJobSubmission
Parameter	Type	Description
isIncrementing	Boolean	Whether the job migrates incremental data
delete_rows	Integer	Number of deleted rows
update_rows	Integer	Number of updated rows
write_rows	Integer	Number of written rows
submission-id	Integer	ID of the submitted job
job-name	String	Job name
creation-user	String	User who created the job
creation-date	Long	Job creation time, accurate to millisecond
execute-date	Long	Job execution time
progress	Float	Job progress. If a job fails, the value is -1. Otherwise, the value ranges from 0 to 100.
status	String	Job status BOOTING: The job is starting. FAILURE_ON_SUBMIT: The job failed to be submitted. RUNNING: The job is running. SUCCEEDED: The job was successfully executed. FAILED: The job execution failed. UNKNOWN: The job status is unknown. NEVER_EXECUTED: The job was not executed.
isStopingIncrement	String	Whether to stop incremental data migration
is-execute-auto	Boolean	Whether to execute the job as scheduled
last-update-date	Long	Time when the job was last updated
last-udpate-user	String	User who last updated the job status
isDeleteJob	Boolean	Whether to delete the job after it is executed

Example Requests

Randomly selecting a CDM cluster and creating a job named es_css to migrate tables from Elasticsearch to DIS

POST /v1.1/1551c7f6c808414d8e9f3c514a170f2e/clusters/job

{
  "jobs" : [ {
    "job_type" : "NORMAL_JOB",
    "from-connector-name" : "elasticsearch-connector",
    "to-config-values" : {
      "configs" : [ {
        "inputs" : [ {
          "name" : "toJobConfig.streamName",
          "value" : "dis-lkGm"
        }, {
          "name" : "toJobConfig.separator",
          "value" : "|"
        }, {
          "name" : "toJobConfig.columnList",
          "value" : "1&2&3"
        } ],
        "name" : "toJobConfig"
      } ]
    },
    "to-link-name" : "dis",
    "driver-config-values" : {
      "configs" : [ {
        "inputs" : [ {
          "name" : "throttlingConfig.numExtractors",
          "value" : "1"
        }, {
          "name" : "throttlingConfig.submitToCluster",
          "value" : "false"
        }, {
          "name" : "throttlingConfig.numLoaders",
          "value" : "1"
        }, {
          "name" : "throttlingConfig.recordDirtyData",
          "value" : "false"
        } ],
        "name" : "throttlingConfig"
      }, {
        "inputs" : [ ],
        "name" : "jarConfig"
      }, {
        "inputs" : [ {
          "name" : "schedulerConfig.isSchedulerJob",
          "value" : "false"
        }, {
          "name" : "schedulerConfig.disposableType",
          "value" : "NONE"
        } ],
        "name" : "schedulerConfig"
      }, {
        "inputs" : [ ],
        "name" : "transformConfig"
      }, {
        "inputs" : [ {
          "name" : "retryJobConfig.retryJobType",
          "value" : "NONE"
        } ],
        "name" : "retryJobConfig"
      } ]
    },
    "from-config-values" : {
      "configs" : [ {
        "inputs" : [ {
          "name" : "fromJobConfig.index",
          "value" : "52est"
        }, {
          "name" : "fromJobConfig.type",
          "value" : "est_array"
        }, {
          "name" : "fromJobConfig.columnList",
          "value" : "array_f1_int:long&array_f2_text:string&array_f3_object:nested"
        }, {
          "name" : "fromJobConfig.splitNestedField",
          "value" : "false"
        } ],
        "name" : "fromJobConfig"
      } ]
    },
    "to-connector-name" : "dis-connector",
    "name" : "es_css",
    "from-link-name" : "css"
  } ],
  "clusters" : [ "b0791496-e111-4e75-b7ca-9277aeab9297", "c2db1191-eb6c-464a-a0d3-b434e6c6df26", "c2db1191-eb6c-464a-a0d3-b434e6c6df26" ]
}

Example Responses

Status code: 200

{
  "submissions" : [ {
    "isIncrementing" : false,
    "job-name" : "obs2obs-03",
    "submission-id" : 13,
    "isStopingIncrement" : "",
    "last-update-date" : 1635909057030,
    "is-execute-auto" : false,
    "delete_rows" : 0,
    "write_rows" : 0,
    "isDeleteJob" : false,
    "creation-user" : "cdmUser",
    "progress" : 0,
    "creation-date" : 1635909057030,
    "update_rows" : 0,
    "status" : "PENDING",
    "execute-date" : 1635909057030
  } ]
}