Help Center/ Cloud Data Migration/ API Reference/ API/ Job Management/ Creating and Executing a Job in a Random Cluster

Updated on 2022-09-15 GMT+08:00

View PDF

Creating and Executing a Job in a Random Cluster

Function

This API is used to create and execute a job in a random cluster.

URI

POST /v1.1/{project_id}/clusters/job

**Table 1** Path Parameters
Parameter	Mandatory	Type	Description
project_id	Yes	String	Project ID

Request Parameters

**Table 2** Request header parameters
Parameter	Mandatory	Type	Description
X-Auth-Token	Yes	String	User token. It can be obtained by calling the IAM API (value of X-Subject-Token in the response header).
X-Language	Yes	String	Request language

**Table 3** Request body parameters
Parameter	Mandatory	Type	Description
jobs	Yes	Array of Job objects	Job list. For details, see the descriptions of jobs parameters.
clusters	Yes	Array of strings	IDs of CDM clusters. The system selects a random cluster in running state from the specified clusters and creates and executes a migration job in the cluster.

**Table 4** Job
Parameter	Mandatory	Type	Description
job_type	No	String	Job type NORMAL_JOB: table/file migration BATCH_JOB: entire DB migration SCENARIO_JOB: scenario migration Enumeration values: NORMAL_JOB BATCH_JOB SCENARIO_JOB
from-connector-name	Yes	String	Source link type
to-config-values	Yes	ConfigValues object	Destination link parameter configuration
to-link-name	Yes	String	Destination link name
driver-config-values	Yes	ConfigValues object	Job parameter configuration
from-config-values	Yes	ConfigValues object	Source link parameter configuration
to-connector-name	No	String	Destination link type
name	No	String	Job name, which contains 1 to 240 characters Minimum: 1 Maximum: 240
from-link-name	No	String	Source link name
creation-user	No	String	User who created the job
creation-date	No	Long	Time when the job was created, accurate to millisecond
update-date	No	Long	Time when the job was last updated, accurate to millisecond
is_incre_job	No	Boolean	Incremental or not
flag	No	Integer	Flag
files_read	No	Integer	Number of read files
update-user	No	String	User who last updated the job
external_id	No	String	External ID
type	No	String	Task type
execute_start_date	No	Long	Execution start date
delete_rows	No	Integer	Number of deleted rows
enabled	No	Boolean	Whether to activate the link
bytes_written	No	Long	Number of written bytes
id	No	Integer	Job ID
is_use_sql	No	Boolean	Whether to use SQL statements
update_rows	No	Integer	Number of updated rows
group_name	No	String	Group name
bytes_read	No	Long	Number of read bytes
execute_update_date	No	Long	Execution update date
write_rows	No	Integer	Number of written rows
rows_written	No	Integer	Number of written rows
rows_read	No	Long	Number of rows that are read
files_written	No	Integer	Number of written files
is_incrementing	No	Boolean	Incremental or not
execute_create_date	No	Long	Execution creation date
status	No	String	Job execution status BOOTING: The job is starting. RUNNING: The job is running. SUCCEEDED: The job was successfully executed. FAILED: The job execution failed. NEW: The job was not executed.

**Table 5** ConfigValues
Parameter	Mandatory	Type	Description
configs	Yes	Array of configs objects	The data structures of source link parameters, destination link parameters, and job parameters are the same. However, the inputs parameter varies. For details, see the descriptions of configs parameters.
extended-configs	No	extended-configs object	Extended configuration. For details, see the descriptions of extended-configs parameters.

**Table 6** extended-configs
Parameter	Mandatory	Type	Description
name	No	String	Name
value	No	String	Value

**Table 7** extended-configs
Parameter	Mandatory	Type	Description
name	No	String	Name
value	No	String	Value

**Table 8** configs
Parameter	Mandatory	Type	Description
inputs	Yes	Array of Input objects	Input parameter list. Each element in the list is in name,value format. For details, see the descriptions of inputs parameters. In the from-config-values data structure, the value of this parameter varies with the source link type. For details, see section "Source Job Parameters" in the Cloud Data Migration User Guide. In the to-cofig-values data structure, the value of this parameter varies with the destination link type. For details, see section "Destination Job Parameters" in the Cloud Data Migration User Guide. For details about the inputs parameter in the driver-config-values data structure, see the job parameter descriptions.
name	Yes	String	Configuration name. The value is fromJobConfig for a source job, toJobConfig for a destination job, and linkConfig for a link.
id	No	Integer	Configuration ID
type	No	String	Configuration type

**Table 9** Input
Parameter	Mandatory	Type	Description
name	Yes	String	Parameter name
value	Yes	String	Parameter value
type	No	String	Value type

**Table 10** extended-configs
Parameter	Mandatory	Type	Description
name	No	String	Name
value	No	String	Value

Response Parameters

Status code: 200

**Table 11** Response body parameters
Parameter	Type	Description
submissions	Array of StartJobSubmission objects	Job running information. For details, see the descriptions of submission parameters.

**Table 12** StartJobSubmission
Parameter	Type	Description
isIncrementing	Boolean	Whether the job migrates incremental data
delete_rows	Integer	Number of deleted rows
update_rows	Integer	Number of updated rows
write_rows	Integer	Number of written rows
submission-id	Integer	ID of the submitted job
job-name	String	Job name
creation-user	String	User who created the job
creation-date	Long	Job creation time, accurate to millisecond
execute-date	Long	Job execution time
progress	Float	Job progress. If a job fails, the value is -1. Otherwise, the value ranges from 0 to 100.
status	String	Job status BOOTING: The job is starting. FAILURE_ON_SUBMIT: The job failed to be submitted. RUNNING: The job is running. SUCCEEDED: The job was successfully executed. FAILED: The job execution failed. UNKNOWN: The job status is unknown. NEVER_EXECUTED: The job was not executed.
isStopingIncrement	String	Whether to stop incremental data migration
is-execute-auto	Boolean	Whether to execute the job as scheduled
last-update-date	Long	Time when the job was last updated
last-udpate-user	String	User who last updated the job status
isDeleteJob	Boolean	Whether to delete the job after it is executed

Example Requests

POST /v1.1/1551c7f6c808414d8e9f3c514a170f2e/clusters/job

{
  "jobs" : [ {
    "job_type" : "NORMAL_JOB",
    "from-connector-name" : "elasticsearch-connector",
    "to-config-values" : {
      "configs" : [ {
        "inputs" : [ {
          "name" : "toJobConfig.streamName",
          "value" : "dis-lkGm"
        }, {
          "name" : "toJobConfig.separator",
          "value" : "|"
        }, {
          "name" : "toJobConfig.columnList",
          "value" : "1&2&3"
        } ],
        "name" : "toJobConfig"
      } ]
    },
    "to-link-name" : "dis",
    "driver-config-values" : {
      "configs" : [ {
        "inputs" : [ {
          "name" : "throttlingConfig.numExtractors",
          "value" : "1"
        }, {
          "name" : "throttlingConfig.submitToCluster",
          "value" : "false"
        }, {
          "name" : "throttlingConfig.numLoaders",
          "value" : "1"
        }, {
          "name" : "throttlingConfig.recordDirtyData",
          "value" : "false"
        } ],
        "name" : "throttlingConfig"
      }, {
        "inputs" : { },
        "name" : "jarConfig"
      }, {
        "inputs" : [ {
          "name" : "schedulerConfig.isSchedulerJob",
          "value" : "false"
        }, {
          "name" : "schedulerConfig.disposableType",
          "value" : "NONE"
        } ],
        "name" : "schedulerConfig"
      }, {
        "inputs" : { },
        "name" : "transformConfig"
      }, {
        "inputs" : [ {
          "name" : "retryJobConfig.retryJobType",
          "value" : "NONE"
        } ],
        "name" : "retryJobConfig"
      } ]
    },
    "from-config-values" : {
      "configs" : [ {
        "inputs" : [ {
          "name" : "fromJobConfig.index",
          "value" : "52est"
        }, {
          "name" : "fromJobConfig.type",
          "value" : "est_array"
        }, {
          "name" : "fromJobConfig.columnList",
          "value" : "array_f1_int:long&array_f2_text:string&array_f3_object:nested"
        }, {
          "name" : "fromJobConfig.splitNestedField",
          "value" : "false"
        } ],
        "name" : "fromJobConfig"
      } ]
    },
    "to-connector-name" : "dis-connector",
    "name" : "es_css",
    "from-link-name" : "css"
  } ],
  "clusters" : [ "b0791496-e111-4e75-b7ca-9277aeab9297", "c2db1191-eb6c-464a-a0d3-b434e6c6df26", "c2db1191-eb6c-464a-a0d3-b434e6c6df26" ]
}

Example Responses

Status code: 200

{
  "submissions" : [ {
    "isIncrementing" : false,
    "job-name" : "obs2obs-03",
    "submisson-id" : 13,
    "isStopingIncrement" : "",
    "last-update-date" : 1635909057030,
    "is-execute-auto" : false,
    "delete_rows" : 0,
    "write_rows" : 0,
    "last-update-user" : "mwx5316849",
    "isDeleteJob" : false,
    "creation-user" : "mwx5316849",
    "progress" : 0,
    "creation-date" : 1635909057030,
    "update_rows" : 0,
    "status" : "PENDING"
  } ]
}