Viewing Job Details

workspace

No

String

Workspace ID.

• If this parameter is not set, data in the default workspace is queried by default.
• To query data in other workspaces, this header must be carried.

name

Yes

String

Job name. The name contains a maximum of 128 characters, including only letters, numbers, hyphens (-), underscores (_), and periods (.). The job name must be unique.

nodes

Yes

List<Node>

Node definition. For details, see Table 3.

schedule

Yes

Schedule data structure

Scheduling configuration. For details, see Table 4.

params

No

List<Param>

Job parameter definition. For details, see Table 5.

directory

No

String

Path of a job in the directory tree. If the directory of the path does not exist during job creation, a directory is automatically created in the root directory /, for example, /dir/a/.

processType

Yes

String

Job type.

• REAL_TIME: real-time processing
• BATCH: batch processing

id

No

Long

Job ID This parameter is required for querying jobs.

createTime

No

Long

Job creation time

downstreamJobs

No

String

Name of the downstream job that depends on the current job

lastUpdateUser

No

String

User who last updated the job

logPath

No

String

OBS path for storing job run logs

basicConfig

No

BasicConfig data structure

Basic job information. For details, see Table 28.

description

No

String

Job description

cleanoverdueDays

No

integer

Maximum timeout duration of the job

cleanWaitingJob

No

String

Whether to clear waiting jobs

emptyRunningJob

No

String

Whether to enable dry run

version

No

String

Job version

name

Yes

String

Node name. The name contains a maximum of 128 characters, including only letters, numbers, hyphens (-), underscores (_), and periods (.). Names of the nodes in a job must be unique.

type

Yes

String

Node type. The options are as follows:

• Hive SQL: Runs Hive SQL scripts.
• Spark SQL: Runs Spark SQL scripts.
• DWS SQL: Runs DWS SQL scripts.
• DLI SQL: Runs DLI SQL scripts.
• Shell: Runs shell SQL scripts.
• CDM Job: Runs CDM jobs.
• DIS Transfer Task: Creates DIS dump tasks.
• CloudTableManager: Manages CloudTable tables, including creating and deleting tables.
• OBS Manager: Manages OBS paths, including creating and deleting paths.
• RestClient: Sends REST API requests.
• SMN: Sends short messages or emails.
• MRS Spark: Runs Spark jobs of MRS.
• MapReduce: Runs MapReduce jobs of MRS.
• MRSFlinkJob: Runs FlinkJob jobs of MRS.
• MRS HetuEngine: Runs HetuEngine jobs of MRS.
• DLI Spark: Runs Spark jobs of DLF.
• RDSSQL: Transfers SQL statements to RDS for execution.

location

Yes

Location data structure

Location of a node on the job canvas. For details, see Table 6.

preNodeName

No

List<String>

Name of the previous node on which the current node depends.

conditions

No

List<Condition>

Node execution condition. Whether the node is executed or not depends on the calculation result of the EL expression saved in the expression field of condition. For details, see Table 7.

properties

Yes

List<Property>

Node properties. For details, see Table 13.

Each type of node has its own property definition.

• Hive SQL: For details, see Table 14.
• Spark SQL: For details, see Table 15.
• DWS SQL: For details, see Table 16.
• DLI SQL: For details, see Table 17.
• Shell: For details, see Table 18.
• CDM Job: For details, see Table 19.
• DIS TransferTask: For details, see Table 20.
• CloudTableManager: For details, see Table 21.
• OBSManager: For details, see Table 22.
• RESTAPI: For details, see Table 23.
• SMN: For details, see Table 24.
• MRS Spark: For details, see Table 25.
• MapReduce: For details, see Table 26.
• DLI Spark: For details, see Table 27.
• MRS Flink: For details, see Table 29.
• MRS HetuEngine: For details, see Table 30.

pollingInterval

No

Int

Interval at which node running results are checked.

Unit: second; value range: 1 to 60

Default value: 10

execTimeOutRetry

No

String

Whether to retry a node upon timeout. The default value is false.

maxExecutionTime

No

Int

Maximum execution time of a node. If a node is not executed within the maximum execution time, the node is set to the failed state.

Unit: minute; value range: 5 to 1440

Default value: 60

retryTimes

No

Int

Number of the node retries. The value ranges from 0 to 5. 0 indicates no retry.

Default value: 0

retryInterval

No

Int

Interval at which a retry is performed upon a failure. The value ranges from 5 to 120.

Unit: second

Default value: 120

failPolicy

No

String

Node failure policy.

• FAIL: Terminate the execution of the current job.
• IGNORE: Continue to execute the next node.
• SUSPEND: Suspend the execution of the current job.
• FAIL_CHILD: Terminate the execution of the subsequent node.

  The default value is FAIL.

eventTrigger

No

Event data structure

Event trigger for the real-time job node. For details, see Table 10.

cronTrigger

No

Cron data structure

Cron trigger for the real-time job node. For details, see Table 8.

type

Yes

String

Scheduling type.

• EXECUTE_ONCE: The job runs immediately and runs only once.
• CRON: The job runs periodically.
• EVENT: The job is triggered by events.

cron

No

Data structure

When type is set to CRON, configure the scheduling frequency and start time. For details, see Table 9.

event

No

Data structure

When type is set to EVENT, configure information such as the event source. For details, see Table 10.

name

Yes

String

Parameter name. It contains only letters, digits, hyphens (-), and underscores (_), and can contain no more than 64 characters.

value

Yes

String

Value of the parameter. It cannot exceed 1,024 characters.

type

No

String

Type

• variable
• constants

  Default value: variable

x

Yes

Int

Position of the node on the horizontal axis of the job canvas.

y

Yes

Int

Position of the node on the vertical axis of the job canvas.

preNodeName

Yes

String

Name of the previous node on which the current node depends.

expression

Yes

String

EL expression. If the calculation result of the EL expression is true, this node is executed.

startTime

Yes

String

Scheduling start time in the format of yyyy-MM-dd'T'HH:mm:ssZ, which is an ISO 8601 time format. For example, 2018-10-22T23:59:59+08, which indicates that a job starts to be scheduled at 23:59:59 on October 22nd, 2018.

endTime

No

String

Scheduling end time in the format of yyyy-MM-dd'T'HH:mm:ssZ, which is an ISO 8601 time format. For example, 2018-10-22T23:59:59+08, which indicates that a job stops to be scheduled at 23:59:59 on October 22nd, 2018. If the end time is not set, the job will continuously be executed based on the scheduling period.

expression

Yes

String

Cron expression in the format of <second><minute><hour><day><month><week>. For details about the value input in each field, see Table 11.

expressionTimeZone

No

String

Time zone corresponding to the Cron expression, for example, GMT+8.

Default value: time zone where DataArts Studio is located

period

Yes

String

Job execution interval consisting of a time and time unit

Example: 1 hours, 1 days, 1 weeks, 1 months

The value must match the value of expression.

dependPrePeriod

No

Boolean

Indicates whether to depend on the execution result of the current job's dependent job in the previous scheduling period.

Default value: false

dependJob

No

DependJob data structure

Dependent job configuration. For details, see Table 12 DependJob data structure description.

concurrent

No

Integer

Number of concurrent executions allowed

startTime

Yes

String

Scheduling start time in the format of yyyy-MM-dd'T'HH:mm:ssZ, which is an ISO 8601 time format. For example, 2018-10-22T23:59:59+08, which indicates that a job starts to be scheduled at 23:59:59 on October 22nd, 2018.

endTime

No

String

Scheduling end time in the format of yyyy-MM-dd'T'HH:mm:ssZ, which is an ISO 8601 time format. For example, 2018-10-22T23:59:59+08, which indicates that a job stops to be scheduled at 23:59:59 on October 22nd, 2018. If the end time is not set, the job will continuously be executed based on the scheduling period.

expression

Yes

String

Cron expression in the format of <second><minute><hour><day><month><week>. For details about the value input in each field, see Table 11.

expressionTimeZone

No

String

Time zone corresponding to the Cron expression, for example, GMT+8.

Default value: time zone where DataArts Studio is located

dependPrePeriod

No

Boolean

Indicates whether to depend on the execution result of the current job's dependent job in the previous scheduling period.

Default value: false

dependJobs

No

DependJob data structure

Dependent job configuration. For details, see Table 12 DependJob data structure description.

eventType

Yes

String

Select the corresponding connection name and topic. When a new Kafka message is received, the job is triggered.

Set this parameter to KAFKA.

Event type. Currently, only newly reported data events from the DIS stream can be monitored. Each time a data record is reported, the job runs once.

This parameter is set to DIS.

Select the OBS path to be listened to. If new files exist in the path, scheduling is triggered. The path name can be referenced using variable Job.trigger.obsNewFiles. The prerequisite is that DIS notifications have been configured for the OBS path.

Set this parameter to OBS.

channel

Yes

String

DIS stream name.

Perform the following operations to obtain the stream name:

1. Log in to the management console.
2. Click Data Ingestion Service and select Stream Management from the left navigation pane.
3. The stream management page lists the existing streams.

failPolicy

No

String

Job failure policy.

• SUSPEND: Suspend the event.
• IGNORE: Ignore the failure and process with the next event.

Default value: SUSPEND

concurrent

No

int

Number of the concurrently scheduled jobs.

Value range: 1 to 128

Default value: 1

readPolicy

No

String

Access policy.

• LAST: Access data from the last location.
• NEW: Access data from a new location.

Default value: LAST

Second

0-59

, - * /

In the current version, only 0 is allowed.

Minute

0-59

, - * /

-

Hour

0-23

, - * /

-

Day

1-31

, - * ? / L W C

-

Month

1-12

, - * /

In the current version, only * is allowed.

Week

1-7

, - * ? / L C #

Starting from Sunday.

jobs

Yes

List<String>

A list of dependent jobs. Only the existing jobs can be depended on.

dependPeriod

No

String

Dependency period.

• SAME_PERIOD: To run a job or not depends on the execution result of its depended job in the current scheduling period.
• PRE_PERIOD: To run a job or not depends on the execution result of its depended job in the previous scheduling period.

Default value: SAME_PERIOD

dependFailPolicy

No

String

Dependency job failure policy.

• FAIL: Stop the job and set the job to the failed state.
• IGNORE: Continue to run the job.
• SUSPEND: Suspend the job.

Default value: FAIL

sameWorkSpaceJobs

No

DependWorkspaceJob data structure

List of dependent jobs in the current workspace. For details, see Table 31.

otherWorkSpaceJobs

No

DependWorkspaceJob data structure

List of dependent jobs in other workspaces. For details, see Table 31.

name

Yes

String

Property name

value

Yes

String

Property value

scriptName

Yes

String

Script name

database

No

String

Database name

Database in the MRS Hive. The default value is default.

connectionName

No

String

Name of a connection

scriptArgs

No

String

Script parameter in format of key and value. Multiple parameters are separated by newlines (\n), for example, key1=value1\nkey2=value2.

scriptName

Yes

String

Script name

database

No

String

Database name

Database in the MRS Spark SQL. The default value is default.

connectionName

No

String

Name of a connection

scriptArgs

No

String

Script parameter in format of key and value. Multiple parameters are separated by newlines (\n), for example, key1=value1\nkey2=value2.

scriptName

Yes

String

Script name

database

No

String

Database name

Database in DWS. The default value is postgres.

connectionName

No

String

Name of a connection

scriptArgs

No

String

Script parameter in format of key and value. Multiple parameters are separated by newlines (\n), for example, key1=value1\nkey2=value2.

scriptName

Yes

String

Script name

database

No

String

Database name

Database in DLI.

connectionName

No

String

Name of a connection

scriptArgs

No

String

Script parameter in format of key and value. Multiple parameters are separated by newlines (\n), for example, key1=value1\nkey2=value2.

scriptName

Yes

String

Script name

connectionName

Yes

String

Name of a connection

arguments

No

String

Shell script parameter.

clusterName

Yes

String

Cluster name

You can obtain the cluster name from the CDM cluster list on the DataArts Migration page of the DataArts Studio console.

jobName

Yes

String

Job name

To obtain the job name, access the DataArts Studio console, choose DataArts Migration, click a cluster name on the Cluster Management page, and click Job Management on the displayed page.

streamName

Yes

String

DIS stream name.

Perform the following operations to obtain the stream name:

1. Log in to the management console.
2. Click Data Ingestion Service and select Stream Management from the left navigation pane.
3. The stream management page lists the existing streams.

destinationType

Yes

String

Dump target

• CloudTable
• OBS

duplicatePolicy

Yes

String

Duplicate name policy.

• OVERWRITE
• IGNORE

configuration

Yes

Data structure

Dump configuration. For details, see the description of the obs_destination_descriptor and cloudtable_destination_descriptor parameters in .

namespace

No

String

namespaces

Default value: default

action

Yes

String

Action type

• CREATE_TABLE: Create a table.
• DELETE_TABLE: Delete a table.

table

No

String

Table name

columnFamily

No

String

Column family

action

Yes

String

Action type

• CREATE_PATH: Create an OBS path.
• DELETE_PATH: Delete an OBS path.

path

Yes

String

OBS path

url

Yes

String

URL address

URL of the cloud service.

method

Yes

String

HTTP method

• GET
• POST
• PUT
• DELETE

headers

No

String

HTTP message header in the format of <message header name>=<value>. Multiple message headers are separated by newlines.

body

No

String

Message body

topic

Yes

String

SMN topic URN.

Perform the following operations to obtain an SMN topic URN:

1. Log in to the management console.
2. Click Simple Message Notification and choose Topic Management > Topics from the list on the left.

You can obtain the SMN topic URN in the topic list.

subject

Yes

String

Message title, which is used as the subject of an email sent to a subscriber.

messageType

Yes

String

Message type

• NORMAL
• STRUCTURE
• TEMPLATE

message

Yes

String

Message to be sent.

clusterName

Yes

String

MRS cluster name.

Perform the following operations to obtain the MRS cluster name:

1. Log in to the management console.
2. Click MapReduce Service and choose Clusters > Active Clusters from the left navigation pane.

You can obtain the cluster name from the active clusters.

jobName

Yes

String

MRS job name.

The job name is user-defined.

resourcePath

Yes

String

OBS resource path of the custom Spark JAR package

parameters

Yes

String

Custom parameters of the Spark JAR package

You can specify parameters for a custom JAR package.

input

No

String

Input path

Input data path of the MRS Spark job. The path can be an HDFS or OBS path.

output

No

String

Output path

Output data path of the MRS Spark job. The path can be an HDFS or OBS path.

programParameter

No

String

Program parameter

Multiple key-value pairs are allowed and separated by vertical bars (|).

clusterName

Yes

String

MRS cluster name.

Perform the following operations to obtain the MRS cluster name:

1. Log in to the management console.
2. Click MapReduce Service and choose Clusters > Active Clusters from the left navigation pane.

You can obtain the cluster name from the active clusters.

jobName

Yes

String

MRS job name.

The job name is user-defined.

resourcePath

Yes

String

Resource path

parameters

Yes

String

Job parameters

input

Yes

String

Input path

Input data path of the MapReduce job. The path can be an HDFS or OBS path.

output

Yes

String

Output path

Output data path of the MapReduce job. The path can be an HDFS or OBS path.

clusterName

Yes

String

DLI queue name

Perform the following operations to obtain the DLI queue name:

1. Log in to the management console.
2. Click Data Lake Insight and then Queue Management.

You can obtain the queue name from the queue management list.

jobName

Yes

String

DLI job name.

Perform the following operations to obtain the job name:

1. Log in to the management console.
2. Click Data Lake Insight and then Spark Jobs.
3. Choose Job Management.

You can obtain the job name from the job management list.

resourceType

No

String

Type of the running resource of the DLI job . This parameter is optional.

1. OBS path: OBS

2. DLI package: DLIResources

jobClass

No

String

Main class name. When the application type is .jar, the main class name cannot be empty.

resourcePath

Yes

String

JAR package resource path.

jarArgs

No

String

Main-class entry parameter.

sparkConfig

No

String

Running parameter of the Spark job.

owner

No

String

Job owner. The length cannot exceed 128 characters.

isIgnoreWaiting

No

integer

Whether to exclude waiting time from the instance timeout duration. The value can be 0 or 1. 0 indicates that the waiting time is not excluded from the instance timeout duration, and 1 indicates that the waiting time is excluded from the instance timeout duration

priority

No

int

Job priority. The value ranges from 0 to 2. The default value is 0. 0 indicates a top priority, 1 indicates a medium priority, and 2 indicates a low priority.

executeUser

No

String

Job execution user. The value must be an existing username.

instanceTimeout

No

int

Instance timeout interval. The unit is minute. The value ranges from 5 to 1440. The default value is 60.

customFields

No

Map<String,String>

User-defined field. The length cannot exceed 2048 characters.

agency

No

String

Agency name

tags

No

List<String>

Job tags

clusterName

Yes

String

MRS cluster name.

Perform the following operations to obtain the MRS cluster name:

1. Log in to the management console.
2. Click MapReduce Service and choose Clusters > Active Clusters from the left navigation pane.

You can obtain the cluster name from the active clusters.

jobName

Yes

String

MRS job name.

The job name is user-defined.

flinkJobType

Yes

String

Flink job type, which can be FLink SQL or Flink JAR

flinkJobProcessType

Yes

String

Flink job processing mode, which can be batch or stream

scriptName

No

String

SQL script associated with the Flink SQL job

resourcePath

No

String

OBS resource path of the custom Flink JAR package

input

No

String

Input path

Input data path of the MRS Flink job. The path can be an HDFS or OBS path.

output

No

String

Output path

Output data path of the MRS Flink job. The path can be an HDFS or OBS path.

programParameter

No

String

Program parameter

Multiple key-value pairs are allowed and separated by vertical bars (|).

clusterName

Yes

String

MRS cluster name.

Perform the following operations to obtain the MRS cluster name:

1. Log in to the management console.
2. Click MapReduce Service and choose Clusters > Active Clusters from the left navigation pane.

You can obtain the cluster name from the active clusters.

jobName

Yes

String

MRS job name.

The job name is user-defined.

statementOrScript

Yes

String

Whether to use an SQL statement for the node or associate an SQL script with the node

scriptName

No

String

SQL script to be associated with the node

statement

No

String

Custom content of the SQL statement

Data Warehouse

Yes

String

Data connection required by HetuEngine

Schema

Yes

String

Name of the schema to be accessed through HetuEngine

Database

Yes

String

Name of the database to be accessed through HetuEngine

Queue

No

String

Name of the resource queue required by HetuEngine

workSpace

Yes

String

Workspace name

jobName

Yes

String

Job name

dependOnLastPeriod

No

Boolean

Whether to depend on the latest instance of an upstream job