Help Center/ Data Lake Insight/ API Reference/ Spark Job-related APIs/ Creating a Batch Processing Job
Updated on 2026-07-16 GMT+08:00
Online Debugging
CLI Examples
View PDF
Share

Creating a Batch Processing Job

Function

This API is used to create a batch processing job in a queue.

During the Spark job submission process, if the job fails to acquire resources successfully for an extended period, the job status will change to dead after waiting for approximately 3 hours. For details about Spark job statuses, see Table 7.

Authorization

Each account has all the permissions required to call all APIs, but IAM users must be assigned the required permissions.

  • If you are using role/policy-based authorization, see the required permissions in Introduction.
  • If you are using identity policy-based authorization, the following identity policy-based permissions are required.

    Action

    Access Level

    Resource Type (*: required)

    Condition Key

    Alias

    Dependency

    dli:queue:submitJob

    write

    queue *

    -

    dli:catalog:get: Access to external catalogs (when using an external catalog).

    iam:agencies:pass: Use agency permissions (when an agency is configured).

URI

  • URI format

    POST /v2.0/{project_id}/batches

  • Parameter descriptions
    Table 1 URI parameter

    Parameter

    Mandatory

    Type

    Description

    project_id

    Yes

    String

    Definition

    Project ID, which is used for resource isolation. For how to obtain a project ID, see Obtaining a Project ID.

    Example: 48cc2c48765f481480c7db940d6409d1

    Constraints

    N/A

    Range

    The value can contain up to 64 characters. Only letters and digits are allowed.

    Default Value

    N/A

Request Parameters

Table 2 Request parameters

Parameter

Mandatory

Type

Description

file

Yes

String

Definition

Name of the package that is of the JAR or pyFile type and has been uploaded to the DLI resource management system.

You can also specify an OBS path, for example, obs://Bucket name/Package name.

Constraints

Spark 3.3.x or later supports only packages in OBS paths.

Range

N/A

Default Value

N/A

className

Yes

String

Definition

Java/Spark main class of the batch processing job

Constraints

N/A

Range

N/A

Default Value

N/A

queue

No

String

Definition

Queue name. Set this parameter to the name of the created DLI queue. The queue must be of the general-purpose type.

Constraints

  • This parameter is compatible with the cluster_name parameter. That is, if cluster_name is used to specify a queue, the queue is still valid.
  • The queue parameter is recommended. If queue and cluster_name are both set, the value of queue is used.

Range

N/A

Default Value

N/A

cluster_name

No

String

Definition

Queue name. Set this parameter to the created DLI queue name.

Constraints

You are advised to use the queue parameter. The queue and cluster_name parameters cannot coexist.

Range

N/A

Default Value

N/A

args

No

Array of strings

Definition

Input parameters of the main class, that is, application parameters.

Constraints

N/A

Range

N/A

Default Value

N/A

sc_type

No

String

Definition

Compute resource type. Currently, resource types A, B, and C are available. If this parameter is not specified, the minimum configuration (type A) is used. For details about resource types, see Table 3.

Constraints

N/A

Range

N/A

Default Value

N/A

jars

No

Array of strings

Definition

Name of the package that is of the JAR type and has been uploaded to the DLI resource management system. You can also specify an OBS path, for example, obs://Bucket name/Package name.

Constraints

N/A

Range

N/A

Default Value

N/A

pyFiles

No

Array of strings

Definition

Name of the package that is of the PyFile type and has been uploaded to the DLI resource management system. You can also specify an OBS path, for example, obs://Bucket name/Package name.

Constraints

N/A

Range

N/A

Default Value

N/A

files

No

Array of strings

Definition

Name of the package that is of the file type and has been uploaded to the DLI resource management system. You can also specify an OBS path, for example, obs://Bucket name/Package name.

Constraints

N/A

Range

N/A

Default Value

N/A

modules

No

Array of strings

Definition

Name of the dependency system resource module. You can check the module name using the Querying Resource Packages in a Group (Deprecated) API.

Constraints

N/A

Range

DLI provides dependencies for executing datasource jobs. The following table lists the dependency modules corresponding to different services.
  • CloudTable/MRS HBase: sys.datasource.hbase
  • CloudTable/MRS OpenTSDB: sys.datasource.opentsdb
  • RDS MySQL: sys.datasource.rds
  • RDS Postgre: preset
  • DWS: preset
  • CSS: sys.datasource.css

Default Value

N/A

resources

No

Array of objects

Definition

JSON object list, including the name and type of the JSON package that has been uploaded to the queue. For details, see Table 4.

Constraints

Spark 3.3.x or later does not support this parameter. Configure resource package information in jars, pyFiles, and files.

Range

N/A

Default Value

N/A

groups

No

Array of objects

Definition

JSON object list, including the package group resource. For details about the format, see the request example. If the type of the name in resources is not verified, the package with the name exists in the group. For details, see Table 5.

Constraints

Spark 3.3.x or later does not support group information configuration.

Range

N/A

Default Value

N/A

conf

No

Object

Definition

Batch configuration item. For details, see Spark Configuration.

Constraints

N/A

Range

N/A

Default Value

N/A

name

No

String

Definition

Batch processing task name. The value contains a maximum of 128 characters.

Constraints

N/A

Range

N/A

Default Value

N/A

driverMemory

No

String

Definition

Driver memory of the Spark application, for example, 2 GB and 2048 MB. This configuration will replace the default settings in sc_type. When using it, you must include the unit, otherwise it will fail to start.

Constraints

N/A

Range

N/A

Default Value

N/A

driverCores

No

Integer

Definition

Number of CPU cores of the Spark application driver. This configuration item replaces the default parameter in sc_type.

Constraints

N/A

Range

N/A

Default Value

N/A

executorMemory

No

String

Definition

Executor memory of the Spark application, for example, 2 GB and 2048 MB. This configuration will replace the default settings in sc_type. When using it, you must include the unit, otherwise it will fail to start.

Constraints

N/A

Range

N/A

Default Value

N/A

executorCores

No

Integer

Definition

Number of CPU cores of each Executor in the Spark application. This configuration item replaces the default parameter in sc_type.

Constraints

N/A

Range

N/A

Default Value

N/A

numExecutors

No

Integer

Definition

Number of Executors in a Spark application. This configuration item replaces the default parameter in sc_type.

Constraints

N/A

Range

N/A

Default Value

N/A

obs_bucket

No

String

Definition

OBS bucket for storing the Spark jobs. Set this parameter when you need to save jobs.

Constraints

N/A

Range

N/A

Default Value

N/A

auto_recovery

No

Boolean

Definition

Whether to enable the retry function. If enabled, Spark jobs will be automatically retried after an exception occurs. The default value is false.

Constraints

N/A

Range

N/A

Default Value

false

max_retry_times

No

Integer

Definition

Maximum retry times. The maximum value is 100, and the default value is 20.

Constraints

N/A

Range

N/A

Default Value

20

feature

No

String

Definition

Job feature. Type of the Spark image used by a job.

Constraints

N/A

Range

  • basic: indicates that the basic Spark image provided by DLI is used.
  • custom: indicates that the user-defined Spark image is used.

Default Value

N/A

spark_version

No

String

Definition

Version of the Spark component

Constraints

N/A

Range

  • If the in-use Spark version is 2.3.2, this parameter is not required.

Default Value

N/A

execution_agency_urn

No

String

Definition

Name of the agency authorized to DLI. This parameter is configurable in Spark 3.3.1.

Constraints

N/A

Range

N/A

Default Value

N/A

image

No

String

Definition

Custom image. The format is Organization name/Image name:Image version.

Constraints

This parameter is valid only when feature is set to custom. You can use this parameter with the feature parameter to specify a user-defined Spark image for job running.

For details about how to use custom images, see Data Lake Insight User Guide.

Range

N/A

Default Value

N/A

catalog_name

No

String

Definition

To access metadata, set this parameter to dli.

Constraints

N/A

Range

N/A

Default Value

N/A
Table 3 Resource types

Resource Type

Physical Resource

driverCores

executorCores

driverMemory

executorMemory

numExecutor

A

8 vCPUs, 32 GB memory

2

1

7 GB

4 GB

6

B

16 vCPUs, 64 GB memory

2

2

7 GB

8 GB

7

C

32 vCPUs, 128 GB memory

4

2

15 GB

8 GB

14
Table 4 resources parameters

Parameter

Mandatory

Type

Description

name

No

String

Definition

Resource name You can also specify an OBS path, for example, obs://Bucket name/Package name.

Constraints

N/A

Range

N/A

Default Value

N/A

type

No

String

Definition

Resource type.

Constraints

N/A

Range

N/A

Default Value

N/A
Table 5 groups parameters

Parameter

Mandatory

Type

Description

name

No

String

Definition

User group name

Constraints

N/A

Range

N/A

Default Value

N/A

resources

No

Array of objects

Definition

User group resource For details, see Table 4.

Constraints

N/A

Range

N/A

Default Value

N/A

Response Parameters

Table 6 Response parameters

Parameter

Mandatory

Type

Description

id

No

String

Definition

ID of a batch processing job.

Range

N/A

appId

No

String

Definition

Back-end application ID of a batch processing job.

Range

N/A

name

No

String

Definition

Batch processing task name. The value contains a maximum of 128 characters.

Range

N/A

owner

No

String

Definition

Owner of a batch processing job.

Range

N/A

proxyUser

No

String

Definition

Proxy user (resource tenant) to which a batch processing job belongs.

Range

N/A

state

No

String

Definition

Status of a batch processing job. For details, see Table 7.

Range

N/A

kind

No

String

Definition

Type of a batch processing job. Only Spark parameters are supported.

Range

N/A

log

No

Array of strings

Definition

Last 10 records of the current batch processing job.

Range

N/A

sc_type

No

String

Definition

Compute resource type. Currently, the value can be A, B, or C. If the compute resource type is customized, value CUSTOMIZED is returned.

Range

  • A: physical resources, with 8 vCPUs and 32 GB of memory
    • driverCores: 2;
    • executorCores: 1;
    • driverMemory: 7G;
    • executorMemory: 4G;
    • numExecutor: 6.
  • B: physical resources, with 16 vCPUs and 64 GB of memory
    • driverCores: 2;
    • executorCores: 2;
    • driverMemory: 7G;
    • executorMemory: 8G;
    • numExecutor: 7.
  • C: physical resources, with 32 vCPUs and 128 GB of memory
    • driverCores: 4;
    • executorCores: 2;
    • driverMemory: 15G;
    • executorMemory: 8G;
    • numExecutor: 14.

cluster_name

No

String

Definition

Queue where a batch processing job is located.

Range

N/A

queue

Yes

String

Definition

Queue name. Set this parameter to the name of the created DLI queue.

  • This parameter is compatible with the cluster_name parameter. That is, if cluster_name is used to specify a queue, the queue is still valid.
  • You are advised to use the queue parameter. The queue and cluster_name parameters cannot coexist.

Range

N/A

image

No

String

Definition

Custom image. The format is Organization name/Image name:Image version.

Range

This parameter is valid only when feature is set to custom. You can use this parameter with the feature parameter to specify a user-defined Spark image for job running.

For details about how to use custom images, see Data Lake Insight User Guide.

create_time

No

Long

Definition

Time when a batch processing job is created. The timestamp is in milliseconds.

Range

N/A

update_time

No

Long

Definition

Time when a batch processing job is updated. The timestamp is in milliseconds.

Range

N/A

duration

No

Long

Definition

Job running duration, in milliseconds.

Range

N/A
Table 7 Batch processing job statuses

Parameter

Type

Description

starting

String

The batch processing job is being started.

running

String

The batch processing job is executing a task.

dead

String

The batch processing job has exited.

success

String

The batch processing job is successfully executed.

recovering

String

The batch processing job is being restored.

Example Request

Create a Spark job. Set the Spark main class of the job to org.apache.spark.examples.SparkPi, specify the program package to batchTest/spark-examples_2.11-2.1.0.luxor.jar, and load the program package whose type is jar and the resource package whose type is files.

{
    "file": "batchTest/spark-examples_2.11-2.1.0.luxor.jar",
    "className": "org.apache.spark.examples.SparkPi",
    "sc_type": "A",
    "jars": ["demo-1.0.0.jar"],
    "files": ["count.txt"],
    "resources":[
                   {"name": "groupTest/testJar.jar", "type": "jar"},
                   {"name": "kafka-clients-0.10.0.0.jar", "type": "jar"}],
    "groups": [
                   {"name": "groupTestJar", "resources": [{"name": "testJar.jar", "type": "jar"}, {"name": "testJar1.jar", "type": "jar"}]}, 
                   {"name": "batchTest", "resources":  [{"name": "luxor.jar", "type": "jar"}]}],
    "queue": "test",
    "name": "TestDemo4",
    "feature": "basic",
    "execution_agency_urn": "myAgencyName",
    "spark_version": "2.3.2"
}

The batchTest/spark-examples_2.11-2.1.0.luxor.jar file has been uploaded through API involved in Uploading a Package Group (Deprecated).

Example Response

{
  "id": "07a3e4e6-9a28-4e92-8d3f-9c538621a166",
  "appId": "",
  "name": "",
  "owner": "test1",
  "proxyUser": "",
  "state": "starting",
  "kind": "",
  "log": [],
  "sc_type": "CUSTOMIZED",
  "cluster_name": "aaa",
  "queue": "aaa",
  "create_time": 1607589874156,
  "update_time": 1607589874156
}

Status Codes

Table 8 describes status codes.

Table 8 Status codes

Status Code

Description

200

The job is created successfully.

400

Request error.

500

Internal server error.

Error Codes

If an error occurs when this API is called, the system does not return the result similar to the preceding example, but returns an error code and error message. For details, see Error Codes.

Parent topic: Spark Job-related APIs