Updated on 2022-12-14 GMT+08:00

Adding and Executing a Job

Function

This API is used to add and submit a job in an MRS cluster.

  • On the Dashboard tab page of the cluster details page, click Click to synchronize on the right side of IAM User Sync to synchronize IAM users. Then submit a job through this API.

URI

POST /v2/{project_id}/clusters/{cluster_id}/job-executions
Table 1 URI parameters

Parameter

Mandatory

Type

Description

project_id

Yes

String

Project ID. For details about how to obtain the project ID, see Obtaining a Project ID.

cluster_id

Yes

String

Cluster ID. For details on how to obtain the cluster ID, see Obtaining a Cluster ID.

Request Parameters

Table 2 Request parameters

Parameter

Mandatory

Type

Description

job_type

Yes

String

Type of a job.

  • MapReduce
  • SparkSubmit
  • SparkPython: Example request of a SparkPython job (Jobs of this type will be converted to SparkSubmit jobs for submission. The job type is displayed as SparkSubmit on the MRS console. Select SparkSubmit when you call an API to query the job list.)
  • HiveScript
  • HiveSql
  • DistCp, importing and exporting data
  • SparkScript
  • SparkSql
  • Flink
NOTE:

Spark, Hive, and Flink jobs can be added to only clusters that include Spark, Hive, and Flink components.

job_name

Yes

String

Job name. It contains 1 to 64 characters. Only letters, digits, hyphens (-), and underscores (_) are allowed.

NOTE:

Identical job names are allowed but not recommended.

arguments

No

Array of strings

Key parameter for program execution. The parameter is specified by the function of the user's program. MRS is only responsible for loading the parameter.

The value can contain a maximum of 150,000 characters. Special characters (;|&>'<$!\\) are not allowed. This parameter can be left blank.

NOTE:
  • If you enter a parameter with sensitive information (such as the login password), the parameter may be exposed in the job details display and log printing. Exercise caution when performing this operation.
  • For MRS 3.x or later, a file path on OBS can start with obs://. To use this format to submit HiveScript or HiveSQL jobs, choose Components > Hive > Service Configuration on the cluster details page. Set Basic to All, and search for core.site.customized.configs. Add the endpoint configuration item fs.obs.endpoint of OBS and enter the endpoint corresponding to OBS in Value. For details, see Endpoints.

properties

No

Map<String,String>

Program system parameter.

The parameter contains a maximum of 2,048 characters, excluding special characters such as ><|'`&!\, and can be left blank.

Response Parameters

Status code: 200

Table 3 Response body parameter

Parameter

Type

Description

job_submit_result

JobSubmitResult object

The job execution result. For details about the parameters, see Table 4.

Table 4 JobSubitResult parameters

Parameter

Type

Description

job_id

String

Job ID

state

String

Job submission status.

  • COMPLETE: The job is submitted.
  • JOBSTAT_SUBMIT_FAILED: Failed to submit the job.

Status code: 500

Table 5 Response body parameters

Parameter

Type

Description

error_code

String

The error code.

error_msg

String

The error message.

Example Request

You must have prepared the OBS paths, sample files, endpoints, and AKs/SKs when submitting a request.

  • Example request of a MapReduce job:
    POST https://{endpoint}/v2/{project_id}/clusters/{cluster_id}/job-executions
    
    {
        "job_name":"MapReduceTest",
        "job_type":"MapReduce",
        "arguments":[
            "obs://obs-test/program/hadoop-mapreduce-examples-x.x.x.jar",
            "wordcount",
            "obs://obs-test/input/",
            "obs://obs-test/job/mapreduce/output"
        ],
        "properties":{
            "fs.obs.endpoint":"obs endpoint",
            "fs.obs.access.key":"xxx",
            "fs.obs.secret.key":"yyy"
        }
    }
  • Example request of a SparkSubmit job:
    POST https://{endpoint}/v2/{project_id}/clusters/{cluster_id}/job-executions
    
    {
        "job_name":"SparkSubmitTest",
        "job_type":"SparkSubmit",
        "arguments":[
            "--master",
            "yarn",
            "--deploy-mode",
            "cluster",
            "--py-files",
            "obs://obs-test/a.py",
            "--conf",
            "spark.yarn.appMasterEnv.PYTHONPATH=/tmp:$PYTHONPATH",
            "--conf",
            "spark.yarn.appMasterEnv.aaa=aaaa",
            "--conf",
            "spark.executorEnv.aaa=executoraaa",
            "--properties-file",
            "obs://obs-test/test-spark.conf",
            "obs://obs-test/pi.py",
            "100000"
        ],
        "properties":{
            "fs.obs.access.key":"xxx",
            "fs.obs.secret.key":"yyy"
        }
    }
  • Example request of a HiveScript job:
    POST https://{endpoint}/v2/{project_id}/clusters/{cluster_id}/job-executions
    
    {
        "job_name":"HiveScriptTest",
        "job_type":"HiveScript",
        "arguments":[
            "obs://obs-test/sql/test_script.sql"
        ],
        "properties":{
            "fs.obs.endpoint":"obs endpoint",
            "fs.obs.access.key":"xxx",
            "fs.obs.secret.key":"yyy"
        }
    }
  • Example request of a HiveSql job:
    POST https://{endpoint}/v2/{project_id}/clusters/{cluster_id}/job-executions
    
    {
      "job_name" : "HiveSqlTest",
      "job_type" : "HiveSql",
      "arguments" : [ "DROP TABLE IF EXISTS src_wordcount;\ncreate external table src_wordcount(line string) row format delimited fields terminated by \"\\n\" stored as textfile location \"obs://donotdel-gxc/input/\";\ninsert into src_wordcount values(\"v1\")" ],
      "properties" : {
        "fs.obs.endpoint" : "obs endpoint",
        "fs.obs.access.key" : "xxx",
        "fs.obs.secret.key" : "yyy"
      }
    }
  • Example request of a DistCp job:
    POST https://{endpoint}/v2/{project_id}/clusters/{cluster_id}/job-executions
    
    {
        "job_name":"DistCpTest",
        "job_type":"DistCp",
        "arguments":[
            "obs://obs-test/DistcpJob/",
            "/user/test/sparksql/"
        ],
        "properties":{
            "fs.obs.endpoint":"obs endpoint",
            "fs.obs.access.key":"xxx",
            "fs.obs.secret.key":"yyy"
        }
    }
  • Example request of a SparkScript job:
    POST https://{endpoint}/v2/{project_id}/clusters/{cluster_id}/job-executions
    
    {
        "job_name":"SparkScriptTest",
        "job_type":"SparkScript",
        "arguments":[
            "op-key1",
            "op-value1",
            "op-key2",
            "op-value2",
            "obs://obs-test/sql/test_script.sql"
        ],
        "properties":{
            "fs.obs.access.key":"xxx",
            "fs.obs.secret.key":"yyy"
        }
    } 
  • Example request of a SparkSql job:
    POST https://{endpoint}/v2/{project_id}/clusters/{cluster_id}/job-executions
    
    {
        "job_name":"SparkSqlTest",
        "job_type":"SparkSql",
        "arguments":[
            "op-key1",
            "op-value1",
            "op-key2",
            "op-value2",
            "create table student_info3 (id string,name string,gender string,age int,addr string);"
        ],
        "properties":{
            "fs.obs.access.key":"xxx",
            "fs.obs.secret.key":"yyy"
        }
    } 
  • Example request of a Flink job:
    POST https://{endpoint}/v2/{project_id}/clusters/{cluster_id}/job-executions
    
    {
        "job_name":"FlinkTest",
        "job_type":"Flink",
        "arguments":[
            "run",
            "-d",
            "-ynm",
            "testExcutorejobhdfsbatch",
            "-m",
            "yarn-cluster",
            "hdfs://test/examples/batch/WordCount.jar"
        ],
        "properties":{
            "fs.obs.endpoint":"obs endpoint",
            "fs.obs.access.key":"xxx",
            "fs.obs.secret.key":"yyy"
        }
    }
  • The following shows an example request of a SparkPython job (Jobs of this type will be converted to SparkSubmit jobs for submission. The job type is displayed as SparkSubmit on the MRS console. Select SparkSubmit when you call an API to query the job list.)
    POST https://{endpoint}/v2/{project_id}/clusters/{cluster_id}/job-executions
    
    {
      "job_name" : "SparkPythonTest",
      "job_type" : "SparkPython",
      "arguments" : [ "--master", "yarn", "--deploy-mode", "cluster", "--py-files", "obs://obs-test/a.py", "--conf", "spark.yarn.appMasterEnv.PYTHONPATH=/tmp:$PYTHONPATH", "--conf", "spark.yarn.appMasterEnv.aaa=aaaa", "--conf", "spark.executorEnv.aaa=executoraaa", "--properties-file", "obs://obs-test/test-spark.conf", "obs://obs-test/pi.py", "100000" ],
      "properties" : {
        "fs.obs.access.key" : "xxx",
        "fs.obs.secret.key" : "yyy"
      }
    }

Example Response

Status code: 200

  • Example of a successful response
    {
      "job_submit_result":{
          "job_id":"44b37a20-ffe8-42b1-b42b-78a5978d7e40",
          "state":"COMPLETE"
      }
    }

Status code: 500

  • Example of a failed response
    {
    "error_msg": Hive jobs cannot be submitted.
    "error_code":"0168"
    }

Status Codes

See Status Codes.

Error Codes

See Error Codes.