Adding and Executing a Job - CreateExecuteJob

Function

This API is used to add and submit a job in an MRS cluster.

• If you want to use the OBS encryption function, follow instructions in Using OBS to Encrypt Data for Running Jobs to configure related information and call an API to run a job.
• On the Dashboard tab page of the cluster details page, click Click to synchronize on the right side of IAM User Sync to synchronize IAM users. Then submit a job through this API.

Constraints

None

Debugging

You can debug this API in API Explorer. Automatic authentication is supported. API Explorer can automatically generate sample SDK code and supports sample SDK code debugging.

Authorization Information

Each account has all the permissions required to call all APIs, but IAM users must be assigned the required permissions.

• If you are using role/policy-based authorization, see Permissions Policies and Supported Actions for details on the required permissions.
• If you are using identity policy-based authorization, the following identity policy-based permissions are required.

  Action	Access Level	Resource Type (*: required)	Condition Key	Alias	Dependency
  mrs:cluster:createJob	Write	cluster *	g:ResourceTag/<tag-key> g:EnterpriseProjectId	mrs:job:submit	-

URI

Request Parameters

Response Parameters

Status code: 200

Status code: 400

Example Request

You must have prepared the OBS paths, sample files, endpoints, and AKs/SKs when submitting a request.

• Create a MapReduce job.

  POST https://{endpoint}/v2/{project_id}/clusters/{cluster_id}/job-executions
  
  {
      "job_name":"MapReduceTest",
      "job_type":"MapReduce",
      "arguments":[
          "obs://obs-test/program/hadoop-mapreduce-examples-x.x.x.jar",
          "wordcount",
          "obs://obs-test/input/",
          "obs://obs-test/job/mapreduce/output"
      ],
      "properties":{
          "fs.obs.endpoint":"obs endpoint",
          "fs.obs.access.key":"xxx",
          "fs.obs.secret.key":"yyy"
      }
  }

• Create a SparkSubmit job.

  POST https://{endpoint}/v2/{project_id}/clusters/{cluster_id}/job-executions
  
  {
      "job_name":"SparkSubmitTest",
      "job_type":"SparkSubmit",
      "arguments":[
          "--master",
          "yarn",
          "--deploy-mode",
          "cluster",
          "--py-files",
          "obs://obs-test/a.py",
          "--conf",
          "spark.yarn.appMasterEnv.PYTHONPATH=/tmp:$PYTHONPATH",
          "--conf",
          "spark.yarn.appMasterEnv.aaa=aaaa",
          "--conf",
          "spark.executorEnv.aaa=executoraaa",
          "--properties-file",
          "obs://obs-test/test-spark.conf",
          "obs://obs-test/pi.py",
          "100000"
      ],
      "properties":{
          "fs.obs.access.key":"xxx",
          "fs.obs.secret.key":"yyy"
      }
  }

• Create a HiveScript job.

  POST https://{endpoint}/v2/{project_id}/clusters/{cluster_id}/job-executions
  
  {
      "job_name":"HiveScriptTest",
      "job_type":"HiveScript",
      "arguments":[
          "obs://obs-test/sql/test_script.sql"
      ],
      "properties":{
          "fs.obs.endpoint":"obs endpoint",
          "fs.obs.access.key":"xxx",
          "fs.obs.secret.key":"yyy"
      }
  }

• Create a HiveSql job.

  POST https://{endpoint}/v2/{project_id}/clusters/{cluster_id}/job-executions
  
  {
    "job_name" : "HiveSqlTest",
    "job_type" : "HiveSql",
    "arguments" : [ "DROP TABLE IF EXISTS src_wordcount;\ncreate external table src_wordcount(line string) row format delimited fields terminated by \"\\n\" stored as textfile location \"obs://donotdel-gxc/input/\";\ninsert into src_wordcount values(\"v1\")" ],
    "properties" : {
      "fs.obs.endpoint" : "obs endpoint",
      "fs.obs.access.key" : "xxx",
      "fs.obs.secret.key" : "yyy"
    }
  }

• Create a DistCp job.

  POST https://{endpoint}/v2/{project_id}/clusters/{cluster_id}/job-executions
  
  {
      "job_name":"DistCpTest",
      "job_type":"DistCp",
      "arguments":[
          "obs://obs-test/DistcpJob/",
          "/user/test/sparksql/"
      ],
      "properties":{
          "fs.obs.endpoint":"obs endpoint",
          "fs.obs.access.key":"xxx",
          "fs.obs.secret.key":"yyy"
      }
  }

• Create a SparkScript job.

  POST https://{endpoint}/v2/{project_id}/clusters/{cluster_id}/job-executions
  
  {
      "job_name":"SparkScriptTest",
      "job_type":"SparkScript",
      "arguments":[
          "op-key1",
          "op-value1",
          "op-key2",
          "op-value2",
          "obs://obs-test/sql/test_script.sql"
      ],
      "properties":{
          "fs.obs.access.key":"xxx",
          "fs.obs.secret.key":"yyy"
      }
  } 

• Create a SparkSql job.

  POST https://{endpoint}/v2/{project_id}/clusters/{cluster_id}/job-executions
  
  {
      "job_name":"SparkSqlTest",
      "job_type":"SparkSql",
      "arguments":[
          "op-key1",
          "op-value1",
          "op-key2",
          "op-value2",
          "create table student_info3 (id string,name string,gender string,age int,addr string);"
      ],
      "properties":{
          "fs.obs.access.key":"xxx",
          "fs.obs.secret.key":"yyy"
      }
  } 

• Create a Flink job.

  POST https://{endpoint}/v2/{project_id}/clusters/{cluster_id}/job-executions
  
  {
      "job_name":"FlinkTest",
      "job_type":"Flink",
      "arguments":[
          "run",
          "-d",
          "-ynm",
          "testExcutorejobhdfsbatch",
          "-m",
          "yarn-cluster",
          "hdfs://test/examples/batch/WordCount.jar"
      ],
      "properties":{
          "fs.obs.endpoint":"obs endpoint",
          "fs.obs.access.key":"xxx",
          "fs.obs.secret.key":"yyy"
      }
  }

• Create a SparkPython job (Jobs of this type will be converted to SparkSubmit jobs for submission. The job type is displayed as SparkSubmit on the MRS console. Select SparkSubmit when you call an API to query the job list.)

  POST https://{endpoint}/v2/{project_id}/clusters/{cluster_id}/job-executions
  
  {
    "job_name" : "SparkPythonTest",
    "job_type" : "SparkPython",
    "arguments" : [ "--master", "yarn", "--deploy-mode", "cluster", "--py-files", "obs://obs-test/a.py", "--conf", "spark.yarn.appMasterEnv.PYTHONPATH=/tmp:$PYTHONPATH", "--conf", "spark.yarn.appMasterEnv.aaa=aaaa", "--conf", "spark.executorEnv.aaa=executoraaa", "--properties-file", "obs://obs-test/test-spark.conf", "obs://obs-test/pi.py", "100000" ],
    "properties" : {
      "fs.obs.access.key" : "xxx",
      "fs.obs.secret.key" : "yyy"
    }
  }

Example Response

Status code: 200

• Example of a successful response

  {
    "job_submit_result":{
        "job_id":"44b37a20-ffe8-42b1-b42b-78a5978d7e40",
        "state":"COMPLETE"
    }
  }

Status code: 400

• Example of a failed response

  {
  "error_msg": Hive jobs cannot be submitted.
  "error_code":"0168"
  }

Status Codes

See Status Codes.

Error Codes

See Error Codes.