Adding and Executing a Job

Function

This API is used to add and submit a job in an MRS cluster.

• On the Dashboard tab page of the cluster details page, click Click to synchronize on the right side of IAM User Sync to synchronize IAM users. Then submit a job through this API.

URI

• Format

  POST /v2/{project_id}/clusters/{cluster_id}/job-executions

• Parameter description

  Table 1 URI parameter description

  Parameter	Mandatory	Description
  project_id	Yes	Project ID. For details on how to obtain the project ID, see Obtaining a Project ID.
  cluster_id	Yes	Cluster ID. For details on how to obtain the cluster ID, see Obtaining a Cluster ID.

Request

Response

Example

You must have prepared the OBS paths, sample files, endpoints, and AKs/SKs when submitting a request.

• Example request

  The following is an example of a MapReduce job request:

  {
      "job_name":"MapReduceTest",
      "job_type":"MapReduce",
      "arguments":[
          "obs://obs-test/program/hadoop-mapreduce-examples-x.x.x.jar",
          "wordcount",
          "obs://obs-test/input/",
          "obs://obs-test/job/mapreduce/output"
      ],
      "properties":{
          "fs.obs.endpoint":"obs endpoint",
          "fs.obs.access.key":"xxx",
          "fs.obs.secret.key":"yyy"
      }
  }

  The following is an example of a SparkSubmit job request:

  {
      "job_name":"SparkSubmitTest",
      "job_type":"SparkSubmit",
      "arguments":[
          "--master",
          "yarn",
          "--deploy-mode",
          "cluster",
          "--py-files",
          "obs://obs-test/a.py",
          "--conf",
          "spark.yarn.appMasterEnv.PYTHONPATH=/tmp:$PYTHONPATH",
          "--conf",
          "spark.yarn.appMasterEnv.aaa=aaaa",
          "--conf",
          "spark.executorEnv.aaa=executoraaa",
          "--properties-file",
          "obs://obs-test/test-spark.conf",
          "obs://obs-test/pi.py",
          "100000"
      ],
      "properties":{
          "fs.obs.access.key":"xxx",
          "fs.obs.secret.key":"yyy"
      }
  }

  The following is an example of a HiveScript job request:

  {
      "job_name":"HiveScriptTest",
      "job_type":"HiveScript",
      "arguments":[
          "obs://obs-test/sql/test_script.sql"
      ],
      "properties":{
          "fs.obs.endpoint":"obs endpoint",
          "fs.obs.access.key":"xxx",
          "fs.obs.secret.key":"yyy"
      }
  }

  The following is an example of a HiveSQL job request:

  {
      "job_name":"HiveSqlTest",
      "job_type":"HiveSql",
      "arguments": [
  		"DROP TABLE IF EXISTS src_wordcount;\ncreate external table src_wordcount(line string) row format delimited fields terminated by \"\\n\" stored as textfile location \"obs://donotdel-gxc/input/\";\ninsert into src_wordcount values(\"v1\");"
  	],
      "properties":{
          "fs.obs.endpoint":"obs endpoint",
          "fs.obs.access.key":"xxx",
          "fs.obs.secret.key":"yyy"
      }
  }

  The following is an example of a DistCp job request:

  {
      "job_name":"DistCpTest",
      "job_type":"DistCp",
      "arguments":[
          "obs://obs-test/DistcpJob/",
          "/user/test/sparksql/"
      ],
      "properties":{
          "fs.obs.endpoint":"obs endpoint",
          "fs.obs.access.key":"xxx",
          "fs.obs.secret.key":"yyy"
      }
  }

  The following is an example of a SparkScript job request:

  {
      "job_name":"SparkScriptTest",
      "job_type":"SparkScript",
      "arguments":[
          "op-key1",
          "op-value1",
          "op-key2",
          "op-value2",
          "obs://obs-test/sql/test_script.sql"
      ],
      "properties":{
          "fs.obs.access.key":"xxx",
          "fs.obs.secret.key":"yyy"
      }
  } 

  The following is an example of a SparkSQL job request:

  {
      "job_name":"SparkSqlTest",
      "job_type":"SparkSql",
      "arguments":[
          "op-key1",
          "op-value1",
          "op-key2",
          "op-value2",
          "create table student_info3 (id string,name string,gender string,age int,addr string);"
      ],
      "properties":{
          "fs.obs.access.key":"xxx",
          "fs.obs.secret.key":"yyy"
      }
  } 

  The following is an example of a Flink job request:

  {
      "job_name":"FlinkTest",
      "job_type":"Flink",
      "arguments":[
          "run",
          "-d",
          "-ynm",
          "testExcutorejobhdfsbatch",
          "-m",
          "yarn-cluster",
          "hdfs://test/examples/batch/WordCount.jar"
      ],
      "properties":{
          "fs.obs.endpoint":"obs endpoint",
          "fs.obs.access.key":"xxx",
          "fs.obs.secret.key":"yyy"
      }
  }

  The following shows an example request of a SparkPython job (Jobs of this type will be converted to SparkSubmit jobs for submission. The job type is displayed as SparkSubmit on the MRS console. Select SparkSubmit when you call an API to query the job list.)

  POST https://{endpoint}/v2/{project_id}/clusters/{cluster_id}/job-executions
  
  {
    "job_name" : "SparkPythonTest",
    "job_type" : "SparkPython",
    "arguments" : [ "--master", "yarn", "--deploy-mode", "cluster", "--py-files", "obs://obs-test/a.py", "--conf", "spark.yarn.appMasterEnv.PYTHONPATH=/tmp:$PYTHONPATH", "--conf", "spark.yarn.appMasterEnv.aaa=aaaa", "--conf", "spark.executorEnv.aaa=executoraaa", "--properties-file", "obs://obs-test/test-spark.conf", "obs://obs-test/pi.py", "100000" ],
    "properties" : {
      "fs.obs.access.key" : "xxx",
      "fs.obs.secret.key" : "yyy"
    }
  }

• Example response
  • Example of a successful response

    {
      "job_submit_result":{
          "job_id":"44b37a20-ffe8-42b1-b42b-78a5978d7e40",
          "state":"COMPLETE"
      }
    }

  • Example of a failed response

    {
    "error_msg": Hive jobs cannot be submitted.
    "error_code":"0168"
    }

Status Code

For details about status codes, see Status Codes.