Help Center/ MapReduce Service/ API Reference/ API V2/ Job Management APIs/ Adding and Executing a Job - CreateExecuteJob
Updated on 2026-09-15 GMT+08:00
View PDF
Share

Adding and Executing a Job - CreateExecuteJob

Function

This API is used to add and submit a job in an MRS cluster. It supports multiple job types, including MapReduce, Spark, Hive, and more.

Before using this API, you must synchronize IAM users by clicking Synchronize next to IAM User Sync on the Dashboard tab of the cluster details page. After synchronization, you can submit a job through this API.

If you need to use the OBS encryption feature, perform the relevant configuration as described in Using OBS to Encrypt Data for Running Jobs, and then call the API to run the job.

For all examples, prepare the OBS paths, sample files, endpoints, and AK/SK in advance, and replace them with actual values when submitting requests.

Constraints

None

Debugging

You can debug this API in API Explorer. Automatic authentication is supported. API Explorer can automatically generate sample SDK code and supports sample SDK code debugging.

Authorization Information

Each account has all the permissions required to call all APIs, but IAM users must be assigned the required permissions.

  • If you are using role/policy-based authorization, see Permissions Policies and Supported Actions for details on the required permissions.
  • If you are using identity policy-based authorization, the following identity policy-based permissions are required.

    Action

    Access Level

    Resource Type (*: required)

    Condition Key

    Alias

    Dependency

    mrs:cluster:createJob

    Write

    cluster *

    • g:ResourceTag/<tag-key>

    • g:EnterpriseProjectId
    • mrs:job:submit

    -

URI

POST /v2/{project_id}/clusters/{cluster_id}/job-executions
Table 1 URI parameters

Parameter

Mandatory

Type

Description

project_id

Yes

String

Definition

Project ID. For details about how to obtain the project ID, see Obtaining a Project ID.

Constraints

N/A

Range

The value must consist of 1 to 64 characters. Only letters and digits are allowed.

Default Value

N/A

cluster_id

Yes

String

Definition

Cluster ID. For details on how to obtain the cluster ID, see Obtaining a Cluster ID.

Constraints

N/A

Range

The value can contain 1 to 64 characters, including only letters, digits, underscores (_), and hyphens (-).

Default Value

N/A

Request Parameters

Table 2 Request parameters

Parameter

Mandatory

Type

Description

job_type

Yes

String

Definition

The job type.

Constraints

N/A

Range

  • MapReduce: provides a distributed data processing model and execution environment capable of rapidly handling large-scale data in parallel. With MRS, you can submit MapReduce JAR programs.

  • SparkSubmit: allows you to submit Spark JAR and Spark Python programs and run Spark applications to compute and process user data.

  • SparkPython: SparkPython jobs are converted to SparkSubmit jobs for submission. On the MRS console, the job type is displayed as SparkSubmit. When calling an API to query the job list, select SparkSubmit.

  • HiveScript: Hive is an open-source data warehouse that runs on Hadoop. With MRS, you can submit HiveScript scripts for execution.

  • HiveSql: Hive is an open-source data warehouse that runs on Hadoop. With MRS, you can directly execute Hive SQL statements.

  • DistCp: a Hadoop tool used to efficiently import and export data between distributed file systems (such as HDFS).

  • SparkScript: allows you to submit SparkScript scripts and batch execute Spark SQL statements.

  • Spark SQL: allows you to use SQL-like statements provided by Spark to query and analyze user data in real time.

  • Flink: a distributed big data processing engine that can perform stateful computations over both unbounded and bounded data streams.

Default Value

N/A

job_name

Yes

String

Definition

Job name.

Constraints

N/A

Range

The value can contain 1 to 64 characters, including only letters, digits, underscores (_), and hyphens (-).

Identical job names are allowed but not recommended.

Default Value

N/A

arguments

No

Array of strings

Definition

Key parameter for program execution. The parameter is specified by the function of the user's program. MRS is only responsible for loading the parameter.

Each string represents a program execution parameter.

Constraints

The value can contain a maximum of 150,000 characters. Special characters (;|&>'<$!\\) are not allowed. This parameter can be left blank.

NOTE:
  • When entering parameters with sensitive information, such as a login password, be aware that these values may appear in job details or logs. Exercise caution when using such inputs.
  • For MRS 1.9.2 or later, a file path on OBS can start with obs://. To use this format to submit HiveScript or HiveSQL jobs, choose Components > Hive > Service Configuration on the cluster details page. Set Type to All, and search for core.site.customized.configs. Add the endpoint configuration item fs.obs.endpoint of OBS and enter the endpoint corresponding to OBS in Value. For details, see Endpoints.
  • For MRS 3.x or later, a file path on OBS can start with obs://. To use this format to submit HiveScript or HiveSQL jobs, choose Components > Hive > Service Configuration on the cluster details page. Set Basic to All, and search for core.site.customized.configs. Add the endpoint configuration item fs.obs.endpoint of OBS and enter the endpoint corresponding to OBS in Value. For details, see Endpoints.

Range

N/A

Default Value

N/A

properties

No

Map<String,String>

Definition

Program system parameters. Used to configure optimization parameters for the job (such as threads, memory, and CPU cores) to improve resource utilization efficiency and job execution performance.

The two strings represent the parameter name and value. For example, the parameter name -ytm indicates the total process memory size of a single TaskManager container, and the parameter value is 8192.

Constraints

The parameter contains a maximum of 2,048 characters, excluding special characters such as ><|'`&!\, and can be left blank.

Range

N/A

Default Value

N/A

Response Parameters

Status code: 200

Table 3 Response body parameter

Parameter

Type

Description

job_submit_result

JobSubmitResult object

Definition

The job execution result. For details about the parameters, see Table 4.

Range

N/A
Table 4 JobSubmitResult parameters

Parameter

Type

Description

job_id

String

Definition

Job ID

Range

N/A

state

String

Definition

Job submission status.

Range

  • COMPLETE: The job is submitted.
  • FAILED: Failed to submit the job.

Status code: 400

Table 5 Response body parameters

Parameter

Type

Description

error_code

String

Definition

Error code.

Range

400: The operation failed.

error_msg

String

Definition

Error message.

Range

400: The operation failed.

Example Request

You must have prepared the OBS paths, sample files, endpoints, and AKs/SKs when submitting a request.

  • Create a MapReduce job. 
    POST https://{endpoint}/v2/{project_id}/clusters/{cluster_id}/job-executions

{
  "job_name" : "MapReduceTest",
  "job_type" : "MapReduce",
  "arguments" : [ "obs://obs-test/program/hadoop-mapreduce-examples-x.x.x.jar", "wordcount", "obs://obs-test/input/", "obs://obs-test/job/mapreduce/output" ],
  "properties" : {
    "fs.obs.endpoint" : "obs endpoint",
    "fs.obs.access.key" : "xxx",
    "fs.obs.secret.key" : "yyy"
  }
}
  • Create a SparkSubmit job. 
    POST https://{endpoint}/v2/{project_id}/clusters/{cluster_id}/job-executions

{
  "job_name" : "SparkSubmitTest",
  "job_type" : "SparkSubmit",
  "arguments" : [ "--master", "yarn", "--deploy-mode", "cluster", "--py-files", "obs://obs-test/a.py", "--conf", "spark.yarn.appMasterEnv.PYTHONPATH=/tmp:$PYTHONPATH", "--conf", "spark.yarn.appMasterEnv.aaa=aaaa", "--conf", "spark.executorEnv.aaa=executoraaa", "--properties-file", "obs://obs-test/test-spark.conf", "obs://obs-test/pi.py", "100000" ],
  "properties" : {
    "fs.obs.access.key" : "xxx",
    "fs.obs.secret.key" : "yyy"
  }
}
  • Create a HiveScript job. 
    POST https://{endpoint}/v2/{project_id}/clusters/{cluster_id}/job-executions

{
  "job_name" : "HiveScriptTest",
  "job_type" : "HiveScript",
  "arguments" : [ "obs://obs-test/sql/test_script.sql" ],
  "properties" : {
    "fs.obs.endpoint" : "obs endpoint",
    "fs.obs.access.key" : "xxx",
    "fs.obs.secret.key" : "yyy"
  }
}
  • Create a HiveSql job. 
    POST https://{endpoint}/v2/{project_id}/clusters/{cluster_id}/job-executions

{
  "job_name" : "HiveSqlTest",
  "job_type" : "HiveSql",
  "arguments" : [ "DROP TABLE IF EXISTS src_wordcount;\ncreate external table src_wordcount(line string) row format delimited fields terminated by \"\\n\" stored as textfile location \"obs://donotdel-gxc/input/\";\ninsert into src_wordcount values(\"v1\")" ],
  "properties" : {
    "fs.obs.endpoint" : "obs endpoint",
    "fs.obs.access.key" : "xxx",
    "fs.obs.secret.key" : "yyy"
  }
}
  • Create a DistCp job. 
    POST https://{endpoint}/v2/{project_id}/clusters/{cluster_id}/job-executions

{
  "job_name" : "DistCpTest",
  "job_type" : "DistCp",
  "arguments" : [ "obs://obs-test/DistcpJob/", "/user/test/sparksql/" ],
  "properties" : {
    "fs.obs.endpoint" : "obs endpoint",
    "fs.obs.access.key" : "xxx",
    "fs.obs.secret.key" : "yyy"
  }
}
  • Create a SparkScript job. 
    POST https://{endpoint}/v2/{project_id}/clusters/{cluster_id}/job-executions

{
  "job_type" : "SparkScript",
  "job_name" : "SparkScriptTest",
  "arguments" : [ "op-key1", "op-value1", "op-key2", "op-value2", "obs://obs-test/sql/test_script.sql" ],
  "properties" : {
    "fs.obs.access.key" : "xxx",
    "fs.obs.secret.key" : "yyy"
  }
}
  • Create a SparkSql job. 
    POST https://{endpoint}/v2/{project_id}/clusters/{cluster_id}/job-executions

{
  "job_type" : "SparkSql",
  "job_name" : "SparkSqlTest",
  "arguments" : [ "op-key1", "op-value1", "op-key2", "op-value2", "create table student_info3 (id string,name string,gender string,age int,addr string);" ],
  "properties" : {
    "fs.obs.access.key" : "xxx",
    "fs.obs.secret.key" : "yyy"
  }
}
  • Create a Flink job. 
    POST https://{endpoint}/v2/{project_id}/clusters/{cluster_id}/job-executions

{
  "job_name" : "flinkTest",
  "job_type" : "Flink",
  "arguments" : [ "run", "-d", "-ynm", "testExcutorejobhdfsbatch", "-m", "yarn-cluster", "hdfs://test/examples/batch/WordCount.jar" ],
  "properties" : {
    "fs.obs.endpoint" : "obs endpoint",
    "fs.obs.access.key" : "xxx",
    "fs.obs.secret.key" : "yyy"
  }
}
  • Create a SparkPython job (Jobs of this type will be converted to SparkSubmit jobs for submission. The job type is displayed as SparkSubmit on the MRS console. Select SparkSubmit when you call an API to query the job list.) 
    POST https://{endpoint}/v2/{project_id}/clusters/{cluster_id}/job-executions

{
  "job_type" : "SparkPython",
  "job_name" : "SparkPythonTest",
  "arguments" : [ "--master", "yarn", "--deploy-mode", "cluster", "--py-files", "obs://obs-test/a.py", "--conf", "spark.yarn.appMasterEnv.PYTHONPATH=/tmp:$PYTHONPATH", "--conf", "spark.yarn.appMasterEnv.aaa=aaaa", "--conf", "spark.executorEnv.aaa=executoraaa", "--properties-file", "obs://obs-test/test-spark.conf", "obs://obs-test/pi.py", "100000" ],
  "properties" : {
    "fs.obs.access.key" : "xxx",
    "fs.obs.secret.key" : "yyy"
  }
}

Example Response

Status code: 200

  • Example of a successful response 
    {
  "job_submit_result":{
      "job_id":"44b37a20-ffe8-42b1-b42b-78a5978d7e40",
      "state":"COMPLETE"
  }
}

Status code: 400

  • Example of a failed response 
    {
"error_msg": Hive jobs cannot be submitted.
"error_code":"0168"
}

SDK Sample Code

The SDK sample code is as follows.

  • Create a MapReduce job.

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
    		 
    package com.huaweicloud.sdk.test;

import com.huaweicloud.sdk.core.auth.ICredential;
import com.huaweicloud.sdk.core.auth.BasicCredentials;
import com.huaweicloud.sdk.core.exception.ConnectionException;
import com.huaweicloud.sdk.core.exception.RequestTimeoutException;
import com.huaweicloud.sdk.core.exception.ServiceResponseException;
import com.huaweicloud.sdk.mrs.v2.region.MrsRegion;
import com.huaweicloud.sdk.mrs.v2.*;
import com.huaweicloud.sdk.mrs.v2.model.*;

import java.util.List;
import java.util.ArrayList;
import java.util.Map;
import java.util.HashMap;

public class CreateExecuteJobSolution {

    public static void main(String[] args) {
        // The AK and SK used for authentication are hard-coded or stored in plaintext, which has great security risks. It is recommended that the AK and SK be stored in ciphertext in configuration files or environment variables and decrypted during use to ensure security.
        // In this example, AK and SK are stored in environment variables for authentication. Before running this example, set environment variables CLOUD_SDK_AK and CLOUD_SDK_SK in the local environment
        String ak = System.getenv("CLOUD_SDK_AK");
        String sk = System.getenv("CLOUD_SDK_SK");
        String projectId = "{project_id}";

        ICredential auth = new BasicCredentials()
                .withProjectId(projectId)
                .withAk(ak)
                .withSk(sk);

        MrsClient client = MrsClient.newBuilder()
                .withCredential(auth)
                .withRegion(MrsRegion.valueOf("<YOUR REGION>"))
                .build();
        CreateExecuteJobRequest request = new CreateExecuteJobRequest();
        request.withClusterId("{cluster_id}");
        JobExecution body = new JobExecution();
        Map<String, String> listbodyProperties = new HashMap<>();
        listbodyProperties.put("fs.obs.endpoint", "obs endpoint");
        listbodyProperties.put("fs.obs.access.key", "xxx");
        listbodyProperties.put("fs.obs.secret.key", "yyy");
        List<String> listbodyArguments = new ArrayList<>();
        listbodyArguments.add("obs://obs-test/program/hadoop-mapreduce-examples-x.x.x.jar");
        listbodyArguments.add("wordcount");
        listbodyArguments.add("obs://obs-test/input/");
        listbodyArguments.add("obs://obs-test/job/mapreduce/output");
        body.withProperties(listbodyProperties);
        body.withArguments(listbodyArguments);
        body.withJobName("MapReduceTest");
        body.withJobType("MapReduce");
        request.withBody(body);
        try {
            CreateExecuteJobResponse response = client.createExecuteJob(request);
            System.out.println(response.toString());
        } catch (ConnectionException e) {
            e.printStackTrace();
        } catch (RequestTimeoutException e) {
            e.printStackTrace();
        } catch (ServiceResponseException e) {
            e.printStackTrace();
            System.out.println(e.getHttpStatusCode());
            System.out.println(e.getRequestId());
            System.out.println(e.getErrorCode());
            System.out.println(e.getErrorMsg());
        }
    }
}

  • Create a SparkSubmit job.

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
    		 
    package com.huaweicloud.sdk.test;

import com.huaweicloud.sdk.core.auth.ICredential;
import com.huaweicloud.sdk.core.auth.BasicCredentials;
import com.huaweicloud.sdk.core.exception.ConnectionException;
import com.huaweicloud.sdk.core.exception.RequestTimeoutException;
import com.huaweicloud.sdk.core.exception.ServiceResponseException;
import com.huaweicloud.sdk.mrs.v2.region.MrsRegion;
import com.huaweicloud.sdk.mrs.v2.*;
import com.huaweicloud.sdk.mrs.v2.model.*;

import java.util.List;
import java.util.ArrayList;
import java.util.Map;
import java.util.HashMap;

public class CreateExecuteJobSolution {

    public static void main(String[] args) {
        // The AK and SK used for authentication are hard-coded or stored in plaintext, which has great security risks. It is recommended that the AK and SK be stored in ciphertext in configuration files or environment variables and decrypted during use to ensure security.
        // In this example, AK and SK are stored in environment variables for authentication. Before running this example, set environment variables CLOUD_SDK_AK and CLOUD_SDK_SK in the local environment
        String ak = System.getenv("CLOUD_SDK_AK");
        String sk = System.getenv("CLOUD_SDK_SK");
        String projectId = "{project_id}";

        ICredential auth = new BasicCredentials()
                .withProjectId(projectId)
                .withAk(ak)
                .withSk(sk);

        MrsClient client = MrsClient.newBuilder()
                .withCredential(auth)
                .withRegion(MrsRegion.valueOf("<YOUR REGION>"))
                .build();
        CreateExecuteJobRequest request = new CreateExecuteJobRequest();
        request.withClusterId("{cluster_id}");
        JobExecution body = new JobExecution();
        Map<String, String> listbodyProperties = new HashMap<>();
        listbodyProperties.put("fs.obs.access.key", "xxx");
        listbodyProperties.put("fs.obs.secret.key", "yyy");
        List<String> listbodyArguments = new ArrayList<>();
        listbodyArguments.add("--master");
        listbodyArguments.add("yarn");
        listbodyArguments.add("--deploy-mode");
        listbodyArguments.add("cluster");
        listbodyArguments.add("--py-files");
        listbodyArguments.add("obs://obs-test/a.py");
        listbodyArguments.add("--conf");
        listbodyArguments.add("spark.yarn.appMasterEnv.PYTHONPATH=/tmp:$PYTHONPATH");
        listbodyArguments.add("--conf");
        listbodyArguments.add("spark.yarn.appMasterEnv.aaa=aaaa");
        listbodyArguments.add("--conf");
        listbodyArguments.add("spark.executorEnv.aaa=executoraaa");
        listbodyArguments.add("--properties-file");
        listbodyArguments.add("obs://obs-test/test-spark.conf");
        listbodyArguments.add("obs://obs-test/pi.py");
        listbodyArguments.add("100000");
        body.withProperties(listbodyProperties);
        body.withArguments(listbodyArguments);
        body.withJobName("SparkSubmitTest");
        body.withJobType("SparkSubmit");
        request.withBody(body);
        try {
            CreateExecuteJobResponse response = client.createExecuteJob(request);
            System.out.println(response.toString());
        } catch (ConnectionException e) {
            e.printStackTrace();
        } catch (RequestTimeoutException e) {
            e.printStackTrace();
        } catch (ServiceResponseException e) {
            e.printStackTrace();
            System.out.println(e.getHttpStatusCode());
            System.out.println(e.getRequestId());
            System.out.println(e.getErrorCode());
            System.out.println(e.getErrorMsg());
        }
    }
}

  • Create a HiveScript job.

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
    		 
    package com.huaweicloud.sdk.test;

import com.huaweicloud.sdk.core.auth.ICredential;
import com.huaweicloud.sdk.core.auth.BasicCredentials;
import com.huaweicloud.sdk.core.exception.ConnectionException;
import com.huaweicloud.sdk.core.exception.RequestTimeoutException;
import com.huaweicloud.sdk.core.exception.ServiceResponseException;
import com.huaweicloud.sdk.mrs.v2.region.MrsRegion;
import com.huaweicloud.sdk.mrs.v2.*;
import com.huaweicloud.sdk.mrs.v2.model.*;

import java.util.List;
import java.util.ArrayList;
import java.util.Map;
import java.util.HashMap;

public class CreateExecuteJobSolution {

    public static void main(String[] args) {
        // The AK and SK used for authentication are hard-coded or stored in plaintext, which has great security risks. It is recommended that the AK and SK be stored in ciphertext in configuration files or environment variables and decrypted during use to ensure security.
        // In this example, AK and SK are stored in environment variables for authentication. Before running this example, set environment variables CLOUD_SDK_AK and CLOUD_SDK_SK in the local environment
        String ak = System.getenv("CLOUD_SDK_AK");
        String sk = System.getenv("CLOUD_SDK_SK");
        String projectId = "{project_id}";

        ICredential auth = new BasicCredentials()
                .withProjectId(projectId)
                .withAk(ak)
                .withSk(sk);

        MrsClient client = MrsClient.newBuilder()
                .withCredential(auth)
                .withRegion(MrsRegion.valueOf("<YOUR REGION>"))
                .build();
        CreateExecuteJobRequest request = new CreateExecuteJobRequest();
        request.withClusterId("{cluster_id}");
        JobExecution body = new JobExecution();
        Map<String, String> listbodyProperties = new HashMap<>();
        listbodyProperties.put("fs.obs.endpoint", "obs endpoint");
        listbodyProperties.put("fs.obs.access.key", "xxx");
        listbodyProperties.put("fs.obs.secret.key", "yyy");
        List<String> listbodyArguments = new ArrayList<>();
        listbodyArguments.add("obs://obs-test/sql/test_script.sql");
        body.withProperties(listbodyProperties);
        body.withArguments(listbodyArguments);
        body.withJobName("HiveScriptTest");
        body.withJobType("HiveScript");
        request.withBody(body);
        try {
            CreateExecuteJobResponse response = client.createExecuteJob(request);
            System.out.println(response.toString());
        } catch (ConnectionException e) {
            e.printStackTrace();
        } catch (RequestTimeoutException e) {
            e.printStackTrace();
        } catch (ServiceResponseException e) {
            e.printStackTrace();
            System.out.println(e.getHttpStatusCode());
            System.out.println(e.getRequestId());
            System.out.println(e.getErrorCode());
            System.out.println(e.getErrorMsg());
        }
    }
}

  • Create a HiveSql job.

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
    		 
    package com.huaweicloud.sdk.test;

import com.huaweicloud.sdk.core.auth.ICredential;
import com.huaweicloud.sdk.core.auth.BasicCredentials;
import com.huaweicloud.sdk.core.exception.ConnectionException;
import com.huaweicloud.sdk.core.exception.RequestTimeoutException;
import com.huaweicloud.sdk.core.exception.ServiceResponseException;
import com.huaweicloud.sdk.mrs.v2.region.MrsRegion;
import com.huaweicloud.sdk.mrs.v2.*;
import com.huaweicloud.sdk.mrs.v2.model.*;

import java.util.List;
import java.util.ArrayList;
import java.util.Map;
import java.util.HashMap;

public class CreateExecuteJobSolution {

    public static void main(String[] args) {
        // The AK and SK used for authentication are hard-coded or stored in plaintext, which has great security risks. It is recommended that the AK and SK be stored in ciphertext in configuration files or environment variables and decrypted during use to ensure security.
        // In this example, AK and SK are stored in environment variables for authentication. Before running this example, set environment variables CLOUD_SDK_AK and CLOUD_SDK_SK in the local environment
        String ak = System.getenv("CLOUD_SDK_AK");
        String sk = System.getenv("CLOUD_SDK_SK");
        String projectId = "{project_id}";

        ICredential auth = new BasicCredentials()
                .withProjectId(projectId)
                .withAk(ak)
                .withSk(sk);

        MrsClient client = MrsClient.newBuilder()
                .withCredential(auth)
                .withRegion(MrsRegion.valueOf("<YOUR REGION>"))
                .build();
        CreateExecuteJobRequest request = new CreateExecuteJobRequest();
        request.withClusterId("{cluster_id}");
        JobExecution body = new JobExecution();
        Map<String, String> listbodyProperties = new HashMap<>();
        listbodyProperties.put("fs.obs.endpoint", "obs endpoint");
        listbodyProperties.put("fs.obs.access.key", "xxx");
        listbodyProperties.put("fs.obs.secret.key", "yyy");
        List<String> listbodyArguments = new ArrayList<>();
        listbodyArguments.add("DROP TABLE IF EXISTS src_wordcount;
create external table src_wordcount(line string) row format delimited fields terminated by "\n" stored as textfile location "obs://donotdel-gxc/input/";
insert into src_wordcount values("v1")");
        body.withProperties(listbodyProperties);
        body.withArguments(listbodyArguments);
        body.withJobName("HiveSqlTest");
        body.withJobType("HiveSql");
        request.withBody(body);
        try {
            CreateExecuteJobResponse response = client.createExecuteJob(request);
            System.out.println(response.toString());
        } catch (ConnectionException e) {
            e.printStackTrace();
        } catch (RequestTimeoutException e) {
            e.printStackTrace();
        } catch (ServiceResponseException e) {
            e.printStackTrace();
            System.out.println(e.getHttpStatusCode());
            System.out.println(e.getRequestId());
            System.out.println(e.getErrorCode());
            System.out.println(e.getErrorMsg());
        }
    }
}

  • Create a DistCp job.

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
    		 
    package com.huaweicloud.sdk.test;

import com.huaweicloud.sdk.core.auth.ICredential;
import com.huaweicloud.sdk.core.auth.BasicCredentials;
import com.huaweicloud.sdk.core.exception.ConnectionException;
import com.huaweicloud.sdk.core.exception.RequestTimeoutException;
import com.huaweicloud.sdk.core.exception.ServiceResponseException;
import com.huaweicloud.sdk.mrs.v2.region.MrsRegion;
import com.huaweicloud.sdk.mrs.v2.*;
import com.huaweicloud.sdk.mrs.v2.model.*;

import java.util.List;
import java.util.ArrayList;
import java.util.Map;
import java.util.HashMap;

public class CreateExecuteJobSolution {

    public static void main(String[] args) {
        // The AK and SK used for authentication are hard-coded or stored in plaintext, which has great security risks. It is recommended that the AK and SK be stored in ciphertext in configuration files or environment variables and decrypted during use to ensure security.
        // In this example, AK and SK are stored in environment variables for authentication. Before running this example, set environment variables CLOUD_SDK_AK and CLOUD_SDK_SK in the local environment
        String ak = System.getenv("CLOUD_SDK_AK");
        String sk = System.getenv("CLOUD_SDK_SK");
        String projectId = "{project_id}";

        ICredential auth = new BasicCredentials()
                .withProjectId(projectId)
                .withAk(ak)
                .withSk(sk);

        MrsClient client = MrsClient.newBuilder()
                .withCredential(auth)
                .withRegion(MrsRegion.valueOf("<YOUR REGION>"))
                .build();
        CreateExecuteJobRequest request = new CreateExecuteJobRequest();
        request.withClusterId("{cluster_id}");
        JobExecution body = new JobExecution();
        Map<String, String> listbodyProperties = new HashMap<>();
        listbodyProperties.put("fs.obs.endpoint", "obs endpoint");
        listbodyProperties.put("fs.obs.access.key", "xxx");
        listbodyProperties.put("fs.obs.secret.key", "yyy");
        List<String> listbodyArguments = new ArrayList<>();
        listbodyArguments.add("obs://obs-test/DistcpJob/");
        listbodyArguments.add("/user/test/sparksql/");
        body.withProperties(listbodyProperties);
        body.withArguments(listbodyArguments);
        body.withJobName("DistCpTest");
        body.withJobType("DistCp");
        request.withBody(body);
        try {
            CreateExecuteJobResponse response = client.createExecuteJob(request);
            System.out.println(response.toString());
        } catch (ConnectionException e) {
            e.printStackTrace();
        } catch (RequestTimeoutException e) {
            e.printStackTrace();
        } catch (ServiceResponseException e) {
            e.printStackTrace();
            System.out.println(e.getHttpStatusCode());
            System.out.println(e.getRequestId());
            System.out.println(e.getErrorCode());
            System.out.println(e.getErrorMsg());
        }
    }
}

  • Create a SparkScript job.

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
    		 
    package com.huaweicloud.sdk.test;

import com.huaweicloud.sdk.core.auth.ICredential;
import com.huaweicloud.sdk.core.auth.BasicCredentials;
import com.huaweicloud.sdk.core.exception.ConnectionException;
import com.huaweicloud.sdk.core.exception.RequestTimeoutException;
import com.huaweicloud.sdk.core.exception.ServiceResponseException;
import com.huaweicloud.sdk.mrs.v2.region.MrsRegion;
import com.huaweicloud.sdk.mrs.v2.*;
import com.huaweicloud.sdk.mrs.v2.model.*;

import java.util.List;
import java.util.ArrayList;
import java.util.Map;
import java.util.HashMap;

public class CreateExecuteJobSolution {

    public static void main(String[] args) {
        // The AK and SK used for authentication are hard-coded or stored in plaintext, which has great security risks. It is recommended that the AK and SK be stored in ciphertext in configuration files or environment variables and decrypted during use to ensure security.
        // In this example, AK and SK are stored in environment variables for authentication. Before running this example, set environment variables CLOUD_SDK_AK and CLOUD_SDK_SK in the local environment
        String ak = System.getenv("CLOUD_SDK_AK");
        String sk = System.getenv("CLOUD_SDK_SK");
        String projectId = "{project_id}";

        ICredential auth = new BasicCredentials()
                .withProjectId(projectId)
                .withAk(ak)
                .withSk(sk);

        MrsClient client = MrsClient.newBuilder()
                .withCredential(auth)
                .withRegion(MrsRegion.valueOf("<YOUR REGION>"))
                .build();
        CreateExecuteJobRequest request = new CreateExecuteJobRequest();
        request.withClusterId("{cluster_id}");
        JobExecution body = new JobExecution();
        Map<String, String> listbodyProperties = new HashMap<>();
        listbodyProperties.put("fs.obs.access.key", "xxx");
        listbodyProperties.put("fs.obs.secret.key", "yyy");
        List<String> listbodyArguments = new ArrayList<>();
        listbodyArguments.add("op-key1");
        listbodyArguments.add("op-value1");
        listbodyArguments.add("op-key2");
        listbodyArguments.add("op-value2");
        listbodyArguments.add("obs://obs-test/sql/test_script.sql");
        body.withProperties(listbodyProperties);
        body.withArguments(listbodyArguments);
        body.withJobName("SparkScriptTest");
        body.withJobType("SparkScript");
        request.withBody(body);
        try {
            CreateExecuteJobResponse response = client.createExecuteJob(request);
            System.out.println(response.toString());
        } catch (ConnectionException e) {
            e.printStackTrace();
        } catch (RequestTimeoutException e) {
            e.printStackTrace();
        } catch (ServiceResponseException e) {
            e.printStackTrace();
            System.out.println(e.getHttpStatusCode());
            System.out.println(e.getRequestId());
            System.out.println(e.getErrorCode());
            System.out.println(e.getErrorMsg());
        }
    }
}

  • Create a SparkSql job.

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
    		 
    package com.huaweicloud.sdk.test;

import com.huaweicloud.sdk.core.auth.ICredential;
import com.huaweicloud.sdk.core.auth.BasicCredentials;
import com.huaweicloud.sdk.core.exception.ConnectionException;
import com.huaweicloud.sdk.core.exception.RequestTimeoutException;
import com.huaweicloud.sdk.core.exception.ServiceResponseException;
import com.huaweicloud.sdk.mrs.v2.region.MrsRegion;
import com.huaweicloud.sdk.mrs.v2.*;
import com.huaweicloud.sdk.mrs.v2.model.*;

import java.util.List;
import java.util.ArrayList;
import java.util.Map;
import java.util.HashMap;

public class CreateExecuteJobSolution {

    public static void main(String[] args) {
        // The AK and SK used for authentication are hard-coded or stored in plaintext, which has great security risks. It is recommended that the AK and SK be stored in ciphertext in configuration files or environment variables and decrypted during use to ensure security.
        // In this example, AK and SK are stored in environment variables for authentication. Before running this example, set environment variables CLOUD_SDK_AK and CLOUD_SDK_SK in the local environment
        String ak = System.getenv("CLOUD_SDK_AK");
        String sk = System.getenv("CLOUD_SDK_SK");
        String projectId = "{project_id}";

        ICredential auth = new BasicCredentials()
                .withProjectId(projectId)
                .withAk(ak)
                .withSk(sk);

        MrsClient client = MrsClient.newBuilder()
                .withCredential(auth)
                .withRegion(MrsRegion.valueOf("<YOUR REGION>"))
                .build();
        CreateExecuteJobRequest request = new CreateExecuteJobRequest();
        request.withClusterId("{cluster_id}");
        JobExecution body = new JobExecution();
        Map<String, String> listbodyProperties = new HashMap<>();
        listbodyProperties.put("fs.obs.access.key", "xxx");
        listbodyProperties.put("fs.obs.secret.key", "yyy");
        List<String> listbodyArguments = new ArrayList<>();
        listbodyArguments.add("op-key1");
        listbodyArguments.add("op-value1");
        listbodyArguments.add("op-key2");
        listbodyArguments.add("op-value2");
        listbodyArguments.add("create table student_info3 (id string,name string,gender string,age int,addr string);");
        body.withProperties(listbodyProperties);
        body.withArguments(listbodyArguments);
        body.withJobName("SparkSqlTest");
        body.withJobType("SparkSql");
        request.withBody(body);
        try {
            CreateExecuteJobResponse response = client.createExecuteJob(request);
            System.out.println(response.toString());
        } catch (ConnectionException e) {
            e.printStackTrace();
        } catch (RequestTimeoutException e) {
            e.printStackTrace();
        } catch (ServiceResponseException e) {
            e.printStackTrace();
            System.out.println(e.getHttpStatusCode());
            System.out.println(e.getRequestId());
            System.out.println(e.getErrorCode());
            System.out.println(e.getErrorMsg());
        }
    }
}

  • Create a Flink job.

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
    		 
    package com.huaweicloud.sdk.test;

import com.huaweicloud.sdk.core.auth.ICredential;
import com.huaweicloud.sdk.core.auth.BasicCredentials;
import com.huaweicloud.sdk.core.exception.ConnectionException;
import com.huaweicloud.sdk.core.exception.RequestTimeoutException;
import com.huaweicloud.sdk.core.exception.ServiceResponseException;
import com.huaweicloud.sdk.mrs.v2.region.MrsRegion;
import com.huaweicloud.sdk.mrs.v2.*;
import com.huaweicloud.sdk.mrs.v2.model.*;

import java.util.List;
import java.util.ArrayList;
import java.util.Map;
import java.util.HashMap;

public class CreateExecuteJobSolution {

    public static void main(String[] args) {
        // The AK and SK used for authentication are hard-coded or stored in plaintext, which has great security risks. It is recommended that the AK and SK be stored in ciphertext in configuration files or environment variables and decrypted during use to ensure security.
        // In this example, AK and SK are stored in environment variables for authentication. Before running this example, set environment variables CLOUD_SDK_AK and CLOUD_SDK_SK in the local environment
        String ak = System.getenv("CLOUD_SDK_AK");
        String sk = System.getenv("CLOUD_SDK_SK");
        String projectId = "{project_id}";

        ICredential auth = new BasicCredentials()
                .withProjectId(projectId)
                .withAk(ak)
                .withSk(sk);

        MrsClient client = MrsClient.newBuilder()
                .withCredential(auth)
                .withRegion(MrsRegion.valueOf("<YOUR REGION>"))
                .build();
        CreateExecuteJobRequest request = new CreateExecuteJobRequest();
        request.withClusterId("{cluster_id}");
        JobExecution body = new JobExecution();
        Map<String, String> listbodyProperties = new HashMap<>();
        listbodyProperties.put("fs.obs.endpoint", "obs endpoint");
        listbodyProperties.put("fs.obs.access.key", "xxx");
        listbodyProperties.put("fs.obs.secret.key", "yyy");
        List<String> listbodyArguments = new ArrayList<>();
        listbodyArguments.add("run");
        listbodyArguments.add("-d");
        listbodyArguments.add("-ynm");
        listbodyArguments.add("testExcutorejobhdfsbatch");
        listbodyArguments.add("-m");
        listbodyArguments.add("yarn-cluster");
        listbodyArguments.add("hdfs://test/examples/batch/WordCount.jar");
        body.withProperties(listbodyProperties);
        body.withArguments(listbodyArguments);
        body.withJobName("flinkTest");
        body.withJobType("Flink");
        request.withBody(body);
        try {
            CreateExecuteJobResponse response = client.createExecuteJob(request);
            System.out.println(response.toString());
        } catch (ConnectionException e) {
            e.printStackTrace();
        } catch (RequestTimeoutException e) {
            e.printStackTrace();
        } catch (ServiceResponseException e) {
            e.printStackTrace();
            System.out.println(e.getHttpStatusCode());
            System.out.println(e.getRequestId());
            System.out.println(e.getErrorCode());
            System.out.println(e.getErrorMsg());
        }
    }
}

  • Create a SparkPython job (Jobs of this type will be converted to SparkSubmit jobs for submission. The job type is displayed as SparkSubmit on the MRS console. Select SparkSubmit when you call an API to query the job list.)

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
    		 
    package com.huaweicloud.sdk.test;

import com.huaweicloud.sdk.core.auth.ICredential;
import com.huaweicloud.sdk.core.auth.BasicCredentials;
import com.huaweicloud.sdk.core.exception.ConnectionException;
import com.huaweicloud.sdk.core.exception.RequestTimeoutException;
import com.huaweicloud.sdk.core.exception.ServiceResponseException;
import com.huaweicloud.sdk.mrs.v2.region.MrsRegion;
import com.huaweicloud.sdk.mrs.v2.*;
import com.huaweicloud.sdk.mrs.v2.model.*;

import java.util.List;
import java.util.ArrayList;
import java.util.Map;
import java.util.HashMap;

public class CreateExecuteJobSolution {

    public static void main(String[] args) {
        // The AK and SK used for authentication are hard-coded or stored in plaintext, which has great security risks. It is recommended that the AK and SK be stored in ciphertext in configuration files or environment variables and decrypted during use to ensure security.
        // In this example, AK and SK are stored in environment variables for authentication. Before running this example, set environment variables CLOUD_SDK_AK and CLOUD_SDK_SK in the local environment
        String ak = System.getenv("CLOUD_SDK_AK");
        String sk = System.getenv("CLOUD_SDK_SK");
        String projectId = "{project_id}";

        ICredential auth = new BasicCredentials()
                .withProjectId(projectId)
                .withAk(ak)
                .withSk(sk);

        MrsClient client = MrsClient.newBuilder()
                .withCredential(auth)
                .withRegion(MrsRegion.valueOf("<YOUR REGION>"))
                .build();
        CreateExecuteJobRequest request = new CreateExecuteJobRequest();
        request.withClusterId("{cluster_id}");
        JobExecution body = new JobExecution();
        Map<String, String> listbodyProperties = new HashMap<>();
        listbodyProperties.put("fs.obs.access.key", "xxx");
        listbodyProperties.put("fs.obs.secret.key", "yyy");
        List<String> listbodyArguments = new ArrayList<>();
        listbodyArguments.add("--master");
        listbodyArguments.add("yarn");
        listbodyArguments.add("--deploy-mode");
        listbodyArguments.add("cluster");
        listbodyArguments.add("--py-files");
        listbodyArguments.add("obs://obs-test/a.py");
        listbodyArguments.add("--conf");
        listbodyArguments.add("spark.yarn.appMasterEnv.PYTHONPATH=/tmp:$PYTHONPATH");
        listbodyArguments.add("--conf");
        listbodyArguments.add("spark.yarn.appMasterEnv.aaa=aaaa");
        listbodyArguments.add("--conf");
        listbodyArguments.add("spark.executorEnv.aaa=executoraaa");
        listbodyArguments.add("--properties-file");
        listbodyArguments.add("obs://obs-test/test-spark.conf");
        listbodyArguments.add("obs://obs-test/pi.py");
        listbodyArguments.add("100000");
        body.withProperties(listbodyProperties);
        body.withArguments(listbodyArguments);
        body.withJobName("SparkPythonTest");
        body.withJobType("SparkPython");
        request.withBody(body);
        try {
            CreateExecuteJobResponse response = client.createExecuteJob(request);
            System.out.println(response.toString());
        } catch (ConnectionException e) {
            e.printStackTrace();
        } catch (RequestTimeoutException e) {
            e.printStackTrace();
        } catch (ServiceResponseException e) {
            e.printStackTrace();
            System.out.println(e.getHttpStatusCode());
            System.out.println(e.getRequestId());
            System.out.println(e.getErrorCode());
            System.out.println(e.getErrorMsg());
        }
    }
}

  • Create a MapReduce job.

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
    		 
    # coding: utf-8

import os
from huaweicloudsdkcore.auth.credentials import BasicCredentials
from huaweicloudsdkmrs.v2.region.mrs_region import MrsRegion
from huaweicloudsdkcore.exceptions import exceptions
from huaweicloudsdkmrs.v2 import *

if __name__ == "__main__":
    # The AK and SK used for authentication are hard-coded or stored in plaintext, which has great security risks. It is recommended that the AK and SK be stored in ciphertext in configuration files or environment variables and decrypted during use to ensure security.
    # In this example, AK and SK are stored in environment variables for authentication. Before running this example, set environment variables CLOUD_SDK_AK and CLOUD_SDK_SK in the local environment
    ak = os.environ["CLOUD_SDK_AK"]
    sk = os.environ["CLOUD_SDK_SK"]
    projectId = "{project_id}"

    credentials = BasicCredentials(ak, sk, projectId)

    client = MrsClient.new_builder() \
        .with_credentials(credentials) \
        .with_region(MrsRegion.value_of("<YOUR REGION>")) \
        .build()

    try:
        request = CreateExecuteJobRequest()
        request.cluster_id = "{cluster_id}"
        listPropertiesbody = {
            "fs.obs.endpoint": "obs endpoint",
            "fs.obs.access.key": "xxx",
            "fs.obs.secret.key": "yyy"
        }
        listArgumentsbody = [
            "obs://obs-test/program/hadoop-mapreduce-examples-x.x.x.jar",
            "wordcount",
            "obs://obs-test/input/",
            "obs://obs-test/job/mapreduce/output"
        ]
        request.body = JobExecution(
            properties=listPropertiesbody,
            arguments=listArgumentsbody,
            job_name="MapReduceTest",
            job_type="MapReduce"
        )
        response = client.create_execute_job(request)
        print(response)
    except exceptions.ClientRequestException as e:
        print(e.status_code)
        print(e.request_id)
        print(e.error_code)
        print(e.error_msg)

  • Create a SparkSubmit job.

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
    		 
    # coding: utf-8

import os
from huaweicloudsdkcore.auth.credentials import BasicCredentials
from huaweicloudsdkmrs.v2.region.mrs_region import MrsRegion
from huaweicloudsdkcore.exceptions import exceptions
from huaweicloudsdkmrs.v2 import *

if __name__ == "__main__":
    # The AK and SK used for authentication are hard-coded or stored in plaintext, which has great security risks. It is recommended that the AK and SK be stored in ciphertext in configuration files or environment variables and decrypted during use to ensure security.
    # In this example, AK and SK are stored in environment variables for authentication. Before running this example, set environment variables CLOUD_SDK_AK and CLOUD_SDK_SK in the local environment
    ak = os.environ["CLOUD_SDK_AK"]
    sk = os.environ["CLOUD_SDK_SK"]
    projectId = "{project_id}"

    credentials = BasicCredentials(ak, sk, projectId)

    client = MrsClient.new_builder() \
        .with_credentials(credentials) \
        .with_region(MrsRegion.value_of("<YOUR REGION>")) \
        .build()

    try:
        request = CreateExecuteJobRequest()
        request.cluster_id = "{cluster_id}"
        listPropertiesbody = {
            "fs.obs.access.key": "xxx",
            "fs.obs.secret.key": "yyy"
        }
        listArgumentsbody = [
            "--master",
            "yarn",
            "--deploy-mode",
            "cluster",
            "--py-files",
            "obs://obs-test/a.py",
            "--conf",
            "spark.yarn.appMasterEnv.PYTHONPATH=/tmp:$PYTHONPATH",
            "--conf",
            "spark.yarn.appMasterEnv.aaa=aaaa",
            "--conf",
            "spark.executorEnv.aaa=executoraaa",
            "--properties-file",
            "obs://obs-test/test-spark.conf",
            "obs://obs-test/pi.py",
            "100000"
        ]
        request.body = JobExecution(
            properties=listPropertiesbody,
            arguments=listArgumentsbody,
            job_name="SparkSubmitTest",
            job_type="SparkSubmit"
        )
        response = client.create_execute_job(request)
        print(response)
    except exceptions.ClientRequestException as e:
        print(e.status_code)
        print(e.request_id)
        print(e.error_code)
        print(e.error_msg)

  • Create a HiveScript job.

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
    		 
    # coding: utf-8

import os
from huaweicloudsdkcore.auth.credentials import BasicCredentials
from huaweicloudsdkmrs.v2.region.mrs_region import MrsRegion
from huaweicloudsdkcore.exceptions import exceptions
from huaweicloudsdkmrs.v2 import *

if __name__ == "__main__":
    # The AK and SK used for authentication are hard-coded or stored in plaintext, which has great security risks. It is recommended that the AK and SK be stored in ciphertext in configuration files or environment variables and decrypted during use to ensure security.
    # In this example, AK and SK are stored in environment variables for authentication. Before running this example, set environment variables CLOUD_SDK_AK and CLOUD_SDK_SK in the local environment
    ak = os.environ["CLOUD_SDK_AK"]
    sk = os.environ["CLOUD_SDK_SK"]
    projectId = "{project_id}"

    credentials = BasicCredentials(ak, sk, projectId)

    client = MrsClient.new_builder() \
        .with_credentials(credentials) \
        .with_region(MrsRegion.value_of("<YOUR REGION>")) \
        .build()

    try:
        request = CreateExecuteJobRequest()
        request.cluster_id = "{cluster_id}"
        listPropertiesbody = {
            "fs.obs.endpoint": "obs endpoint",
            "fs.obs.access.key": "xxx",
            "fs.obs.secret.key": "yyy"
        }
        listArgumentsbody = [
            "obs://obs-test/sql/test_script.sql"
        ]
        request.body = JobExecution(
            properties=listPropertiesbody,
            arguments=listArgumentsbody,
            job_name="HiveScriptTest",
            job_type="HiveScript"
        )
        response = client.create_execute_job(request)
        print(response)
    except exceptions.ClientRequestException as e:
        print(e.status_code)
        print(e.request_id)
        print(e.error_code)
        print(e.error_msg)

  • Create a HiveSql job.

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
    		 
    # coding: utf-8

import os
from huaweicloudsdkcore.auth.credentials import BasicCredentials
from huaweicloudsdkmrs.v2.region.mrs_region import MrsRegion
from huaweicloudsdkcore.exceptions import exceptions
from huaweicloudsdkmrs.v2 import *

if __name__ == "__main__":
    # The AK and SK used for authentication are hard-coded or stored in plaintext, which has great security risks. It is recommended that the AK and SK be stored in ciphertext in configuration files or environment variables and decrypted during use to ensure security.
    # In this example, AK and SK are stored in environment variables for authentication. Before running this example, set environment variables CLOUD_SDK_AK and CLOUD_SDK_SK in the local environment
    ak = os.environ["CLOUD_SDK_AK"]
    sk = os.environ["CLOUD_SDK_SK"]
    projectId = "{project_id}"

    credentials = BasicCredentials(ak, sk, projectId)

    client = MrsClient.new_builder() \
        .with_credentials(credentials) \
        .with_region(MrsRegion.value_of("<YOUR REGION>")) \
        .build()

    try:
        request = CreateExecuteJobRequest()
        request.cluster_id = "{cluster_id}"
        listPropertiesbody = {
            "fs.obs.endpoint": "obs endpoint",
            "fs.obs.access.key": "xxx",
            "fs.obs.secret.key": "yyy"
        }
        listArgumentsbody = [
            "DROP TABLE IF EXISTS src_wordcount;
        create external table src_wordcount(line string) row format delimited fields terminated by "\n" stored as textfile location "obs://donotdel-gxc/input/";
        insert into src_wordcount values("v1")"
        ]
        request.body = JobExecution(
            properties=listPropertiesbody,
            arguments=listArgumentsbody,
            job_name="HiveSqlTest",
            job_type="HiveSql"
        )
        response = client.create_execute_job(request)
        print(response)
    except exceptions.ClientRequestException as e:
        print(e.status_code)
        print(e.request_id)
        print(e.error_code)
        print(e.error_msg)

  • Create a DistCp job.

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
    		 
    # coding: utf-8

import os
from huaweicloudsdkcore.auth.credentials import BasicCredentials
from huaweicloudsdkmrs.v2.region.mrs_region import MrsRegion
from huaweicloudsdkcore.exceptions import exceptions
from huaweicloudsdkmrs.v2 import *

if __name__ == "__main__":
    # The AK and SK used for authentication are hard-coded or stored in plaintext, which has great security risks. It is recommended that the AK and SK be stored in ciphertext in configuration files or environment variables and decrypted during use to ensure security.
    # In this example, AK and SK are stored in environment variables for authentication. Before running this example, set environment variables CLOUD_SDK_AK and CLOUD_SDK_SK in the local environment
    ak = os.environ["CLOUD_SDK_AK"]
    sk = os.environ["CLOUD_SDK_SK"]
    projectId = "{project_id}"

    credentials = BasicCredentials(ak, sk, projectId)

    client = MrsClient.new_builder() \
        .with_credentials(credentials) \
        .with_region(MrsRegion.value_of("<YOUR REGION>")) \
        .build()

    try:
        request = CreateExecuteJobRequest()
        request.cluster_id = "{cluster_id}"
        listPropertiesbody = {
            "fs.obs.endpoint": "obs endpoint",
            "fs.obs.access.key": "xxx",
            "fs.obs.secret.key": "yyy"
        }
        listArgumentsbody = [
            "obs://obs-test/DistcpJob/",
            "/user/test/sparksql/"
        ]
        request.body = JobExecution(
            properties=listPropertiesbody,
            arguments=listArgumentsbody,
            job_name="DistCpTest",
            job_type="DistCp"
        )
        response = client.create_execute_job(request)
        print(response)
    except exceptions.ClientRequestException as e:
        print(e.status_code)
        print(e.request_id)
        print(e.error_code)
        print(e.error_msg)

  • Create a SparkScript job.

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
    		 
    # coding: utf-8

import os
from huaweicloudsdkcore.auth.credentials import BasicCredentials
from huaweicloudsdkmrs.v2.region.mrs_region import MrsRegion
from huaweicloudsdkcore.exceptions import exceptions
from huaweicloudsdkmrs.v2 import *

if __name__ == "__main__":
    # The AK and SK used for authentication are hard-coded or stored in plaintext, which has great security risks. It is recommended that the AK and SK be stored in ciphertext in configuration files or environment variables and decrypted during use to ensure security.
    # In this example, AK and SK are stored in environment variables for authentication. Before running this example, set environment variables CLOUD_SDK_AK and CLOUD_SDK_SK in the local environment
    ak = os.environ["CLOUD_SDK_AK"]
    sk = os.environ["CLOUD_SDK_SK"]
    projectId = "{project_id}"

    credentials = BasicCredentials(ak, sk, projectId)

    client = MrsClient.new_builder() \
        .with_credentials(credentials) \
        .with_region(MrsRegion.value_of("<YOUR REGION>")) \
        .build()

    try:
        request = CreateExecuteJobRequest()
        request.cluster_id = "{cluster_id}"
        listPropertiesbody = {
            "fs.obs.access.key": "xxx",
            "fs.obs.secret.key": "yyy"
        }
        listArgumentsbody = [
            "op-key1",
            "op-value1",
            "op-key2",
            "op-value2",
            "obs://obs-test/sql/test_script.sql"
        ]
        request.body = JobExecution(
            properties=listPropertiesbody,
            arguments=listArgumentsbody,
            job_name="SparkScriptTest",
            job_type="SparkScript"
        )
        response = client.create_execute_job(request)
        print(response)
    except exceptions.ClientRequestException as e:
        print(e.status_code)
        print(e.request_id)
        print(e.error_code)
        print(e.error_msg)

  • Create a SparkSql job.

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
    		 
    # coding: utf-8

import os
from huaweicloudsdkcore.auth.credentials import BasicCredentials
from huaweicloudsdkmrs.v2.region.mrs_region import MrsRegion
from huaweicloudsdkcore.exceptions import exceptions
from huaweicloudsdkmrs.v2 import *

if __name__ == "__main__":
    # The AK and SK used for authentication are hard-coded or stored in plaintext, which has great security risks. It is recommended that the AK and SK be stored in ciphertext in configuration files or environment variables and decrypted during use to ensure security.
    # In this example, AK and SK are stored in environment variables for authentication. Before running this example, set environment variables CLOUD_SDK_AK and CLOUD_SDK_SK in the local environment
    ak = os.environ["CLOUD_SDK_AK"]
    sk = os.environ["CLOUD_SDK_SK"]
    projectId = "{project_id}"

    credentials = BasicCredentials(ak, sk, projectId)

    client = MrsClient.new_builder() \
        .with_credentials(credentials) \
        .with_region(MrsRegion.value_of("<YOUR REGION>")) \
        .build()

    try:
        request = CreateExecuteJobRequest()
        request.cluster_id = "{cluster_id}"
        listPropertiesbody = {
            "fs.obs.access.key": "xxx",
            "fs.obs.secret.key": "yyy"
        }
        listArgumentsbody = [
            "op-key1",
            "op-value1",
            "op-key2",
            "op-value2",
            "create table student_info3 (id string,name string,gender string,age int,addr string);"
        ]
        request.body = JobExecution(
            properties=listPropertiesbody,
            arguments=listArgumentsbody,
            job_name="SparkSqlTest",
            job_type="SparkSql"
        )
        response = client.create_execute_job(request)
        print(response)
    except exceptions.ClientRequestException as e:
        print(e.status_code)
        print(e.request_id)
        print(e.error_code)
        print(e.error_msg)

  • Create a Flink job.

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
    		 
    # coding: utf-8

import os
from huaweicloudsdkcore.auth.credentials import BasicCredentials
from huaweicloudsdkmrs.v2.region.mrs_region import MrsRegion
from huaweicloudsdkcore.exceptions import exceptions
from huaweicloudsdkmrs.v2 import *

if __name__ == "__main__":
    # The AK and SK used for authentication are hard-coded or stored in plaintext, which has great security risks. It is recommended that the AK and SK be stored in ciphertext in configuration files or environment variables and decrypted during use to ensure security.
    # In this example, AK and SK are stored in environment variables for authentication. Before running this example, set environment variables CLOUD_SDK_AK and CLOUD_SDK_SK in the local environment
    ak = os.environ["CLOUD_SDK_AK"]
    sk = os.environ["CLOUD_SDK_SK"]
    projectId = "{project_id}"

    credentials = BasicCredentials(ak, sk, projectId)

    client = MrsClient.new_builder() \
        .with_credentials(credentials) \
        .with_region(MrsRegion.value_of("<YOUR REGION>")) \
        .build()

    try:
        request = CreateExecuteJobRequest()
        request.cluster_id = "{cluster_id}"
        listPropertiesbody = {
            "fs.obs.endpoint": "obs endpoint",
            "fs.obs.access.key": "xxx",
            "fs.obs.secret.key": "yyy"
        }
        listArgumentsbody = [
            "run",
            "-d",
            "-ynm",
            "testExcutorejobhdfsbatch",
            "-m",
            "yarn-cluster",
            "hdfs://test/examples/batch/WordCount.jar"
        ]
        request.body = JobExecution(
            properties=listPropertiesbody,
            arguments=listArgumentsbody,
            job_name="flinkTest",
            job_type="Flink"
        )
        response = client.create_execute_job(request)
        print(response)
    except exceptions.ClientRequestException as e:
        print(e.status_code)
        print(e.request_id)
        print(e.error_code)
        print(e.error_msg)

  • Create a SparkPython job (Jobs of this type will be converted to SparkSubmit jobs for submission. The job type is displayed as SparkSubmit on the MRS console. Select SparkSubmit when you call an API to query the job list.)

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
    		 
    # coding: utf-8

import os
from huaweicloudsdkcore.auth.credentials import BasicCredentials
from huaweicloudsdkmrs.v2.region.mrs_region import MrsRegion
from huaweicloudsdkcore.exceptions import exceptions
from huaweicloudsdkmrs.v2 import *

if __name__ == "__main__":
    # The AK and SK used for authentication are hard-coded or stored in plaintext, which has great security risks. It is recommended that the AK and SK be stored in ciphertext in configuration files or environment variables and decrypted during use to ensure security.
    # In this example, AK and SK are stored in environment variables for authentication. Before running this example, set environment variables CLOUD_SDK_AK and CLOUD_SDK_SK in the local environment
    ak = os.environ["CLOUD_SDK_AK"]
    sk = os.environ["CLOUD_SDK_SK"]
    projectId = "{project_id}"

    credentials = BasicCredentials(ak, sk, projectId)

    client = MrsClient.new_builder() \
        .with_credentials(credentials) \
        .with_region(MrsRegion.value_of("<YOUR REGION>")) \
        .build()

    try:
        request = CreateExecuteJobRequest()
        request.cluster_id = "{cluster_id}"
        listPropertiesbody = {
            "fs.obs.access.key": "xxx",
            "fs.obs.secret.key": "yyy"
        }
        listArgumentsbody = [
            "--master",
            "yarn",
            "--deploy-mode",
            "cluster",
            "--py-files",
            "obs://obs-test/a.py",
            "--conf",
            "spark.yarn.appMasterEnv.PYTHONPATH=/tmp:$PYTHONPATH",
            "--conf",
            "spark.yarn.appMasterEnv.aaa=aaaa",
            "--conf",
            "spark.executorEnv.aaa=executoraaa",
            "--properties-file",
            "obs://obs-test/test-spark.conf",
            "obs://obs-test/pi.py",
            "100000"
        ]
        request.body = JobExecution(
            properties=listPropertiesbody,
            arguments=listArgumentsbody,
            job_name="SparkPythonTest",
            job_type="SparkPython"
        )
        response = client.create_execute_job(request)
        print(response)
    except exceptions.ClientRequestException as e:
        print(e.status_code)
        print(e.request_id)
        print(e.error_code)
        print(e.error_msg)

  • Create a MapReduce job.

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
    		 
    package main

import (
	"fmt"
	"github.com/huaweicloud/huaweicloud-sdk-go-v3/core/auth/basic"
    mrs "github.com/huaweicloud/huaweicloud-sdk-go-v3/services/mrs/v2"
	"github.com/huaweicloud/huaweicloud-sdk-go-v3/services/mrs/v2/model"
    region "github.com/huaweicloud/huaweicloud-sdk-go-v3/services/mrs/v2/region"
)

func main() {
    // The AK and SK used for authentication are hard-coded or stored in plaintext, which has great security risks. It is recommended that the AK and SK be stored in ciphertext in configuration files or environment variables and decrypted during use to ensure security.
    // In this example, AK and SK are stored in environment variables for authentication. Before running this example, set environment variables CLOUD_SDK_AK and CLOUD_SDK_SK in the local environment
    ak := os.Getenv("CLOUD_SDK_AK")
    sk := os.Getenv("CLOUD_SDK_SK")
    projectId := "{project_id}"

    auth, err := basic.NewCredentialsBuilder().
        WithAk(ak).
        WithSk(sk).
        WithProjectId(projectId).
        SafeBuild()

    if err != nil {
        fmt.Println(err)
        return
    }

    hcClient, err := mrs.MrsClientBuilder().
         WithRegion(region.ValueOf("<YOUR REGION>")).
         WithCredential(auth).
         SafeBuild()


    if err != nil {
        fmt.Println(err)
        return
    }

    client := mrs.NewMrsClient(hcClient)

    request := &model.CreateExecuteJobRequest{}
	request.ClusterId = "{cluster_id}"
	var listPropertiesbody = map[string]string{
        "fs.obs.endpoint": "obs endpoint",
        "fs.obs.access.key": "xxx",
        "fs.obs.secret.key": "yyy",
    }
	var listArgumentsbody = []string{
        "obs://obs-test/program/hadoop-mapreduce-examples-x.x.x.jar",
	    "wordcount",
	    "obs://obs-test/input/",
	    "obs://obs-test/job/mapreduce/output",
    }
	request.Body = &model.JobExecution{
		Properties: listPropertiesbody,
		Arguments: &listArgumentsbody,
		JobName: "MapReduceTest",
		JobType: "MapReduce",
	}
	response, err := client.CreateExecuteJob(request)
	if err == nil {
        fmt.Printf("%+v\n", response)
    } else {
        fmt.Println(err)
    }
}

  • Create a SparkSubmit job.

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
    		 
    package main

import (
	"fmt"
	"github.com/huaweicloud/huaweicloud-sdk-go-v3/core/auth/basic"
    mrs "github.com/huaweicloud/huaweicloud-sdk-go-v3/services/mrs/v2"
	"github.com/huaweicloud/huaweicloud-sdk-go-v3/services/mrs/v2/model"
    region "github.com/huaweicloud/huaweicloud-sdk-go-v3/services/mrs/v2/region"
)

func main() {
    // The AK and SK used for authentication are hard-coded or stored in plaintext, which has great security risks. It is recommended that the AK and SK be stored in ciphertext in configuration files or environment variables and decrypted during use to ensure security.
    // In this example, AK and SK are stored in environment variables for authentication. Before running this example, set environment variables CLOUD_SDK_AK and CLOUD_SDK_SK in the local environment
    ak := os.Getenv("CLOUD_SDK_AK")
    sk := os.Getenv("CLOUD_SDK_SK")
    projectId := "{project_id}"

    auth, err := basic.NewCredentialsBuilder().
        WithAk(ak).
        WithSk(sk).
        WithProjectId(projectId).
        SafeBuild()

    if err != nil {
        fmt.Println(err)
        return
    }

    hcClient, err := mrs.MrsClientBuilder().
         WithRegion(region.ValueOf("<YOUR REGION>")).
         WithCredential(auth).
         SafeBuild()


    if err != nil {
        fmt.Println(err)
        return
    }

    client := mrs.NewMrsClient(hcClient)

    request := &model.CreateExecuteJobRequest{}
	request.ClusterId = "{cluster_id}"
	var listPropertiesbody = map[string]string{
        "fs.obs.access.key": "xxx",
        "fs.obs.secret.key": "yyy",
    }
	var listArgumentsbody = []string{
        "--master",
	    "yarn",
	    "--deploy-mode",
	    "cluster",
	    "--py-files",
	    "obs://obs-test/a.py",
	    "--conf",
	    "spark.yarn.appMasterEnv.PYTHONPATH=/tmp:$PYTHONPATH",
	    "--conf",
	    "spark.yarn.appMasterEnv.aaa=aaaa",
	    "--conf",
	    "spark.executorEnv.aaa=executoraaa",
	    "--properties-file",
	    "obs://obs-test/test-spark.conf",
	    "obs://obs-test/pi.py",
	    "100000",
    }
	request.Body = &model.JobExecution{
		Properties: listPropertiesbody,
		Arguments: &listArgumentsbody,
		JobName: "SparkSubmitTest",
		JobType: "SparkSubmit",
	}
	response, err := client.CreateExecuteJob(request)
	if err == nil {
        fmt.Printf("%+v\n", response)
    } else {
        fmt.Println(err)
    }
}

  • Create a HiveScript job.

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
    		 
    package main

import (
	"fmt"
	"github.com/huaweicloud/huaweicloud-sdk-go-v3/core/auth/basic"
    mrs "github.com/huaweicloud/huaweicloud-sdk-go-v3/services/mrs/v2"
	"github.com/huaweicloud/huaweicloud-sdk-go-v3/services/mrs/v2/model"
    region "github.com/huaweicloud/huaweicloud-sdk-go-v3/services/mrs/v2/region"
)

func main() {
    // The AK and SK used for authentication are hard-coded or stored in plaintext, which has great security risks. It is recommended that the AK and SK be stored in ciphertext in configuration files or environment variables and decrypted during use to ensure security.
    // In this example, AK and SK are stored in environment variables for authentication. Before running this example, set environment variables CLOUD_SDK_AK and CLOUD_SDK_SK in the local environment
    ak := os.Getenv("CLOUD_SDK_AK")
    sk := os.Getenv("CLOUD_SDK_SK")
    projectId := "{project_id}"

    auth, err := basic.NewCredentialsBuilder().
        WithAk(ak).
        WithSk(sk).
        WithProjectId(projectId).
        SafeBuild()

    if err != nil {
        fmt.Println(err)
        return
    }

    hcClient, err := mrs.MrsClientBuilder().
         WithRegion(region.ValueOf("<YOUR REGION>")).
         WithCredential(auth).
         SafeBuild()


    if err != nil {
        fmt.Println(err)
        return
    }

    client := mrs.NewMrsClient(hcClient)

    request := &model.CreateExecuteJobRequest{}
	request.ClusterId = "{cluster_id}"
	var listPropertiesbody = map[string]string{
        "fs.obs.endpoint": "obs endpoint",
        "fs.obs.access.key": "xxx",
        "fs.obs.secret.key": "yyy",
    }
	var listArgumentsbody = []string{
        "obs://obs-test/sql/test_script.sql",
    }
	request.Body = &model.JobExecution{
		Properties: listPropertiesbody,
		Arguments: &listArgumentsbody,
		JobName: "HiveScriptTest",
		JobType: "HiveScript",
	}
	response, err := client.CreateExecuteJob(request)
	if err == nil {
        fmt.Printf("%+v\n", response)
    } else {
        fmt.Println(err)
    }
}

  • Create a HiveSql job.

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
    		 
    package main

import (
	"fmt"
	"github.com/huaweicloud/huaweicloud-sdk-go-v3/core/auth/basic"
    mrs "github.com/huaweicloud/huaweicloud-sdk-go-v3/services/mrs/v2"
	"github.com/huaweicloud/huaweicloud-sdk-go-v3/services/mrs/v2/model"
    region "github.com/huaweicloud/huaweicloud-sdk-go-v3/services/mrs/v2/region"
)

func main() {
    // The AK and SK used for authentication are hard-coded or stored in plaintext, which has great security risks. It is recommended that the AK and SK be stored in ciphertext in configuration files or environment variables and decrypted during use to ensure security.
    // In this example, AK and SK are stored in environment variables for authentication. Before running this example, set environment variables CLOUD_SDK_AK and CLOUD_SDK_SK in the local environment
    ak := os.Getenv("CLOUD_SDK_AK")
    sk := os.Getenv("CLOUD_SDK_SK")
    projectId := "{project_id}"

    auth, err := basic.NewCredentialsBuilder().
        WithAk(ak).
        WithSk(sk).
        WithProjectId(projectId).
        SafeBuild()

    if err != nil {
        fmt.Println(err)
        return
    }

    hcClient, err := mrs.MrsClientBuilder().
         WithRegion(region.ValueOf("<YOUR REGION>")).
         WithCredential(auth).
         SafeBuild()


    if err != nil {
        fmt.Println(err)
        return
    }

    client := mrs.NewMrsClient(hcClient)

    request := &model.CreateExecuteJobRequest{}
	request.ClusterId = "{cluster_id}"
	var listPropertiesbody = map[string]string{
        "fs.obs.endpoint": "obs endpoint",
        "fs.obs.access.key": "xxx",
        "fs.obs.secret.key": "yyy",
    }
	var listArgumentsbody = []string{
        "DROP TABLE IF EXISTS src_wordcount;
    create external table src_wordcount(line string) row format delimited fields terminated by "\n" stored as textfile location "obs://donotdel-gxc/input/";
    insert into src_wordcount values("v1")",
    }
	request.Body = &model.JobExecution{
		Properties: listPropertiesbody,
		Arguments: &listArgumentsbody,
		JobName: "HiveSqlTest",
		JobType: "HiveSql",
	}
	response, err := client.CreateExecuteJob(request)
	if err == nil {
        fmt.Printf("%+v\n", response)
    } else {
        fmt.Println(err)
    }
}

  • Create a DistCp job.

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
    		 
    package main

import (
	"fmt"
	"github.com/huaweicloud/huaweicloud-sdk-go-v3/core/auth/basic"
    mrs "github.com/huaweicloud/huaweicloud-sdk-go-v3/services/mrs/v2"
	"github.com/huaweicloud/huaweicloud-sdk-go-v3/services/mrs/v2/model"
    region "github.com/huaweicloud/huaweicloud-sdk-go-v3/services/mrs/v2/region"
)

func main() {
    // The AK and SK used for authentication are hard-coded or stored in plaintext, which has great security risks. It is recommended that the AK and SK be stored in ciphertext in configuration files or environment variables and decrypted during use to ensure security.
    // In this example, AK and SK are stored in environment variables for authentication. Before running this example, set environment variables CLOUD_SDK_AK and CLOUD_SDK_SK in the local environment
    ak := os.Getenv("CLOUD_SDK_AK")
    sk := os.Getenv("CLOUD_SDK_SK")
    projectId := "{project_id}"

    auth, err := basic.NewCredentialsBuilder().
        WithAk(ak).
        WithSk(sk).
        WithProjectId(projectId).
        SafeBuild()

    if err != nil {
        fmt.Println(err)
        return
    }

    hcClient, err := mrs.MrsClientBuilder().
         WithRegion(region.ValueOf("<YOUR REGION>")).
         WithCredential(auth).
         SafeBuild()


    if err != nil {
        fmt.Println(err)
        return
    }

    client := mrs.NewMrsClient(hcClient)

    request := &model.CreateExecuteJobRequest{}
	request.ClusterId = "{cluster_id}"
	var listPropertiesbody = map[string]string{
        "fs.obs.endpoint": "obs endpoint",
        "fs.obs.access.key": "xxx",
        "fs.obs.secret.key": "yyy",
    }
	var listArgumentsbody = []string{
        "obs://obs-test/DistcpJob/",
	    "/user/test/sparksql/",
    }
	request.Body = &model.JobExecution{
		Properties: listPropertiesbody,
		Arguments: &listArgumentsbody,
		JobName: "DistCpTest",
		JobType: "DistCp",
	}
	response, err := client.CreateExecuteJob(request)
	if err == nil {
        fmt.Printf("%+v\n", response)
    } else {
        fmt.Println(err)
    }
}

  • Create a SparkScript job.

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
    		 
    package main

import (
	"fmt"
	"github.com/huaweicloud/huaweicloud-sdk-go-v3/core/auth/basic"
    mrs "github.com/huaweicloud/huaweicloud-sdk-go-v3/services/mrs/v2"
	"github.com/huaweicloud/huaweicloud-sdk-go-v3/services/mrs/v2/model"
    region "github.com/huaweicloud/huaweicloud-sdk-go-v3/services/mrs/v2/region"
)

func main() {
    // The AK and SK used for authentication are hard-coded or stored in plaintext, which has great security risks. It is recommended that the AK and SK be stored in ciphertext in configuration files or environment variables and decrypted during use to ensure security.
    // In this example, AK and SK are stored in environment variables for authentication. Before running this example, set environment variables CLOUD_SDK_AK and CLOUD_SDK_SK in the local environment
    ak := os.Getenv("CLOUD_SDK_AK")
    sk := os.Getenv("CLOUD_SDK_SK")
    projectId := "{project_id}"

    auth, err := basic.NewCredentialsBuilder().
        WithAk(ak).
        WithSk(sk).
        WithProjectId(projectId).
        SafeBuild()

    if err != nil {
        fmt.Println(err)
        return
    }

    hcClient, err := mrs.MrsClientBuilder().
         WithRegion(region.ValueOf("<YOUR REGION>")).
         WithCredential(auth).
         SafeBuild()


    if err != nil {
        fmt.Println(err)
        return
    }

    client := mrs.NewMrsClient(hcClient)

    request := &model.CreateExecuteJobRequest{}
	request.ClusterId = "{cluster_id}"
	var listPropertiesbody = map[string]string{
        "fs.obs.access.key": "xxx",
        "fs.obs.secret.key": "yyy",
    }
	var listArgumentsbody = []string{
        "op-key1",
	    "op-value1",
	    "op-key2",
	    "op-value2",
	    "obs://obs-test/sql/test_script.sql",
    }
	request.Body = &model.JobExecution{
		Properties: listPropertiesbody,
		Arguments: &listArgumentsbody,
		JobName: "SparkScriptTest",
		JobType: "SparkScript",
	}
	response, err := client.CreateExecuteJob(request)
	if err == nil {
        fmt.Printf("%+v\n", response)
    } else {
        fmt.Println(err)
    }
}

  • Create a SparkSql job.

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
    		 
    package main

import (
	"fmt"
	"github.com/huaweicloud/huaweicloud-sdk-go-v3/core/auth/basic"
    mrs "github.com/huaweicloud/huaweicloud-sdk-go-v3/services/mrs/v2"
	"github.com/huaweicloud/huaweicloud-sdk-go-v3/services/mrs/v2/model"
    region "github.com/huaweicloud/huaweicloud-sdk-go-v3/services/mrs/v2/region"
)

func main() {
    // The AK and SK used for authentication are hard-coded or stored in plaintext, which has great security risks. It is recommended that the AK and SK be stored in ciphertext in configuration files or environment variables and decrypted during use to ensure security.
    // In this example, AK and SK are stored in environment variables for authentication. Before running this example, set environment variables CLOUD_SDK_AK and CLOUD_SDK_SK in the local environment
    ak := os.Getenv("CLOUD_SDK_AK")
    sk := os.Getenv("CLOUD_SDK_SK")
    projectId := "{project_id}"

    auth, err := basic.NewCredentialsBuilder().
        WithAk(ak).
        WithSk(sk).
        WithProjectId(projectId).
        SafeBuild()

    if err != nil {
        fmt.Println(err)
        return
    }

    hcClient, err := mrs.MrsClientBuilder().
         WithRegion(region.ValueOf("<YOUR REGION>")).
         WithCredential(auth).
         SafeBuild()


    if err != nil {
        fmt.Println(err)
        return
    }

    client := mrs.NewMrsClient(hcClient)

    request := &model.CreateExecuteJobRequest{}
	request.ClusterId = "{cluster_id}"
	var listPropertiesbody = map[string]string{
        "fs.obs.access.key": "xxx",
        "fs.obs.secret.key": "yyy",
    }
	var listArgumentsbody = []string{
        "op-key1",
	    "op-value1",
	    "op-key2",
	    "op-value2",
	    "create table student_info3 (id string,name string,gender string,age int,addr string);",
    }
	request.Body = &model.JobExecution{
		Properties: listPropertiesbody,
		Arguments: &listArgumentsbody,
		JobName: "SparkSqlTest",
		JobType: "SparkSql",
	}
	response, err := client.CreateExecuteJob(request)
	if err == nil {
        fmt.Printf("%+v\n", response)
    } else {
        fmt.Println(err)
    }
}

  • Create a Flink job.

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
    		 
    package main

import (
	"fmt"
	"github.com/huaweicloud/huaweicloud-sdk-go-v3/core/auth/basic"
    mrs "github.com/huaweicloud/huaweicloud-sdk-go-v3/services/mrs/v2"
	"github.com/huaweicloud/huaweicloud-sdk-go-v3/services/mrs/v2/model"
    region "github.com/huaweicloud/huaweicloud-sdk-go-v3/services/mrs/v2/region"
)

func main() {
    // The AK and SK used for authentication are hard-coded or stored in plaintext, which has great security risks. It is recommended that the AK and SK be stored in ciphertext in configuration files or environment variables and decrypted during use to ensure security.
    // In this example, AK and SK are stored in environment variables for authentication. Before running this example, set environment variables CLOUD_SDK_AK and CLOUD_SDK_SK in the local environment
    ak := os.Getenv("CLOUD_SDK_AK")
    sk := os.Getenv("CLOUD_SDK_SK")
    projectId := "{project_id}"

    auth, err := basic.NewCredentialsBuilder().
        WithAk(ak).
        WithSk(sk).
        WithProjectId(projectId).
        SafeBuild()

    if err != nil {
        fmt.Println(err)
        return
    }

    hcClient, err := mrs.MrsClientBuilder().
         WithRegion(region.ValueOf("<YOUR REGION>")).
         WithCredential(auth).
         SafeBuild()


    if err != nil {
        fmt.Println(err)
        return
    }

    client := mrs.NewMrsClient(hcClient)

    request := &model.CreateExecuteJobRequest{}
	request.ClusterId = "{cluster_id}"
	var listPropertiesbody = map[string]string{
        "fs.obs.endpoint": "obs endpoint",
        "fs.obs.access.key": "xxx",
        "fs.obs.secret.key": "yyy",
    }
	var listArgumentsbody = []string{
        "run",
	    "-d",
	    "-ynm",
	    "testExcutorejobhdfsbatch",
	    "-m",
	    "yarn-cluster",
	    "hdfs://test/examples/batch/WordCount.jar",
    }
	request.Body = &model.JobExecution{
		Properties: listPropertiesbody,
		Arguments: &listArgumentsbody,
		JobName: "flinkTest",
		JobType: "Flink",
	}
	response, err := client.CreateExecuteJob(request)
	if err == nil {
        fmt.Printf("%+v\n", response)
    } else {
        fmt.Println(err)
    }
}

  • Create a SparkPython job (Jobs of this type will be converted to SparkSubmit jobs for submission. The job type is displayed as SparkSubmit on the MRS console. Select SparkSubmit when you call an API to query the job list.)

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
    		 
    package main

import (
	"fmt"
	"github.com/huaweicloud/huaweicloud-sdk-go-v3/core/auth/basic"
    mrs "github.com/huaweicloud/huaweicloud-sdk-go-v3/services/mrs/v2"
	"github.com/huaweicloud/huaweicloud-sdk-go-v3/services/mrs/v2/model"
    region "github.com/huaweicloud/huaweicloud-sdk-go-v3/services/mrs/v2/region"
)

func main() {
    // The AK and SK used for authentication are hard-coded or stored in plaintext, which has great security risks. It is recommended that the AK and SK be stored in ciphertext in configuration files or environment variables and decrypted during use to ensure security.
    // In this example, AK and SK are stored in environment variables for authentication. Before running this example, set environment variables CLOUD_SDK_AK and CLOUD_SDK_SK in the local environment
    ak := os.Getenv("CLOUD_SDK_AK")
    sk := os.Getenv("CLOUD_SDK_SK")
    projectId := "{project_id}"

    auth, err := basic.NewCredentialsBuilder().
        WithAk(ak).
        WithSk(sk).
        WithProjectId(projectId).
        SafeBuild()

    if err != nil {
        fmt.Println(err)
        return
    }

    hcClient, err := mrs.MrsClientBuilder().
         WithRegion(region.ValueOf("<YOUR REGION>")).
         WithCredential(auth).
         SafeBuild()


    if err != nil {
        fmt.Println(err)
        return
    }

    client := mrs.NewMrsClient(hcClient)

    request := &model.CreateExecuteJobRequest{}
	request.ClusterId = "{cluster_id}"
	var listPropertiesbody = map[string]string{
        "fs.obs.access.key": "xxx",
        "fs.obs.secret.key": "yyy",
    }
	var listArgumentsbody = []string{
        "--master",
	    "yarn",
	    "--deploy-mode",
	    "cluster",
	    "--py-files",
	    "obs://obs-test/a.py",
	    "--conf",
	    "spark.yarn.appMasterEnv.PYTHONPATH=/tmp:$PYTHONPATH",
	    "--conf",
	    "spark.yarn.appMasterEnv.aaa=aaaa",
	    "--conf",
	    "spark.executorEnv.aaa=executoraaa",
	    "--properties-file",
	    "obs://obs-test/test-spark.conf",
	    "obs://obs-test/pi.py",
	    "100000",
    }
	request.Body = &model.JobExecution{
		Properties: listPropertiesbody,
		Arguments: &listArgumentsbody,
		JobName: "SparkPythonTest",
		JobType: "SparkPython",
	}
	response, err := client.CreateExecuteJob(request)
	if err == nil {
        fmt.Printf("%+v\n", response)
    } else {
        fmt.Println(err)
    }
}

For SDK sample code of more programming languages, see the Sample Code tab in API Explorer. SDK sample code can be automatically generated.

Status Codes

Status Code

Description

202

Job creation and execution succeeded.

400

Job creation and execution failed.

Error Codes

See Error Codes.

Parent Topic: Job Management APIs