Help Center> SDK Developer Guide> Developer Guide> Python> MRS Python SDK Demo
View PDF

MRS Python SDK Demo

MapReduce Service (MRS) provides users with a complete controllable, enterprise-class, big data cloud service, ensuring the smooth running of big data components, such as Hadoop, Spark, HBase, Kafka, and Storm.

Creating a Cluster and Submitting the Job

You can create a cluster and submit the job using the Python OpenStack SDK based on the following code. After the cluster is created, it will be displayed on the cluster page of the MRS console.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
 
vpc_id = vpc_id
vpc_name = vpc_name
subnet_id = subnet_id
subnet_name = subnet_name
keypair_name = keypair_name

cluster = {
"cluster_name": cluster_name,
"billing_type": billing_type,
"data_center": data_center,
"master_node_num": master_node_num,
"master_node_size": master_node_size,
"core_node_num": core_node_num,
"core_node_size": core_node_size,
"available_zone_id": available_zone_id,
"vpc": vpc,
"vpc_id": vpc_id,
"subnet_id": subnet_id,
"subnet_name": subnet_name,
"cluster_version": cluster_version,
"cluster_type": cluster_type,
"volume_type": volume_type,
"volume_size": volume_size,
"keypair": keypair,
"safe_mode": safe_mode,
"component_list": [{
"component_id": component_id,
"component_name": component_name
}]
}

job = {
"job_type": job_type,
"job_name": job_name,
"jar_path": jar_path,
"arguments": arguments,
"input": input,
"output": output,
"job_log": job_log,
"shutdown_cluster": shutdown_cluster,
"file_action": file_action,
"submit_job_once_cluster_run": submit_job_once_cluster_run,
"hql": hql,
"hive_script_path": hive_script_path
}
cluster = conn.map_reduce.create_cluster_and_run_job(cluster, job)

Adding Clustered Nodes

You can add nodes to a cluster using the Python OpenStack SDK based on the following code by specifying the cluster ID:

1
2
 
expand_node_amount = instances
conn.map_reduce.expand_cluster(id, expand_node_amount)

Querying Cluster Details

You can query details of a cluster using the Python OpenStack SDK based on the following code by specifying the cluster ID:

1
 
cluster = conn.map_reduce.get_cluster(id)

Terminating a Cluster

You can terminate a cluster using the Python OpenStack SDK based on the following code by specifying the cluster ID:

1
 
conn.map_reduce.delete_cluster(id)

Adding and Executing a Job

You can add a job and execute using the Python OpenStack SDK based on the following code. After the job is created, it will be displayed on the job page of the MRS console.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
 
exe = {
"job_type": job_type,
"job_name": job_name,
"cluster_id": cluster_id,
"jar_path": jar_path,
"arguments": arguments,
"input": input,
"output": output,
"job_log": job_log,
"file_action": file_action,
"hql": hql,
"hive_script_path": hive_script_path
}
job_exe = conn.map_reduce.exe_job (**exe)

Querying the Job Exe Object List

You can query job exe object list using the Python OpenStack SDK based on the following code:

1
2
3
4
5
6
7
8
 
query = {
"cluster_id": cluster_id,
"job_name": job_name,
"page_size": page_size,
"current_page": current_page,
"state": state
}
executions = list(conn.map_reduce.job_exes(**query))

Querying Details of a Job Exe Object

You can query details of a job exe object using the Python OpenStack SDK based on the following code by specifying the object ID:

1
 
conn.map_reduce.get_job_exe(id)

Deleting a Job Execution Object

You can delete a job execution object using the Python OpenStack SDK based on the following code by specifying the object ID:

1
 
conn.map_reduce.delete_job_execution(id)
Parent topic: Python