Help Center/ MapReduce Service/ API Reference (Ankara Region)/ API V2/ Cluster Management APIs/ Creating a Cluster and Submitting a Job

Updated on 2024-11-29 GMT+08:00

View PDF

Creating a Cluster and Submitting a Job

Function

This API is used to create an MRS cluster, submit a job, and terminate the cluster after the job is complete. This API is supported in MRS 1.8.9 or later. Before using this API, you need to obtain the following resource information:

Create or query a VPC and subnet.
Create or query a key pair using an ECS.
Obtain the region information by referring to Endpoints.
Obtain the MRS version and the components supported by the MRS version by referring to Obtaining the MRS Cluster Information.

Constraints

None

URI

POST /v2/{project_id}/run-job-flow

**Table 1** URI parameters
Parameter	Mandatory	Type	Description
project_id	Yes	String	Explanation Project ID. For details about how to obtain the project ID, see Obtaining a Project ID. Constraints N/A Value range The value must consist of 1 to 64 characters. Only letters and digits are allowed. Default value N/A

Request Parameters

**Table 2** Request body parameters
Parameter	Mandatory	Type	Description
is_dec_project	No	Boolean	Explanation Whether the resource is a DeC resource. Constraints N/A Value range true: The resource is a DeC resource. false: The resource is not a DeC resource. Default value false
cluster_version	Yes	String	Explanation Cluster version, for example, MRS 3.3.1-LTS. Constraints N/A Value range N/A Default value N/A
cluster_name	Yes	String	Explanation Cluster name. Constraints N/A Value range The cluster name must globally unique. A cluster name can contain only 1 to 64 characters. Only letters, numbers, hyphens (-), and underscores (_) are allowed. Default value N/A
cluster_type	Yes	String	Explanation The cluster type. Constraints N/A Value range ANALYSIS: analysis cluster STREAMING: streaming cluster MIXED: hybrid cluster CUSTOM: custom cluster, which is supported only by MRS 3.x. Default value N/A
region	Yes	String	Explanation Information about the region where the cluster is located. For details, see Endpoints. Constraints N/A Value range N/A Default value N/A
vpc_name	Yes	String	Explanation The name of the VPC where the subnet is located. Obtain the VPC name by performing the following operations on the VPC management console: Log in to the VPC management console. Choose Virtual Private Cloud > My VPCs. On the Virtual Private Cloud page, obtain the VPC name from the list. Constraints N/A Value range N/A Default value N/A
subnet_id	No	String	Explanation The subnet ID. Obtain the subnet ID by performing the following operations on the VPC management console: Log in to the VPC management console. Choose Virtual Private Cloud > My VPCs. Locate the row containing the target VPC and click the number in the Subnets column to view the subnet information. Click the subnet name to obtain the network ID. Constraints At least one of subnet_id and subnet_name must be configured. If the two parameters are configured but do not match the same subnet, the cluster fails to create. subnet_id is recommended. Value range N/A Default value N/A
subnet_name	Yes	String	Explanation The subnet name. Obtain the subnet name by performing the following operations on the VPC management console: Log in to the management console. Choose Virtual Private Cloud > My VPCs. Locate the row that contains the target VPC and click the number in the Subnets column to obtain the subnet name. Constraints At least one of subnet_id and subnet_name must be configured. If the two parameters are configured but do not match the same subnet, the cluster fails to create. If only subnet_name is configured and subnets with the same name exist in the VPC, the first subnet name in the VPC is used when a cluster is created. subnet_id is recommended. Value range N/A Default value N/A
components	Yes	String	Explanation List of component names, which are separated by commas (,). For details about the components that are supported, see "Components Supported by MRS" in Obtaining the MRS Cluster Information. Constraints N/A Value range N/A Default value N/A
external_datasources	No	Array of ClusterDataConnectorMap objects	Explanation When deploying components such as Hive and Ranger, you can associate data connections and store metadata in associated databases. For details about the parameters, see Table 3. Constraints N/A
availability_zone	Yes	String	Explanation The AZ name. Multi-AZ clusters are not supported. For details about AZs, see Endpoints. Constraints N/A Value range N/A Default value N/A
security_groups_id	No	String	Explanation The ID of the security group configured for the cluster. If this parameter is left blank, MRS automatically creates a security group whose name starts with mrs_{cluster_name}. If this parameter is configured, a fixed security group is used to create a cluster. The transferred ID must be the security group ID owned by the current tenant. The security group must include an inbound rule in which all protocols and all ports are allowed and the source is the IP address of the specified node on the management plane. Multiple security group IDs are supported and separated by commas (,). Constraints N/A Value range N/A Default value N/A
auto_create_default_security_group	No	Boolean	Explanation Whether to create the default security group for the MRS cluster. Constraints If this parameter is set to true, the default security group will be created for the cluster regardless of whether security_groups_id is specified. Value range true: The default security group is created for the MRS cluster. false: The default security group is not created. Default value false
safe_mode	Yes	String	Explanation The running mode of an MRS cluster. Constraints N/A Value range SIMPLE: normal cluster. In a normal cluster, Kerberos authentication is disabled, and users can use all functions provided by the cluster. KERBEROS: security cluster. In a security cluster, Kerberos authentication is enabled, and common users cannot use the file management and job management functions of an MRS cluster or view cluster resource usage and the job records of Hadoop and Spark. To use more functions, the users must obtain the relevant permissions from the Manager administrator. Default value N/A
manager_admin_password	Yes	String	Explanation Password of the MRS Manager administrator. Constraints N/A Value range The value must contain 8 to 26 characters. The value must contain at least four of the following: uppercase letters, lowercase letters, numbers, and special characters (!@$%^-_=+[{}]:,./?), but must not contain spaces. The value cannot be the username or the username spelled backwards. Default value N/A
login_mode	Yes	String	Explanation Node login mode. Constraints N/A Value range PASSWORD: password-based login. If this value is selected, node_root_password cannot be left blank. KEYPAIR: key pair used for login. If this value is selected, node_keypair_name cannot be left blank. Default value N/A
node_root_password	No	String	Explanation The password of user root for logging in to a cluster node. Constraints N/A Value range Must be 8 to 26 characters long. Must contain at least four of the following: uppercase letters, lowercase letters, numbers, and special characters (!@$%^-_=+[{}]:,./?), but must not contain spaces. Cannot be the username or the username spelled backwards. Default value N/A
node_keypair_name	No	String	Explanation The name of a key pair. You can use a key pair to log in to a cluster node. Constraints N/A Value range N/A Default value N/A
enterprise_project_id	No	String	Explanation Enterprise project ID. When you create a cluster, associate the enterprise project ID with the cluster. The default value is 0, indicating the default enterprise project. Constraints N/A Value range N/A Default value The default value is 0, indicating the default enterprise project.
eip_address	No	String	Explanation EIP bound to an MRS cluster, which can be used to access MRS Manager. The EIP must have been created and must be in the same region as the cluster. Constraints N/A Value range N/A Default value N/A
eip_id	No	String	Explanation ID of the bound EIP. Constraints ID of the bound EIP. This parameter is mandatory when eip_address is configured. Value range N/A Default value N/A
mrs_ecs_default_agency	No	String	Explanation Name of the agency bound to a cluster node by default. The value is fixed to MRS_ECS_DEFAULT_AGENCY. An agency allows ECS or BMS to manage MRS resources. You can configure an agency of the ECS type to automatically obtain the AK/SK to access OBS. The MRS_ECS_DEFAULT_AGENCY agency has the OBS OperateAccess permission of OBS and the CES FullAccess (for users who have enabled fine-grained policies), CES Administrator, and KMS Administrator permissions in the region where the cluster is located. Constraints N/A Value range N/A Default value N/A
template_id	No	String	Explanation The template used for node deployment when the cluster type is CUSTOM. mgmt_control_combined_v2: template for jointly deploying management and controller nodes. The management and controller roles are co-deployed on the master node, and data instances are deployed in the same node group. This deployment model applies to scenarios where there are fewer than 100 nodes, reducing costs. mgmt_control_separated_v2: The management and control roles are deployed on different master nodes, and data instances are deployed in the same node group. This deployment model applies to a cluster with 100 to 500 nodes and delivers better performance in high-concurrency load scenarios. mgmt_control_data_separated_v2: The management and control roles are deployed on different master nodes, and data instances are deployed in different node groups. This deployment model applies to a cluster with more than 500 nodes. Components can be deployed separately, which can be used for a larger cluster scale. Constraints N/A Value range N/A Default value N/A
tags	No	Array of Tag objects	Explanation Cluster tag information. For details, see Table 4. Constraints A cluster allows a maximum of 10 tags. A tag name (key) must be unique in a cluster.
log_collection	No	Integer	Explanation Whether to collect logs when cluster creation fails. Constraints N/A Value range 0: Do not create an OBS bucket only for log collection when a cluster fails to be created. 1: Create an OBS bucket only for collect logs when a cluster fails to be created. Default value 1
node_groups	Yes	Array of NodeGroupV2 objects	Explanation Information about the node groups that form the cluster. For details about the parameters, see Table 5. Constraints N/A
bootstrap_scripts	No	Array of BootstrapScript objects	Explanation The bootstrap action script. For details about the parameters, see Table 12. Constraints N/A
log_uri	No	String	Explanation The OBS path to which cluster logs are dumped. After the log dump function is enabled, the read and write permissions on the OBS path are required to upload logs. Configure the default agency MRS_ECS_DEFULT_AGENCY or customize an agency with the read and write permissions on the OBS path. This parameter is available only for cluster versions that support dumping cluster logs to OBS. Constraints N/A Value range N/A Default value N/A
component_configs	No	Array of ComponentConfig objects	Explanation The custom configuration of cluster components. This parameter applies only to cluster versions that support the feature of creating a cluster by customizing component configurations. For details about this parameter, see Table 13. Constraints The number of records cannot exceed 50.
delete_when_no_steps	No	Boolean	Explanation Whether to automatically delete the cluster after the job is complete. Constraints N/A Value range true: The cluster is deleted after the job is complete. false: The cluster is not deleted after the job is complete. Default value false
steps	Yes	Array of StepConfig objects	Explanation The job list. For details about this parameter, see Table 15. Constraints The number of records cannot exceed 255.

**Table 3** ClusterDataConnectorMap
Parameter	Mandatory	Type	Description
map_id	No	Integer	Explanation Data connection association ID Constraints N/A Value range N/A Default value N/A
connector_id	No	String	Explanation Data connection ID Constraints N/A Value range N/A Default value N/A
component_name	No	String	Explanation Component name Constraints N/A Value range N/A Default value N/A
role_type	No	String	Explanation Component role type. Constraints N/A Value range hive_metastore: Hive Metastore role hive_data: Hive role hbase_data: HBase role ranger_data: Ranger role Default value N/A
source_type	No	String	Explanation Data connection type Constraints N/A Value range LOCAL_DB: local metadata RDS_POSTGRES: RDS PostgreSQL database RDS_MYSQL: RDS MySQL database gaussdb-mysql: GaussDB(for MySQL) Default value N/A
cluster_id	No	String	Explanation ID of the associated cluster Constraints N/A Value range The value can contain 1 to 64 characters, including only letters, digits, underscores (_), and hyphens (-). Default value N/A
status	No	Integer	Explanation Data connection status. Constraints N/A Value range 0: normal 1: in use Default value N/A

**Table 4** Tag
Parameter	Mandatory	Type	Description
key	Yes	String	Explanation Tag key. Constraints N/A Value range A tag key can contain letters, digits, spaces, and special characters _.:=+-@, but cannot start or end with a space or start with _sys_. The tag key of a resource must be unique. It can contain a maximum of 128 Unicode characters and cannot be an empty string. Default value N/A
value	Yes	String	Explanation Tag value. Constraints N/A Value range The value can contain letters, digits, spaces, and special characters _.:=+-@, but cannot start or end with a space or start with _sys_. The value can contain a maximum of 255 Unicode characters and can be an empty string. Default value N/A

**Table 5** NodeGroupV2
Parameter	Mandatory	Type	Description
group_name	Yes	String	Explanation Node group name. Constraints N/A Value range The value can contain a maximum of 64 characters, including uppercase and lowercase letters, digits and underscores (_). The rules for configuring node groups are as follows: master_node_default_group: master node group, which must be included in all cluster types. core_node_analysis_group: analysis core node group, which must be included in both analysis and hybrid clusters. core_node_streaming_group: streaming core node group, which must be included in both streaming and hybrid clusters. task_node_analysis_group: analysis task node group, which can be selected for analysis clusters and hybrid clusters as needed. task_node_streaming_group: streaming task node group, which can be selected for streaming clusters and hybrid clusters as needed. node_group{x}: node group of a custom cluster. A maximum of nine such node groups can be added for a custom cluster. Default value N/A
node_num	Yes	Integer	Explanation Number of nodes. Constraints The total number of Core and Task nodes cannot exceed 500. Value range 0-500 Default value N/A
node_size	Yes	String	Explanation The instance specifications of a node, for example, c3.4xlarge.2.linux.bigdata. You are advised to obtain the specifications supported by the corresponding version in the corresponding region from the cluster creation page on the MRS console. Constraints N/A Value range N/A Default value N/A
root_volume	No	Volume object	Explanation The system disk information of the node. This parameter is optional for some VMs or the system disk of the BMS and mandatory in other cases. For details about this parameter, see Table 6. Constraints N/A
data_volume	No	Volume object	Explanation Data disk information. For details about the parameter, see Table 6. Constraints This parameter is mandatory when data_volume_count is not 0.
data_volume_count	No	Integer	Explanation Number of data disks of a node. Constraints N/A Value range 0-20 Default value N/A
auto_scaling_policy	No	AutoScalingPolicy object	Explanation The auto scaling rule information. For details about this parameter, see Table 7. Constraints N/A
assigned_roles	No	Array of strings	Explanation This parameter is mandatory when the cluster type is CUSTOM. You can specify the roles deployed in a node group. This parameter is a string array. Each string represents a role expression. Role expression definition: If the role is deployed on all nodes in a node group, set this parameter to {role name}, for example, DataNode. If the role is deployed on a specified subscript node in the node group: {role name}:{index1},{index2}…,{indexN}, for example, NameNode:1,2. The subscript starts from 1. Some roles support multi-instance deployment (that is, multiple instances of the same role are deployed on a node): {role name}[{instance count}], for example, EsNode[9]. For details about the available roles, see Roles and components supported by MRS. Constraints N/A

**Table 6** Volume
Parameter	Mandatory	Type	Description
type	Yes	String	Explanation Disk type. Constraints N/A Value range SATA: common I/O disk SAS: high I/O disk SSD: ultra-high I/O disk GPSSD: general-purpose SSD disk Default value N/A
size	Yes	Integer	Explanation Data disk size in GB. Constraints N/A Value range 10-32768 Default value N/A

**Table 7** AutoScalingPolicy
Parameter	Mandatory	Type	Description
auto_scaling_enable	Yes	Boolean	Whether to enable the auto scaling policy.
min_capacity	Yes	Integer	The minimum number of nodes reserved in the node group. Value range: [0, 500]
max_capacity	Yes	Integer	The maximum number of nodes in the node group. Value range: [0, 500]
resources_plans	No	Array of ResourcesPlan objects	The resource plan list. If this parameter is left blank, the resource plan is disabled. When auto_scaling_enable is set to true, either this parameter or rules must be configured. For details about this parameter, see Table 8.
rules	No	Array of Rule objects	The list of auto scaling rules. When auto_scaling_enable is set to true, either this parameter or resources_plans must be configured. For details about this parameter, see Table 9.
exec_scripts	No	Array of ScaleScript objects	The list of custom scaling automation scripts. If this parameter is left blank, the automation script is disabled. For details about this parameter, see Table 11.

**Table 8** ResourcesPlan
Parameter	Mandatory	Type	Description
period_type	Yes	String	Explanation Cycle type of a resource plan. This parameter can be set to daily only. Constraints N/A Value range N/A Default value N/A
start_time	Yes	String	Explanation Start time of a resource plan. The value is in the format of hour:minute, indicating that the time ranges from 00:00 to 23:59. Constraints N/A Value range N/A Default value N/A
end_time	Yes	String	Explanation End time of a resource plan. The format is the same as that of start_time. Constraints The value cannot be earlier than the start_time, and the interval between start_time and start_time cannot be less than 30 minutes. Value range N/A Default value N/A
min_capacity	Yes	Integer	Explanation Minimum number of the preserved nodes in a node group in a resource plan. Constraints N/A Value range 0-500 Default value N/A
max_capacity	Yes	Integer	Explanation Maximum number of the preserved nodes in a node group in a resource plan. Constraints N/A Value range 0-500 Default value N/A
effective_days	No	Array of strings	Explanation The effective date of a resource plan. If this parameter is left blank, it indicates that the resource plan takes effect every day. The options are as follows: MONDAY, TUESDAY, WEDNESDAY, THURSDAY, FRIDAY, SATURDAY, and SUNDAY Constraints N/A

**Table 9** Rule
Parameter	Mandatory	Type	Description
name	Yes	String	Explanation Name of an auto scaling rule. Constraints N/A Value range The value can contain 1 to 64 characters, including only letters, digits, underscores (_), and hyphens (-). Rule names must be unique in a node group. Default value N/A
description	No	String	Explanation Description about an auto scaling rule. Constraints N/A Value range The value can contain 0 to 1024 characters. Default value N/A
adjustment_type	Yes	String	Explanation Adjustment type of an auto scaling rule. Constraints N/A Value range scale_out: cluster scale-out scale_in: cluster scale-in Default value N/A
cool_down_minutes	Yes	Integer	Explanation The cluster cooling time after an auto scaling rule is triggered, in minutes, during which period no auto scaling operation is performed. Constraints N/A Value range The value ranges from 0 to 10080. 10080 indicates the number of minutes in a week. Default value N/A
scaling_adjustment	Yes	Integer	Explanation Number of nodes that can be adjusted once. Constraints N/A Value range 1-100 Default value N/A
trigger	Yes	Trigger object	Explanation Condition for triggering a rule. For details about this parameter, see Table 10. Constraints N/A

**Table 10** Trigger
Parameter	Mandatory	Type	Description
metric_name	Yes	String	Explanation Metric name. This triggering condition makes a judgment according to the value of the metric. Constraints N/A Value range The value can contains 0 to 64 characters. Default value N/A
metric_value	Yes	String	Explanation Metric threshold to trigger a rule. The value must be an integer or a number with two decimal places. Constraints N/A Value range Only integers or numbers with two decimal places are allowed. Default value N/A
comparison_operator	No	String	Explanation Metric judgment logic operator. Constraints N/A Value range LT: less than GT: greater than LTOE: less than or equal to GTOE: greater than or equal to Default value N/A
evaluation_periods	Yes	Integer	Explanation Number of consecutive five-minute periods, during which a metric threshold is reached Constraints N/A Value range 1-288 Default value N/A

**Table 11** ScaleScript
Parameter	Mandatory	Type	Description
name	Yes	String	Explanation Names of custom scaling automation scripts. Constraints N/A Value range The names in the same cluster must be unique. The value can contain only numbers, letters, spaces, hyphens (-), and underscores (_) and cannot start with a space. The value can contain 1 to 64 characters. Default value N/A
uri	Yes	String	Explanation Path of a custom automation script. Set this parameter to an OBS bucket path or a local VM path. OBS bucket path: Enter a script path, for example, obs://XXX/scale.sh. Local VM path: Enter a script path. The script path must start with a slash (/) and end with .sh. Constraints N/A Value range N/A Default value N/A
parameters	No	String	Explanation Parameters of a custom automation script. Multiple parameters are separated by space. The following predefined system parameters can be transferred: ${mrs_scale_node_num}: The number of nodes to be added or removed ${mrs_scale_type}: The scaling type. The value can be scale_out or scale_in. ${mrs_scale_node_hostnames}: Host names of the nodes to be added or removed ${mrs_scale_node_ips}: IP addresses of the nodes to be added or removed ${mrs_scale_rule_name}: Name of the rule that triggers auto scaling Other user-defined parameters are used in the same way as those of common shell scripts. Parameters are separated by space. Constraints N/A Value range N/A Default value N/A
nodes	Yes	Array of strings	Explanation Name of the node group where the custom automation script is executed. Constraints N/A
active_master	No	Boolean	Explanation Whether the custom automation script runs only on the active Master node. Constraints N/A Value range true: The custom automation script runs only on the active Master nodes. false: The custom automation script can run on all Master nodes. Default value false
fail_action	Yes	String	Explanation Whether to continue executing subsequent scripts and creating a cluster after the custom automation script fails to be executed. You are advised to set this parameter to continue in the commissioning phase so the cluster can continue to be installed and started no matter whether the custom automation script is executed successfully. Constraints The scale-in operation cannot be undone. fail_action must be set to continue for the scripts that are executed after scale-in. Value range continue: Continue to execute subsequent scripts. errorout: Stop the action. Default value N/A
action_stage	Yes	String	Explanation Time when a script is executed. Constraints N/A Value range before_scale_out: before scale-out before_scale_in: before scale-in after_scale_out: after scale-out after_scale_in: after scale-in Default value N/A

**Table 12** BootstrapScript
Parameter	Mandatory	Type	Description
name	Yes	String	Explanation Name of a bootstrap action script. Constraints N/A Value range The names of bootstrap action scripts in the same cluster must be unique. The value can contain only numbers, letters, spaces, hyphens (-), and underscores (_) and cannot start with a space. The value can contain 1 to 64 characters. Default value N/A
uri	Yes	String	Explanation Path of a bootstrap action script. Set this parameter to an OBS bucket path or a local VM path. OBS bucket path: Enter a script path, For example, enter the path of the public sample script provided by MRS. Example: obs://bootstrap/presto/presto-install.sh. If dualroles is installed, the parameter of the presto-install.sh script is dualroles. If worker is installed, the parameter of the presto-install.sh script is worker. Based on the Presto usage habit, you are advised to install dualroles on the active master nodes and worker on the core nodes. Local VM path: Enter a script path. The script path must start with a slash (/) and end with .sh. Constraints N/A Value range N/A Default value N/A
parameters	No	String	Explanation Bootstrap action script parameters Constraints N/A Value range N/A Default value N/A
nodes	Yes	Array of strings	Explanation Name of the node group where the bootstrap action script is executed Constraints N/A
active_master	No	Boolean	Explanation Whether the bootstrap action script runs only on active master nodes. Constraints N/A Value range true: The bootstrap action script runs only on active Master nodes. false: The bootstrap action script can run on all Master nodes. Default value N/A
fail_action	Yes	String	Explanation Whether to continue executing subsequent scripts and creating a cluster after the bootstrap action script fails to execute. The default value is errorout, indicating that the action is stopped. Note: You are advised to set this parameter to continue in the commissioning phase so that the cluster can continue to be installed and started no matter whether the bootstrap action is successful. Constraints N/A Value range continue: Continue to execute subsequent scripts. errorout: Stop the action. Default value errorout
before_component_start	No	Boolean	Explanation Time when the bootstrap action script is executed. Currently, the following two options are available: Before component start and After component start. Constraints N/A Value range true: The bootstrap action script is executed before the component starts. false: The bootstrap action script is executed after the component starts. Default value false
start_time	No	Long	Explanation Execution time of one boot operation script. Constraints N/A Value range N/A Default value N/A
state	No	String	Explanation Running state of an individual bootstrap action script. Constraints N/A Value range PENDING: The script is suspended. IN_PROGRESS: The script is being processed. SUCCESS FAILURE: The script fails to be executed. Default value N/A
action_stages	No	Array of strings	Explanation Select the time when the bootstrap action script is executed. Constraints Enumerated values: BEFORE_COMPONENT_FIRST_START: before initial component starts AFTER_COMPONENT_FIRST_START: after initial component starts BEFORE_SCALE_IN: before scale-in AFTER_SCALE_IN: after scale-in BEFORE_SCALE_OUT: before scale-out AFTER_SCALE_OUT: after scale-out

**Table 13** ComponentConfig
Parameter	Mandatory	Type	Description
component_name	Yes	String	Explanation Component name Constraints N/A Value range N/A Default value N/A
configs	No	Array of Config objects	Explanation Component configuration item list. For details about this parameter, see Table 14. Constraints The number of records cannot exceed 100.

**Table 14** Config
Parameter	Mandatory	Type	Description
key	Yes	String	Explanation Configuration name. Only the configuration names displayed on the MRS component configuration page are supported. Constraints N/A Value range N/A Default value N/A
value	Yes	String	Explanation Configuration value. Constraints N/A Value range N/A Default value N/A
config_file_name	Yes	String	Explanation Configuration file name. Only the file names displayed on the MRS component configuration page are supported. Constraints N/A Value range N/A Default value N/A

**Table 15** StepConfig
Parameter	Mandatory	Type	Description
job_execution	Yes	JobExecution object	Explanation Job parameter. For details about this parameter, see Table 16. Constraints N/A

**Table 16** JobExecution
Parameter	Mandatory	Type	Description
job_type	Yes	String	Explanation Job type. Constraints N/A Value range MapReduce SparkSubmit SparkPython: Example request of a SparkPython job (Jobs of this type will be converted to SparkSubmit jobs for submission. The job type is displayed as SparkSubmit on the MRS console. Select SparkSubmit when you call an API to query the job list.) HiveScript HiveSql DistCp: imports and exports data. SparkScript SparkSql Flink Default value N/A
job_name	Yes	String	Explanation Job name. Constraints N/A Value range A cluster name can contain only 1 to 64 characters. Only letters, digits, hyphens (-), and underscores (_) are allowed. Identical job names are allowed but not recommended. Default value N/A
arguments	No	Array of strings	Explanation Key parameter for program execution. The parameter is specified by the function of the user's program. MRS is only responsible for loading the parameter. Constraints The value can contain a maximum of 150,000 characters. Special characters (;\|&>'<$!"\) are not allowed. This parameter can be left blank. Note: If you enter a parameter with sensitive information (such as the login password), the parameter may be exposed in the job details display and log printing. Exercise caution when performing this operation. If you need to access files stored in OBS via the path starting with obs:// when submitting a HiveScript or HiveSQL job, search for the core.site.customized.configs parameter on the Hive service configuration page, add the endpoint configuration item (fs.obs.endpoint) of OBS, and set the value to the endpoint corresponding to OBS. For details, see Endpoints.
properties	No	Map<String,String>	Explanation Program system parameter. Constraints The value can contain a maximum of 2,048 characters. Special characters (;\|&>'<$!\\) are not allowed. This parameter can be left blank.

Response Parameters

Status code: 200

**Table 17** Response body parameter
Parameter	Type	Description
cluster_id	String	Explanation Cluster ID, which is returned by the system after the cluster is created. Value range N/A

Example Request

Create an MRS 3.3.1-LTS cluster where the custom management nodes and control nodes are the same nodes and submit a HiveScript job.

POST /v2/{project_id}/run-job-flow

{
  "cluster_version" : "MRS 3.3.1-LTS",
  "cluster_name" : "mrs_heshe_dm",
  "cluster_type" : "CUSTOM",
  "charge_info" : {
    "charge_mode" : "postPaid"
  },
  "region" : "",
  "availability_zone" : "",
  "vpc_name" : "vpc-37cd",
  "subnet_id" : "1f8c5ca6-1f66-4096-bb00-baf175954f6e",
  "subnet_name" : "subnet",
  "components" : "Hadoop,Spark2x,HBase,Hive,Hue,Loader,Kafka,Storm,Flume,Flink,Oozie,Ranger,Tez",
  "safe_mode" : "KERBEROS",
  "manager_admin_password" : "your password",
  "login_mode" : "PASSWORD",
  "node_root_password" : "your password",
  "mrs_ecs_default_agency" : "MRS_ECS_DEFAULT_AGENCY",
  "template_id" : "mgmt_control_combined_v2",
  "log_collection" : 1,
  "tags" : [ {
    "key" : "tag1",
    "value" : "111"
  }, {
    "key" : "tag2",
    "value" : "222"
  } ],
  "node_groups" : [ {
    "group_name" : "master_node_default_group",
    "node_num" : 3,
    "node_size" : "Sit3.4xlarge.4.linux.bigdata",
    "root_volume" : {
      "type" : "SAS",
      "size" : 480
    },
    "data_volume" : {
      "type" : "SAS",
      "size" : 600
    },
    "data_volume_count" : 1,
    "assigned_roles" : [ "OMSServer:1,2", "SlapdServer:1,2", "KerberosServer:1,2", "KerberosAdmin:1,2", "quorumpeer:1,2,3", "NameNode:2,3", "Zkfc:2,3", "JournalNode:1,2,3", "ResourceManager:2,3", "JobHistoryServer:2,3", "DBServer:1,3", "Hue:1,3", "LoaderServer:1,3", "MetaStore:1,2,3", "WebHCat:1,2,3", "HiveServer:1,2,3", "HMaster:2,3", "MonitorServer:1,2", "Nimbus:1,2", "UI:1,2", "JDBCServer2x:1,2,3", "JobHistory2x:2,3", "SparkResource2x:1,2,3", "oozie:2,3", "LoadBalancer:2,3", "TezUI:1,3", "TimelineServer:3", "RangerAdmin:1,2", "UserSync:2", "TagSync:2", "KerberosClient", "SlapdClient", "meta", "HSConsole:2,3", "FlinkResource:1,2,3", "DataNode:1,2,3", "NodeManager:1,2,3", "IndexServer2x:1,2", "ThriftServer:1,2,3", "RegionServer:1,2,3", "ThriftServer1:1,2,3", "RESTServer:1,2,3", "Broker:1,2,3", "Supervisor:1,2,3", "Logviewer:1,2,3", "Flume:1,2,3", "HSBroker:1,2,3" ]
  }, {
    "group_name" : "node_group_1",
    "node_num" : 3,
    "node_size" : "Sit3.4xlarge.4.linux.bigdata",
    "root_volume" : {
      "type" : "SAS",
      "size" : 480
    },
    "data_volume" : {
      "type" : "SAS",
      "size" : 600
    },
    "data_volume_count" : 1,
    "assigned_roles" : [ "DataNode", "NodeManager", "RegionServer", "Flume:1", "Broker", "Supervisor", "Logviewer", "HBaseIndexer", "KerberosClient", "SlapdClient", "meta", "HSBroker:1,2", "ThriftServer", "ThriftServer1", "RESTServer", "FlinkResource" ]
  }, {
    "group_name" : "node_group_2",
    "node_num" : 1,
    "node_size" : "Sit3.4xlarge.4.linux.bigdata",
    "root_volume" : {
      "type" : "SAS",
      "size" : 480
    },
    "data_volume" : {
      "type" : "SAS",
      "size" : 600
    },
    "data_volume_count" : 1,
    "assigned_roles" : [ "NodeManager", "KerberosClient", "SlapdClient", "meta", "FlinkResource" ]
  } ],
  "log_uri" : "obs://bucketTest/logs",
  "delete_when_no_steps" : true,
  "steps" : [ {
    "job_execution" : {
      "job_name" : "import_file",
      "job_type" : "DistCp",
      "arguments" : [ "obs://test/test.sql", "/user/hive/input" ]
    }
  }, {
    "job_execution" : {
      "job_name" : "hive_test",
      "job_type" : "HiveScript",
      "arguments" : [ "obs://test/hive/sql/HiveScript.sql" ]
    }
  } ]
}

Example Response

Status code: 200

Example successful response

{
  "cluster_id" : "da1592c2-bb7e-468d-9ac9-83246e95447a"
}

Status Codes

For details, see Status Codes.

Error Codes

See Error Codes.

Parent topic: Cluster Management APIs

Previous topic: Changing a Cluster Name

Next topic: Scaling Out a Cluster

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot