Creating a Cluster and Submitting a Job

project_id

Yes

String

The project ID. For details about how to obtain the project ID, see Obtaining a Project ID.

is_dec_project

No

Boolean

Whether the cluster is specific for the DeC. The default value is false.

cluster_version

Yes

String

The cluster version. Example: MRS 3.1.0.

cluster_name

Yes

String

The cluster name, which must be unique.

A cluster name can contain only 1 to 64 characters. Only letters, numbers, hyphens (-), and underscores (_) are allowed.

cluster_type

Yes

String

The cluster type. The options are as follows:

• ANALYSIS: analysis cluster
• STREAMING: streaming cluster
• MIXED: hybrid cluster
• CUSTOM: custom cluster, which is supported only by MRS 3.x.

charge_info

No

ChargeInfo object

The billing type. For details, see Table 7.

region

Yes

String

Information about the region where the cluster is located. For details, see Endpoints.

vpc_name

Yes

String

The name of the VPC where the subnet is located. Obtain the VPC name by performing the following operations on the VPC management console:

1. Log in to the VPC management console.
2. Choose Virtual Private Cloud > My VPCs. On the Virtual Private Cloud page, obtain the VPC name from the list.

subnet_id

No

String

The subnet ID. Obtain the subnet ID by performing the following operations on the VPC management console:

1. Log in to the VPC management console.
2. Choose Virtual Private Cloud > My VPCs.
3. Locate the row containing the target VPC and click the number in the Subnets column to view the subnet information.
4. Click the subnet name to obtain the network ID. At least one of subnet_id and subnet_name must be configured. If the two parameters are configured but do not match the same subnet, the cluster fails to create. subnet_id is recommended.

subnet_name

Yes

String

The subnet name. Obtain the subnet name by performing the following operations on the VPC management console:

1. Log in to the management console.
2. Choose Virtual Private Cloud > My VPCs.
3. Locate the row containing the target VPC and click the number in the Subnets column to obtain the subnet name. At least one of subnet_id and subnet_name must be configured. If the two parameters are configured but do not match the same subnet, the cluster fails to create. If only subnet_name is configured and subnets with the same name exist in the VPC, the first subnet name in the VPC is used when a cluster is created. subnet_id is recommended.

components

Yes

String

List of component names, which are separated by commas (,). For details about the components that are supported, see "Components Supported by MRS" in Obtaining the MRS Cluster Information.

external_datasources

No

Array of ClusterDataConnectorMap objects

When deploying components such as Hive and Ranger, you can associate data connections and store metadata in associated databases. For details about the parameters, see Table 3.

availability_zone

Yes

String

The AZ name. Multi-AZ clusters are not supported. For details about AZs, see Endpoints.

security_groups_id

No

String

The ID of the security group configured for the cluster.

• If this parameter is left blank, MRS automatically creates a security group whose name starts with mrs_{cluster_name}.
• If this parameter is configured, a fixed security group is used to create a cluster. The transferred ID must be the security group ID owned by the current tenant. The security group must include an inbound rule in which all protocols and all ports are allowed and the source is the IP address of the specified node on the management plane.
• Multiple security group IDs are supported and separated by commas (,).

auto_create_default_security_group

No

Boolean

Whether to create the default security group for the MRS cluster. The default value is false. If this parameter is set to true, the default security group will be created for the cluster regardless of whether security_groups_id is specified.

safe_mode

Yes

String

The running mode of an MRS cluster. The options are as follows:

• SIMPLE: normal cluster. In a normal cluster, Kerberos authentication is disabled, and users can use all functions provided by the cluster.
• KERBEROS: security cluster. In a security cluster, Kerberos authentication is enabled, and common users cannot use the file management and job management functions of an MRS cluster or view cluster resource usage and the job records of Hadoop and Spark. To use more functions, the users must obtain the relevant permissions from the Manager administrator.

manager_admin_password

Yes

String

Password of the MRS Manager administrator. The password must meet the following requirements:

• Must contain 8 to 26 characters.
• Must contain at least four of the following: uppercase letters, lowercase letters, numbers, and special characters (!@$%^-_=+[{}]:,./?), but must not contain spaces.
• Cannot be the username or the username spelled backwards.

login_mode

Yes

String

Node login mode.

• PASSWORD: password-based login. If this value is selected, node_root_password cannot be left blank.
• KEYPAIR: key pair used for login. If this value is selected, node_keypair_name cannot be left blank.

node_root_password

No

String

The password of user root for logging in to a cluster node. A password must meet the following complexity requirements:

• Must be 8 to 26 characters long.
• Must contain at least four of the following: uppercase letters, lowercase letters, numbers, and special characters (!@$%^-_=+[{}]:,./?), but must not contain spaces.
• Cannot be the username or the username spelled backwards.

node_keypair_name

No

String

The name of a key pair. You can use a key pair to log in to a cluster node.

enterprise_project_id

No

String

The enterprise project ID. When you create a cluster, associate the enterprise project ID with the cluster. The default value is 0, indicating the default enterprise project. To obtain the enterprise project ID, see the id value in the enterprise_project field data structure table in "Querying the Enterprise Project List" in Enterprise Management API Reference.

eip_address

No

String

EIP bound to an MRS cluster, which can be used to access MRS Manager. The EIP must have been created and must be in the same region as the cluster.

eip_id

No

String

ID of the bound EIP. This parameter is mandatory when eip_address is configured. To obtain the EIP ID, log in to the VPC console, choose Network > Elastic IP and Bandwidth > Elastic IP, click the EIP to be bound, and obtain the ID in the Basic Information area.

mrs_ecs_default_agency

No

String

Name of the agency bound to a cluster node by default. The value is fixed to MRS_ECS_DEFAULT_AGENCY. An agency allows ECS or BMS to manage MRS resources. You can configure an agency of the ECS type to automatically obtain the AK/SK to access OBS. The MRS_ECS_DEFAULT_AGENCY agency has the OBS OperateAccess permission of OBS and the CES FullAccess (for users who have enabled fine-grained policies), CES Administrator, and KMS Administrator permissions in the region where the cluster is located.

template_id

No

String

The template used for node deployment when the cluster type is CUSTOM.

• mgmt_control_combined_v2: template for jointly deploying management and controller nodes. The management and controller roles are co-deployed on the master node, and data instances are deployed in the same node group. This deployment model applies to scenarios where there are less than 100 nodes, reducing costs.
• mgmt_control_separated_v2: The management and control roles are deployed on different master nodes, and data instances are deployed in the same node group. This deployment model applies to a cluster with 100 to 500 nodes and delivers better performance in high-concurrency load scenarios.
• mgmt_control_data_separated_v2: The management and control roles are deployed on different master nodes, and data instances are deployed in different node groups. This deployment model applies to a cluster with more than 500 nodes. Components can be deployed separately, which can be used for a larger cluster scale.

tags

No

Array of Tag objects

The cluster tags. One cluster can have a maximum of 10 tags. The tag name (key) must be unique. For details about the parameters, see Table 4.

log_collection

No

Integer

Whether to collect logs when cluster creation fails. The default value is 1, indicating that OBS buckets will be created and only used to collect logs that record MRS cluster creation failures. Enumerated values:

• 0: Do not collect logs.
• 1: Collect logs.

node_groups

Yes

Array of NodeGroupV2 objects

Information about the node groups that form the cluster. For details about the parameters, see Table 5.

bootstrap_scripts

No

Array of BootstrapScript objects

The bootstrap action script. For details about the parameters, see Table 13.

log_uri

No

String

The OBS path to which cluster logs are dumped. After the log dump function is enabled, the read and write permissions on the OBS path are required to upload logs. Configure the default agency MRS_ECS_DEFULT_AGENCY or customize an agency with the read and write permissions on the OBS path. For details, see Configuring a Storage-Compute Decoupled Cluster (Agency). This parameter is available only for cluster versions that support dumping cluster logs to OBS.

component_configs

No

Array of ComponentConfig objects

The custom configuration of cluster components. This parameter applies only to cluster versions that support the feature of creating a cluster by customizing component configurations. For details about this parameter, see Table 14.

delete_when_no_steps

No

Boolean

Whether to automatically terminate the cluster after the job is complete. The default value is false.

steps

Yes

Array of StepConfig objects

The job list. For details about this parameter, see Table 16.

map_id

No

Integer

The data connection association ID

connector_id

No

String

The data connection ID

component_name

No

String

The component name

role_type

No

String

The component role type. The options are as follows:

• hive_metastore: Hive Metastore role
• hive_data: Hive role
• hbase_data: HBase role
• ranger_data: Ranger role

source_type

No

String

The data connection type. The options are as follows:

• LOCAL_DB: local metadata
• RDS_POSTGRES: RDS PostgreSQL database
• RDS_MYSQL: RDS MySQL database
• gaussdb-mysql: GaussDB(for MySQL)

cluster_id

No

String

The ID of the associated cluster

status

No

Integer

The data connection status. The options are as follows:

• 0: normal
• 1: in use

key

Yes

String

The tag key

• It contains a maximum of 36 Unicode characters and cannot be an empty string.
• The tag key cannot start or end with spaces or contain non-printable ASCII characters (0–31) and special characters (=*<>\,|/).
• The tag key of a resource must be unique.

value

Yes

String

The value

• The value contains a maximum of 43 Unicode characters and can be an empty string.
• The tag value cannot start or end with spaces or contain non-printable ASCII characters (0–31) and special characters (=*<>\,|/).

group_name

Yes

String

The node group name. The value can contain a maximum of 64 characters, including letters, numbers, and underscores (_). The rules for configuring node groups are as follows:

• master_node_default_group: master node group, which must be included in all cluster types.
• core_node_analysis_group: analysis core node group, which must be included in both analysis and hybrid clusters.
• core_node_streaming_group: streaming core node group, which must be included in both streaming and hybrid clusters.
• task_node_analysis_group: analysis task node group, which can be selected for analysis clusters and hybrid clusters as needed.
• task_node_streaming_group: streaming task node group, which can be selected for streaming clusters and hybrid clusters as needed.
• node_group{x}: node group of a custom cluster. A maximum of nine such node groups can be added for a custom cluster.

node_num

Yes

Integer

The number of nodes. The value ranges from 0 to 500. The maximum number of core and task nodes is 500.

node_size

Yes

String

The instance specifications of a node, for example, c3.4xlarge.2.linux.bigdata. For details about instance specifications, see ECS Specifications Used by MRS and BMS Specifications Used by MRS. You are advised to obtain the specifications supported by the corresponding version in the corresponding region from the cluster creation page on the MRS console.

root_volume

No

Volume object

The system disk information of the node. This parameter is optional for some VMs or the system disk of the BMS and mandatory in other cases. For details about this parameter, see Table 6.

data_volume

No

Volume object

The data disk information. This parameter is mandatory when data_volume_count is not 0. For details about this parameter, see Table 6.

data_volume_count

No

Integer

The number of data disks on a node. The value ranges from 0 to 10.

charge_info

No

ChargeInfo object

The billing type of a node group. The billing types of master and core node groups are the same as those of the cluster. The billing type of the task node group can be different. For details about this parameter, see Table 7.

auto_scaling_policy

No

AutoScalingPolicy object

The auto scaling rule information. For details about this parameter, see Table 8.

assigned_roles

No

Array of strings

This parameter is mandatory when the cluster type is CUSTOM. You can specify the roles deployed in a node group. This parameter is a string array. Each string represents a role expression. Role expression definition:

• If the role is deployed on all nodes in a node group, set this parameter to {role name}, for example, DataNode.
• If the role is deployed on a specified subscript node in the node group: {role name}:{index1},{index2}…,{indexN}, for example, NameNode:1,2. The subscript starts from 1.
• Some roles support multi-instance deployment (that is, multiple instances of the same role are deployed on a node): {role name}[{instance count}], for example, EsNode[9]. For details about the available roles, see Roles and components supported by MRS.

type

Yes

String

The disk type. The options are as follows:

• SATA: common I/O disk
• SAS: high I/O disk
• SSD: ultra-high I/O disk
• GPSSD: general-purpose SSD disk

size

Yes

Integer

The data disk size, in GB. The value ranges from 10 to 32768.

charge_mode

Yes

String

The billing mode. The options are as follows:

• prePaid: the yearly/monthly billing mode. This mode is now supported for the API used to create a cluster, but is not supported for the API used to create a cluster and submit a job.
• postPaid: the pay-per-use billing mode.

period_type

No

String

Subscription period. The value can be:

• month: The cluster is billed by month.
• year: The cluster is billed by year.
• day: The cluster is billed on a pay-per-use basis.

period_num

No

Integer

Number of periods. This parameter is valid and mandatory only when charge_mode is set to prePaid.

• If period_type is set to month, the value ranges from 1 to 9.
• If period_type is set to year, the value ranges from 1 to 3.

is_auto_pay

No

Boolean

Whether the order will be automatically paid. This parameter is available for yearly/monthly mode. By default, the automatic payment is disabled. Available values are as follows:

• true: The system automatically selects available discounts and coupons, and then pays for the order with the account balances. If the automatic payment fails, an order in Pending payment state is generated waiting for manual payment.
• false: The user needs to pay for the bill after using available discounts and coupons.

auto_scaling_enable

Yes

Boolean

Whether to enable the auto scaling policy.

min_capacity

Yes

Integer

The minimum number of nodes reserved in the node group. Value range: [0, 500]

max_capacity

Yes

Integer

The maximum number of nodes in the node group. Value range: [0, 500]

resources_plans

No

Array of ResourcesPlan objects

The resource plan list. If this parameter is left blank, the resource plan is disabled. When auto_scaling_enable is set to true, either this parameter or rules must be configured. For details about this parameter, see Table 9.

rules

No

Array of Rule objects

The list of auto scaling rules. When auto_scaling_enable is set to true, either this parameter or resources_plans must be configured. For details about this parameter, see Table 10.

exec_scripts

No

Array of ScaleScript objects

The list of custom scaling automation scripts. If this parameter is left blank, the automation script is disabled. For details about this parameter, see Table 12.

period_type

Yes

String

The cycle type of a resource plan. Currently, only the following cycle type is supported:

daily

start_time

Yes

String

The start time of a resource plan. The value is in the format of hour:minute, indicating that the time ranges from 00:00 to 23:59.

end_time

Yes

String

The end time of a resource plan. The value is in the same format as that of start_time. The interval between end_time and start_time must be greater than or equal to 30 minutes.

min_capacity

Yes

Integer

The minimum number of reserved nodes in a node group in a resource plan. Value range: [0, 500]

max_capacity

Yes

Integer

The maximum number of reserved nodes in a node group in a resource plan. Value range: [0, 500]

effective_days

No

Array of strings

The effective date of a resource plan. If this parameter is left blank, it indicates that the resource plan takes effect every day. The options are as follows:

MONDAY, TUESDAY, WEDNESDAY, THURSDAY, FRIDAY, SATURDAY, and SUNDAY

name

Yes

String

The name of an auto scaling rule. The name can contain only 1 to 64 characters. Only letters, numbers, hyphens (-), and underscores (_) are allowed. Rule names must be unique in a node group.

description

No

String

The description about an auto scaling rule. It contains a maximum of 1,024 characters.

adjustment_type

Yes

String

The adjustment type of an auto scaling rule. The options are as follows:

• scale_out: cluster scale-out
• scale_in: cluster scale-in

cool_down_minutes

Yes

Integer

The cluster cooling time after an auto scaling rule is triggered, in minutes, during which period no auto scaling operation is performed. The value ranges from 0 to 10080. One week is equal to 10,080 minutes.

scaling_adjustment

Yes

Integer

The number of cluster nodes that can be adjusted at a time. Value range: [1, 100]

trigger

Yes

Trigger object

The condition for triggering a rule. For details about this parameter, see Table 11.

metric_name

Yes

String

The metric name. This triggering condition makes a judgment according to the value of the metric. A metric name contains a maximum of 64 characters.

metric_value

Yes

String

The metric threshold to trigger a rule.

The value must be an integer or a number with two decimal places.

comparison_operator

No

String

The metric judgment logic operator. The options are as follows:

• LT: less than
• GT: greater than
• LTOE: less than or equal to
• GTOE: greater than or equal to

evaluation_periods

Yes

Integer

The number of consecutive five-minute periods, during which a metric threshold is reached. The value ranges from 1 to 288.

name

Yes

String

The name of a custom automation script. Names must be unique in a cluster. The value can contain only numbers, letters, spaces, hyphens (-), and underscores (_) and cannot start with a space. The value can contain 1 to 64 characters.

uri

Yes

String

The path of a custom automation script. Set this parameter to an OBS bucket path or a local VM path.

• OBS bucket path: Enter a script path, for example, obs://XXX/scale.sh.
• Local VM path: Enter a script path. The script path must start with a slash (/) and end with .sh.

parameters

No

String

Parameters of a custom automation script. Multiple parameters are separated by space. The following predefined system parameters can be transferred:

• ${mrs_scale_node_num}: The number of nodes to be added or removed
• ${mrs_scale_type}: The scaling type. The value can be scale_out or scale_in.
• ${mrs_scale_node_hostnames}: Host names of the nodes to be added or removed
• ${mrs_scale_node_ips}: IP addresses of the nodes to be added or removed
• ${mrs_scale_rule_name}: Name of the rule that triggers auto scaling Other user-defined parameters are used in the same way as those of common shell scripts. Parameters are separated by space.

nodes

Yes

Array of strings

The name of the node group where the custom automation script is executed.

active_master

No

Boolean

Whether the custom automation script runs only on the active master node. The default value is false, indicating that the custom automation script can run on all master nodes.

fail_action

Yes

String

Whether to continue executing subsequent scripts and creating a cluster after the custom automation script fails to be executed. Notes:

• You are advised to set this parameter to continue in the commissioning phase so the cluster can continue to be installed and started no matter whether the custom automation script is executed successfully.
• The scale-in operation cannot be undone. fail_action must be set to continue for the scripts that are executed after scale-in. Enumerated values:
• continue: Continue to execute subsequent scripts.
• errorout: Stop the action.

action_stage

Yes

String

The time when a script is executed. Enumerated values:

• before_scale_out: before scale-out
• before_scale_in: before scale-in
• after_scale_out: after scale-out
• after_scale_in: after scale-in

name

Yes

String

The name of a bootstrap action script, which must be unique in a cluster. The value can contain only numbers, letters, spaces, hyphens (-), and underscores (_) and cannot start with a space. The value can contain 1 to 64 characters.

uri

Yes

String

The path of a bootstrap action script. Set this parameter to an OBS bucket path or a local VM path. OBS bucket path: Enter a script path, For example, enter the path of the public sample script provided by MRS. Example: obs://bootstrap/presto/presto-install.sh. If dualroles is installed, the parameter of the presto-install.sh script is dualroles. If worker is installed, the parameter of the presto-install.sh script is worker. Based on the Presto usage habit, you are advised to install dualroles on the active master nodes and worker on the core nodes. Local VM path: Enter a script path. The script path must start with a slash (/) and end with .sh.

parameters

No

String

The bootstrap action script parameters.

nodes

Yes

Array of strings

The name of the node group where the bootstrap action script is executed

active_master

No

Boolean

Whether the bootstrap action script runs only on active master nodes. The default value is false, indicating that the bootstrap action script can run on all master nodes.

fail_action

Yes

String

Whether to continue executing subsequent scripts and creating a cluster after the bootstrap action script fails to execute. The default value is errorout, indicating that the action is stopped. Note: You are advised to set this parameter to continue in the commissioning phase so that the cluster can continue to be installed and started no matter whether the bootstrap action is successful. Enumerated values:

• continue: Continue to execute subsequent scripts.
• errorout: Stop the action.

before_component_start

No

Boolean

The time when the bootstrap action script is executed. Currently, the following two options are available: Before component start and After component start. The default value is false, indicating that the bootstrap action script is executed after the component is started.

start_time

No

Long

The execution time of one bootstrap action script.

state

No

String

The running status of one bootstrap action script. The options are as follows:

• PENDING
• IN_PROGRESS
• SUCCESS
• FAILURE

action_stages

No

Array of strings

Select the time when the bootstrap action script is executed.

• BEFORE_COMPONENT_FIRST_START: before initial component starts
• AFTER_COMPONENT_FIRST_START: after initial component starts
• BEFORE_SCALE_IN: before scale-in
• AFTER_SCALE_IN: after scale-in
• BEFORE_SCALE_OUT: before scale-out
• AFTER_SCALE_OUT: after scale-out

component_name

Yes

String

The component name

configs

No

Array of Config objects

The component configuration item list. For details about this parameter, see Table 15.

key

Yes

String

The configuration name. Only the configuration names displayed on the MRS component configuration page are supported.

value

Yes

String

The configuration value

config_file_name

Yes

String

The configuration file name. Only the file names displayed on the MRS component configuration page are supported.

job_execution

Yes

JobExecution object

The job parameter. For details about this parameter, see Table 17.

job_type

Yes

String

The job type. The options are as follows:

• MapReduce
• SparkSubmit
• SparkPython: Example request of a SparkPython job (Jobs of this type will be converted to SparkSubmit jobs for submission. The job type is displayed as SparkSubmit on the MRS console. Select SparkSubmit when you call an API to query the job list.)
• HiveScript
• HiveSql
• DistCp: imports and exports data.
• SparkScript
• SparkSql
• Flink

job_name

Yes

String

The job name. The value can contain 1 to 64 characters. Only letters, numbers, hyphens (-), and underscores (_) are allowed. Identical job names are allowed but not recommended.

arguments

No

Array of strings

The key parameter for program execution. The parameter is specified by the function of the user's program. MRS is only responsible for loading the parameter. The value can contain a maximum of 150,000 characters. Special characters (;|&>'<$!"\) are not allowed. This parameter can be left blank. Notes:

• If you enter a parameter with sensitive information (such as the login password), the parameter may be exposed in the job details display and log printing. Exercise caution when performing this operation.
• If you need to access files stored in OBS via the path starting with obs:// when submitting a HiveScript or HiveSQL job, search for the core.site.customized.configs parameter on the Hive service configuration page, add the endpoint configuration item (fs.obs.endpoint) of OBS, and set the value to the endpoint corresponding to OBS. For details, see Endpoints.

properties

No

Map<String,String>

The program system parameter. The value can contain a maximum of 2,048 characters. Special characters (;|&>'<$!\\) are not allowed. This parameter can be left blank.

cluster_id

String

The cluster ID, which is returned by the system after the cluster is created.