Creating a Cluster and Executing a Job

VPC

See operation instructions in in the VPC API Reference.

Subnet

See operation instructions in in the VPC API Reference.

Key Pair

See operation instructions in in the ECS API Reference.

Zone

See Endpoints for details about regions and AZs.

Version

Currently, MRS 2.1.1, 3.0.5, and 3.1.0 are supported.

Component

• MRS 3.1.0 supports the following components:
  • An analysis cluster contains the following components: Hadoop, Spark2x, HBase, Hive, Hue, Flink, Oozie, ZooKeeper, Ranger, Tez, Impala, Presto, and Kudu.
  • A streaming cluster contains the following components: Kafka, Flume, ZooKeeper, and Ranger.
  • A hybrid cluster contains the following components: Hadoop, Spark2x, HBase, Hive, Hue, Flink, Oozie, ZooKeeper, Ranger, Tez, Impala, Presto, Kudu, Kafka, and Flume.
  • A custom cluster contains the following components: Hadoop, Spark2x, HBase, Hive, Hue, Kafka, Flume, Flink, Oozie, ZooKeeper, Ranger, Tez, Impala, Presto, ClickHouse, and Kudu.
• MRS 3.0.5 supports the following components:
  • An analysis cluster contains the following components: Hadoop, Spark2x, HBase, Hive, Hue, Loader, Flink, Oozie, ZooKeeper, Ranger, Tez, Impala, Presto, Kudu, and Alluxio.
  • A streaming cluster contains the following components: Kafka, Storm, Flume, ZooKeeper, and Ranger.
  • A hybrid cluster contains the following components: Hadoop, Spark2x, HBase, Hive, Hue, Loader, Flink, Oozie, ZooKeeper, Ranger, Tez, Impala, Presto, Kudu, Alluxio, Kafka, Storm, and Flume.
  • A custom cluster contains the following components: Hadoop, Spark2x, HBase, Hive, Hue, Loader, Kafka, Storm, Flume, Flink, Oozie, ZooKeeper, Ranger, Tez, Impala, Presto, ClickHouse, Kudu, and Alluxio.

cluster_version

Yes

String

The cluster version.

Possible values:

• MRS 3.0.5
• MRS 3.1.0

cluster_name

Yes

String

The cluster name, which must be unique. A cluster name can contain only 1 to 64 characters. Only letters, numbers, hyphens (-), and underscores (_) are allowed.

master_node_num

Yes

Integer

The number of master nodes. If cluster HA is enabled, set this parameter to 2. If cluster HA is disabled, set this parameter to 1. This parameter cannot be set to 1 in MRS 3.x.

core_node_num

Yes

Integer

The number of core nodes.

Value range: 1 to 500

The default maximum number of core nodes is 500. If more than 500 core nodes are required, apply for a higher quota.

billing_type

Yes

Integer

The cluster billing mode.

data_center

Yes

String

The information about the region where the cluster is located. For details, see Endpoints.

vpc

Yes

String

The name of the VPC where the subnet is located. Obtain the VPC name by performing the following operations on the VPC management console:

1. Log in to the management console.
2. Click Virtual Private Cloud and select Virtual Private Cloud from the left list.

On the Virtual Private Cloud page, obtain the VPC name from the list.

master_node_size

Yes

String

The instance specifications of master nodes, for example, c3.4xlarge.2.linux.bigdata. The host specifications supported by MRS are determined by CPU, memory, and disk space. You are advised to obtain the specifications supported by the corresponding version in the corresponding region from the cluster creation page on the MRS console.

core_node_size

Yes

String

The instance specifications of core nodes, for example, c3.4xlarge.2.linux.bigdata. You are advised to obtain the specifications supported by the corresponding version in the corresponding region from the cluster creation page on the MRS console.

component_list

Yes

Array of component_list objects

The list of service components to be installed. For details about the parameters, see Table 4.

available_zone_id

Yes

String

The AZ ID.

vpc_id

Yes

String

The ID of the VPC where the subnet is located.

Obtain the VPC ID by performing the following operations on the VPC management console:

1. Log in to the management console.
2. Click Virtual Private Cloud and select Virtual Private Cloud from the left list.

On the Virtual Private Cloud page, obtain the VPC ID from the list.

subnet_id

Yes

String

The subnet ID.

Obtain the subnet ID by performing the following operations on the VPC management console:

1. Log in to the management console.
2. Click Virtual Private Cloud and select Virtual Private Cloud from the left list.
3. Locate the row that contains the target VPC and click the number in the Subnets column to view the subnet information.
4. Click the subnet name to obtain the network ID.

At least one of subnet_id and subnet_name must be configured. If the two parameters are configured but do not match the same subnet, the cluster fails to create. subnet_id is recommended.

subnet_name

Yes

String

The subnet name.

Obtain the subnet name by performing the following operations on the VPC management console:

1. Log in to the management console.
2. Click Virtual Private Cloud and select Virtual Private Cloud from the left list.
3. Locate the row that contains the target VPC and click the number in the Subnets column to obtain the subnet name.

At least one of subnet_id and subnet_name must be configured. If the two parameters are configured but do not match the same subnet, the cluster fails to create. If only subnet_name is configured and subnets with the same name exist in the VPC, the first subnet name in the VPC is used when a cluster is created. subnet_id is recommended.

security_groups_id

No

String

The ID of the security group configured for the cluster.

• If this parameter is left blank, MRS automatically creates a security group, whose name starts with mrs_{cluster_name}.
• If this parameter is not left blank, a fixed security group is used to create a cluster. The transferred ID must be the security group ID owned by the current tenant. The security group must include an inbound rule in which all protocols and all ports are allowed and the source is the IP address of the specified node on the management plane.

add_jobs

No

Array of add_jobs objects

Jobs can be submitted when a cluster is created. Currently, only one job can be created. For details about the parameters, see Table 5.

volume_size

No

Integer

The data disk storage space of master and core nodes. To increase the data storage capacity, you can add disks when creating a cluster. Select a proper disk storage space based on the following application scenarios:

• Storage-compute decoupling: Data is stored in the OBS system. Costs of clusters are relatively low but computing performance is poor. The clusters can be deleted at any time. It is recommended when data computing is infrequently performed.
• Storage-compute integration: Data is stored in the HDFS system. Costs of clusters are relatively high but computing performance is good. The clusters cannot be deleted in a short term. It is recommended when data computing is frequently performed.

Value range: 100 GB to 32,000 GB. When assigning a value to this parameter, you only need to pass a number, without the unit. This parameter is not recommended. For details, see the description of the volume_type parameter.

volume_type

No

String

The data disk storage type of master and core nodes. Currently, SATA, SAS, SSD, and GPSSD are supported. Disk parameters can be represented by volume_type and volume_size, or multi-disk parameters. If the volume_type and volume_size parameters coexist with the multi-disk parameters, the system reads the volume_type and volume_size parameters first. You are advised to use the multi-disk parameters. Possible values:

• SATA: common I/O
• SAS: high I/O
• SSD: ultra-high I/O
• GPSSD: general-purpose SSD

master_data_volume_type

No

String

This parameter is a multi-disk parameter, indicating the data disk storage type of the master node. Currently, SATA, SAS, SSD, and GPSSD are supported.

master_data_volume_size

No

Integer

This parameter is a multi-disk parameter, indicating the data disk storage space of master nodes. To increase the data storage capacity, you can add disks when creating a cluster.

Value range: 100 GB to 32,000 GB. When assigning a value to this parameter, you only need to pass a number, without the unit.

master_data_volume_count

No

Integer

This parameter is a multi-disk parameter, indicating the number of data disks of the master nodes. The value can only be 1.

core_data_volume_type

No

String

This parameter is a multi-disk parameter, indicating the data disk storage type of core nodes. Currently, SATA, SAS, SSD, and GPSSD are supported.

core_data_volume_size

No

Integer

This parameter is a multi-disk parameter, indicating the data disk storage space of core nodes. To increase the data storage capacity, you can add disks when creating a cluster.

Value range: 100 GB to 32,000 GB. When assigning a value to this parameter, you only need to pass a number, without the unit.

core_data_volume_count

No

Integer

This parameter is a multi-disk parameter, indicating the number of data disks of the core nodes. Value range: 1 to 10

task_node_groups

No

Array of task_node_groups objects

The list of task nodes. For details about the parameters, see Table 6.

bootstrap_scripts

No

Array of bootstrap_script objects

The Bootstrap action script information. For details about the parameters, see Table 8.

node_public_cert_name

No

String

The name of a key pair. You can use a key pair to log in to a cluster node. If login_mode is set to 1, the request body contains the node_public_cert_name field.

cluster_admin_secret

No

String

The password of the MRS Manager administrator. A password must meet the following complexity requirements:

• Must contain 8 to 26 characters.
• Cannot be the username or the username spelled backwards.
• Must contain every type of the following:
  • Lowercase letters
  • Uppercase letters
  • Numbers
  • Special characters (!@$%^-_=+[{}]:,./?)

cluster_master_secret

Yes

String

The password of user root for logging in to a cluster node. If login_mode is set to 0, the request body contains the cluster_master_secret field.

A password must meet the following complexity requirements:

• Must be 8 to 26 characters long.
• Must contain every type of the following: uppercase letters, lowercase letters, numbers, and special characters (!@$%^-_=+[{}]:,./?), but must not contain spaces.
• Cannot be the username or the username spelled backwards.

safe_mode

Yes

Integer

The running mode of an MRS cluster. Possible values:

• 0: normal cluster. In a normal cluster, Kerberos authentication is disabled, and users can use all functions provided by the cluster.
• 1: security cluster. In a security cluster, Kerberos authentication is enabled, and common users cannot use the file management and job management functions of an MRS cluster or view cluster resource usage and the job records of Hadoop and Spark. To use these functions, the users must obtain the relevant permissions from the MRS Manager administrator.

tags

No

Array of tag objects

The cluster tags. For details about the parameters, see Table 9.

• A cluster allows a maximum of 10 tags. A tag name (key) must be unique in a cluster.
• A tag key or value cannot contain the following special characters: =*<>\,|/

cluster_type

No

Integer

The cluster type. Possible values:

• 0: analysis cluster
• 1: streaming cluster

The default value is 0.

log_collection

No

Integer

Whether to collect logs when cluster creation fails.

The default value is 1, indicating that OBS buckets are created only for collecting logs when an MRS cluster fails to create.

Possible values:

• 0: Do not collect logs.
• 1: Collect logs.

enterprise_project_id

No

String

The enterprise project ID.

When you create a cluster, associate the enterprise project ID with the cluster.

The default value is 0, indicating the default enterprise project.

login_mode

No

Integer

The cluster login mode. The default value is 1.

• If login_mode is set to 0, the request body contains the cluster_master_secret field.
• If login_mode is set to 1, the request body contains the node_public_cert_name field.

Possible values:

• 0: password
• 1: key pair

node_groups

No

Array of NodeGroupV11 objects

The list of nodes. For details about the parameters, see Table 10.

Note: Configure either this parameter or the following parameters:

master_node_num, master_node_size, core_node_num, core_node_size, master_data_volume_type, master_data_volume_size, master_data_volume_count, core_data_volume_type, core_data_volume_size, core_data_volume_count, volume_type, volume_size, task_node_groups

component_name

Yes

String

The component name. For details, see the component information in Table 1.

job_type

Yes

Integer

The job type code. Possible values:

• 1: MapReduce
• 2: Spark
• 3: Hive Script
• 4: HiveQL (not supported currently)
• 5: DistCp for importing and exporting data (not supported currently)
• 6: Spark Script
• 7: Spark SQL for submitting Spark SQL statements (not supported currently)
  NOTE:

  Spark and Hive jobs can be created only in clusters that where Spark and Hive are installed.

job_name

Yes

String

The job name. The value can contain 1 to 64 characters. Only letters, numbers, hyphens (-), and underscores (_) are allowed.

jar_path

No

String

The path of the JAR or SQL file for program execution. The value must meet the following requirements:

• The value contains a maximum of 1,023 characters. It cannot contain special characters (;|&>,<'$) and cannot be left blank or all spaces.
• Files can be stored in HDFS or OBS. The path varies depending on the file system.
  • OBS: The path must start with s3a://. Files or programs encrypted by KMS are not supported.
  • HDFS: The path starts with a slash (/).
• Spark Script must end with .sql while MapReduce and Spark Jar must end with .jar. sql and jar are case-insensitive.

arguments

No

String

The key parameter for program execution. The parameter is specified by the function of the user's program. MRS is only responsible for loading the parameter.

The parameter can contain 0 to 150,000 characters, but special characters (;|&>'<$) are not allowed.

input

No

String

The data input path.

The value can contain 0 to 1,023 characters, but special characters (;|&>'<$) are not allowed.

output

No

String

The data output path.

If the specified path does not exist, the system will automatically create it.

The value can contain 0 to 1,023 characters, but special characters (;|&>'<$) are not allowed.

job_log

No

String

The path for storing job logs that record job running status.

The value can contain 0 to 1,023 characters, but special characters (;|&>'<$) are not allowed.

shutdown_cluster

No

Boolean

Whether to terminate the cluster after the job execution is complete.

• true: Terminate the cluster.
• false: Do not terminate the cluster.

file_action

No

String

The action to be performed on a file. Possible values:

• import
• export

submit_job_once_cluster_run

Yes

Boolean

• true: Submit a job during cluster creation.
• false: Submit a job after the cluster is created.

Set this parameter to true in this example.

hql

No

String

The HQL script statement.

hive_script_path

Yes

String

The SQL program path. This parameter is required by Spark Script and Hive Script jobs only, and the value must meet the following requirements:

• The value contains a maximum of 1,023 characters. It cannot contain special characters (;|&><'$) and cannot be left blank or all spaces.
• Files can be stored in HDFS or OBS. The path varies depending on the file system.
  • OBS: The path must start with s3a://. Files or programs encrypted by KMS are not supported.
  • HDFS: The path starts with a slash (/).
• Ends with .sql. sql is case-insensitive.

node_num

Yes

Integer

The number of task nodes. The value ranges from 0 to 500, and the total number of core and task nodes cannot exceed 500.

node_size

Yes

String

The instance specifications of task nodes, for example, c6.4xlarge.4.linux.mrs.

Obtain the instance specifications of the corresponding version in the corresponding region from the cluster createion page of the MRS management console.

data_volume_type

Yes

String

The data disk storage type of task nodes. Possible values:

• SATA: common I/O
• SAS: high I/O
• SSD: ultra-high I/O
• GPSSD: general-purpose SSD

data_volume_count

Yes

Integer

The number of data disks of a task node.

Value range: 0 to 10

data_volume_size

Yes

Integer

The data disk storage space of a task node.

Value range: 100 GB to 32,000 GB

You only need to enter a number and do not need to enter the unit GB.

auto_scaling_policy

No

auto_scaling_policy object

The auto scaling policy. For details, see Table 7.

auto_scaling_enable

Yes

Boolean

Whether to enable the auto scaling policy.

min_capacity

Yes

Integer

The minimum number of nodes reserved in the node group.

Value range: 0 to 500

max_capacity

Yes

Integer

The maximum number of nodes in the node group.

Value range: 0 to 500

resources_plans

No

Array of resources_plan objects

The resource plan list. For details, see Table 12. If this parameter is left blank, the resource plan is disabled.

When auto_scaling_enable is set to true, either this parameter or rules must be configured.

exec_scripts

No

Array of scale_script objects

The list of custom scaling automation scripts. For details, see Table 15. If this parameter is left blank, a hook script is disabled.

rules

No

Array of rules objects

The list of auto scaling rules. For details, see Table 13.

When auto_scaling_enable is set to true, either this parameter or resources_plans must be configured.

name

Yes

String

The name of a bootstrap action script, which must be unique in a cluster.

The value can contain only numbers, letters, spaces, hyphens (-), and underscores (_) and cannot start with a space.

The value can contain 1 to 64 characters.

uri

Yes

String

The path of a Bootstrap action script. Set this parameter to an OBS bucket path or a local VM path.

• OBS bucket path: Enter a script path. For example, enter the path of the public sample script provided by MRS. Example: s3a://bootstrap/presto/presto-install.sh. If dualroles is installed, the parameter of the presto-install.sh script is dualroles. If worker is installed, the parameter of the presto-install.sh script is worker. Based on the Presto usage habit, you are advised to install dualroles on the active Master nodes and worker on the Core nodes.
• Local VM path: Enter a script path. The script path must start with a slash (/) and end with .sh.

parameters

No

String

The bootstrap action script parameters.

nodes

Yes

Array of strings

The type of a node where the bootstrap action script is executed. The value can be Master, Core, or Task.

The node type must be represented in lowercase letters.

active_master

No

Boolean

Whether the bootstrap action script runs only on active master nodes.

The default value is false, indicating that the Bootstrap action script can run on all Master nodes.

before_component_start

No

Boolean

Time when the bootstrap action script is executed. Currently, the following two options are available: Before component start and After component start

The default value is false, indicating that the bootstrap action script is executed after the component is started.

fail_action

Yes

String

Whether to continue executing subsequent scripts and creating a cluster after the Bootstrap action script fails to be executed.

• continue: Continue to execute subsequent scripts.
• errorout: Stop the action.

action_stages

No

Array of strings

Select the time when the bootstrap action script is executed.

• BEFORE_COMPONENT_FIRST_START: before initial component starts
• AFTER_COMPONENT_FIRST_START: after initial component starts
• BEFORE_SCALE_IN: before scale-in
• AFTER_SCALE_IN: after scale-in
• BEFORE_SCALE_OUT: before scale-out
• AFTER_SCALE_OUT: after scale-out

key

Yes

String

The tag key.

• It contains a maximum of 36 Unicode characters and cannot be an empty string.
• The tag key cannot start or end with spaces or contain non-printable ASCII characters (0–31) and special characters (=*<>\,|/).
• The tag key of a resource must be unique.

value

Yes

String

The value.

• The value contains a maximum of 43 Unicode characters and can be an empty string.
• The tag value cannot start or end with spaces or contain non-printable ASCII characters (0–31) and special characters (=*<>\,|/).

group_name

Yes

String

The node group name. Possible values:

• master_node_default_group
• core_node_analysis_group
• core_node_streaming_group
• task_node_analysis_group
• task_node_streaming_group

node_num

Yes

Integer

The number of nodes. The value ranges from 0 to 500, and the default value is 0. The total number of core and task nodes cannot exceed 500.

node_size

Yes

String

The instance specifications of cluster nodes, for example, .

Obtain the instance specifications of the corresponding version in the corresponding region from the cluster createion page of the MRS management console.

root_volume_size

Yes

String

The system disk storage space of a node.

root_volume_type

Yes

String

The system disk storage type of a node. Possible values:

• SATA: common I/O
• SAS: high I/O
• SSD: ultra-high I/O
• GPSSD: general-purpose SSD

data_volume_type

Yes

String

The data disk storage type of a node. Possible values:

• SATA: common I/O
• SAS: high I/O
• SSD: ultra-high I/O
• GPSSD: general-purpose SSD

data_volume_count

Yes

Integer

The number of data disks of a node.

Value range: 0 to 10

data_volume_size

Yes

Integer

The data disk storage space of a node.

Value range: 100 GB to 32,000 GB

auto_scaling_policy

No

auto_scaling_policy object

The auto scaling rule information. This parameter is available only when group_name is set to task_node_analysis_group or task_node_streaming_group. For details, see Table 11.

For details about the parameters, see Table 7.

master_node_num

Yes

Integer

The number of master nodes. If cluster HA is enabled, set this parameter to 2. If cluster HA is disabled, set this parameter to 1.

This parameter cannot be set to 1 in MRS 3.x.

master_node_size

Yes

String

The instance specifications of master nodes, for example, . The host specifications supported by MRS are determined by CPU, memory, and disk space.

Obtain the instance specifications of the corresponding version in the corresponding region from the cluster createion page of the MRS management console.

core_node_num

Yes

Integer

The number of core nodes.

Value range: 1 to 500

A maximum of 500 core nodes are supported by default. If more than 500 core nodes are required, .

core_node_size

Yes

String

The instance specifications of core nodes, for example, .

Obtain the instance specifications of the corresponding version in the corresponding region from the cluster createion page of the MRS management console.

master_data_volume_type

No

String

This parameter is a multi-disk parameter, indicating the data disk storage type of master nodes. Currently, SATA, SAS, and SSD are supported.

master_data_volume_size

No

Integer

This parameter is a multi-disk parameter, indicating the data disk storage space of master nodes. To increase the data storage capacity, you can add disks when creating a cluster.

Value range: 100 GB to 32,000 GB

master_data_volume_count

No

Integer

This parameter is a multi-disk parameter, indicating the number of data disks of the master nodes.

The value can only be 1.

core_data_volume_type

No

String

This parameter is a multi-disk parameter, indicating the data disk storage type of core nodes. Currently, SATA, SAS, and SSD are supported.

core_data_volume_size

No

Integer

This parameter is a multi-disk parameter, indicating the data disk storage space of core nodes. To increase the data storage capacity, you can add disks when creating a cluster.

Value range: 100 GB to 32,000 GB

core_data_volume_count

No

Integer

This parameter is a multi-disk parameter, indicating the number of data disks of core nodes.

Value range: 1 to 10

volume_type

No

String

The data disk storage type of the master and core nodes. Currently, SATA, SAS, and SSD are supported. Disk parameters can be represented by volume_type and volume_size, or multi-disk parameters. If the volume_type and volume_size parameters coexist with the multi-disk parameters, the system reads the volume_type and volume_size parameters first. You are advised to use the multi-disk parameters. Possible values:

• SATA: common I/O
• SAS: high I/O
• SSD: ultra-high I/O

volume_size

No

Integer

The data disk storage space of master and core nodes. To increase the data storage capacity, you can add disks when creating a cluster.

Value range: 100 GB to 32,000 GB

This parameter is not recommended. For details, see the description of the volume_type parameter.

task_node_groups

No

Array

The list of task nodes. For more parameter description, see Table 6.

period_type

Yes

String

Cycle type of a resource plan. Currently, only the following cycle type is supported:

daily

start_time

Yes

String

Start time of a resource plan. The value is in the format of hour:minute, indicating that the time ranges from 0:00 to 23:59.

end_time

Yes

String

End time of a resource plan. The value is in the same format as that of start_time. The interval between end_time and start_time must be greater than or equal to 30 minutes.

min_capacity

Yes

Integer

Minimum number of the preserved nodes in a node group in a resource plan.

Value range: 0 to 500

max_capacity

Yes

Integer

Maximum number of the preserved nodes in a node group in a resource plan.

Value range: 0 to 500

name

Yes

String

Name of an auto scaling rule.

It contains only 1 to 64 characters. Only letters, digits, hyphens (-), and underscores (_) are allowed.

Rule names must be unique in a node group.

description

No

String

Description about an auto scaling rule.

It contains a maximum of 1024 characters.

adjustment_type

Yes

String

Auto scaling rule adjustment type. Possible values:

• scale_out: cluster scale-out
• scale_in: cluster scale-in

cool_down_minutes

Yes

Integer

Cluster cooling time after an auto scaling rule is triggered, when no auto scaling operation is performed. The unit is minute.

Value range: 0 to 10,080. One week is equal to 10,080 minutes.

scaling_adjustment

Yes

Integer

Number of nodes that can be adjusted once.

Value range: 1 to 100

trigger

Yes

trigger object

Condition for triggering a rule. For details, see Table 14.

metric_name

Yes

String

Metric name.

This triggering condition makes a judgment according to the value of the metric.

A metric name contains a maximum of 64 characters.

Table 16 lists the supported metric names.

metric_value

Yes

String

Metric threshold to trigger a rule

The parameter value must be an integer or number with two decimal places only. Table 16 provides value types and ranges corresponding to metric_name.

comparison_operator

No

String

Metric judgment logic operator. Possible values:

• LT: less than
• GT: greater than
• LTOE: less than or equal to
• GTOE: greater than or equal to

evaluation_periods

Yes

Integer

Number of consecutive five-minute periods, during which a metric threshold is reached

Value range: 1 to 288

name

Yes

String

Name of a custom automation script. It must be unique in a same cluster.

The value can contain only digits, letters, spaces, hyphens (-), and underscores (_) and cannot start with a space.

The value can contain 1 to 64 characters.

uri

Yes

String

Path of a custom automation script. Set this parameter to an OBS bucket path or a local VM path.

• OBS bucket path: Enter a script path manually, for example, s3a://XXX/scale.sh.
• Local VM path: Enter a script path. The script path must start with a slash (/) and end with .sh.

parameters

No

String

Parameters of a custom automation script.

• Multiple parameters are separated by space.
• The following predefined system parameters can be transferred:
  • ${mrs_scale_node_num}: Number of the nodes to be added or removed
  • ${mrs_scale_type}: Scaling type. The value can be scale_out or scale_in.
  • ${mrs_scale_node_hostnames}: Host names of the nodes to be added or removed
  • ${mrs_scale_node_ips}: IP addresses of the nodes to be added or removed
  • ${mrs_scale_rule_name}: Name of the rule that triggers auto scaling
• Other user-defined parameters are used in the same way as those of common shell scripts. Parameters are separated by space.

nodes

Yes

Array of string

Type of a node where the custom automation script is executed. The node type can be Master, Core, or Task.

active_master

No

Boolean

Whether the custom automation script runs only on the active Master node.

The default value is false, indicating that the custom automation script can run on all Master nodes.

action_stage

Yes

String

Time when a script is executed.

The following four options are supported:

• before_scale_out: before scale-out
• before_scale_in: before scale-in
• after_scale_out: after scale-out
• after_scale_in: after scale-in

fail_action

Yes

String

Whether to continue to execute subsequent scripts and create a cluster after the custom automation script fails to be executed.

• continue: Continue to execute subsequent scripts.
• errorout: Stop the action.
  NOTE:

  • You are advised to set this parameter to continue in the commissioning phase so that the cluster can continue to be installed and started no matter whether the custom automation script is executed successfully.
  • The scale-in operation cannot be undone. Therefore, fail_action must be set to continue for the scripts that are executed after scale-in.

Streaming cluster

StormSlotAvailable

Integer

Number of available Storm slots.

Value range: 0 to 2147483646

StormSlotAvailablePercentage

Percentage

Percentage of available Storm slots, that is, the proportion of the available slots to total slots.

Value range: 0 to 100

StormSlotUsed

Integer

Number of the used Storm slots.

Value range: 0 to 2147483646

StormSlotUsedPercentage

Percentage

Percentage of the used Storm slots, that is, the proportion of the used slots to total slots.

Value range: 0 to 100

StormSupervisorMemAverageUsage

Integer

Average memory usage of the Supervisor process of Storm.

Value range: 0 to 2147483646

StormSupervisorMemAverageUsagePercentage

Percentage

Average percentage of the used memory of the Supervisor process of Storm to the total memory of the system.

Value range: 0 to 100

StormSupervisorCPUAverageUsagePercentage

Percentage

Average percentage of the used CPUs of the Supervisor process of Storm to the total CPUs.

Value range: 0 to 6000

Analysis cluster

YARNAppPending

Integer

Number of pending tasks on Yarn.

Value range: 0 to 2147483646

YARNAppPendingRatio

Ratio

Ratio of pending tasks on Yarn, that is, the ratio of pending tasks to running tasks on Yarn.

Value range: 0 to 2147483646

YARNAppRunning

Integer

Number of running tasks on Yarn.

Value range: 0 to 2147483646

YARNContainerAllocated

Integer

Number of containers allocated to Yarn.

Value range: 0 to 2147483646

YARNContainerPending

Integer

Number of pending containers on Yarn.

Value range: 0 to 2147483646

YARNContainerPendingRatio

Ratio

Ratio of pending containers on Yarn, that is, the ratio of pending containers to running containers on Yarn.

Value range: 0 to 2147483646

YARNCPUAllocated

Integer

Number of virtual CPUs (vCPUs) allocated to Yarn

Value range: 0 to 2147483646

YARNCPUAvailable

Integer

Number of available vCPUs on Yarn.

Value range: 0 to 2147483646

YARNCPUAvailablePercentage

Percentage

Percentage of available vCPUs on Yarn, that is, the proportion of available vCPUs to total vCPUs.

Value range: 0 to 100

YARNCPUPending

Integer

Number of pending vCPUs on Yarn.

Value range: 0 to 2147483646

YARNMemoryAllocated

Integer

Memory allocated to Yarn. The unit is MB.

Value range: 0 to 2147483646

YARNMemoryAvailable

Integer

Available memory on Yarn. The unit is MB.

Value range: 0 to 2147483646

YARNMemoryAvailablePercentage

Percentage

Percentage of available memory on Yarn, that is, the proportion of available memory to total memory on Yarn.

Value range: 0 to 100

YARNMemoryPending

Integer

Pending memory on Yarn.

Value range: 0 to 2147483646

cluster_id

String

Cluster ID, which is returned by the system after the cluster is created.

result

Boolean

Operation result.

• true: The operation is successful.
• false: The operation failed.

msg

String

System message, which can be empty.

200

The cluster has been created.