Configuring Auto Scaling Metrics

Updated on 2024-10-11 GMT+08:00

View PDF

Auto Scaling Policies by Node Group

When you add a rule, you can refer to Table 1 to configure the corresponding metrics.

**Table 1** Auto scaling metrics
Cluster Type	Metric	Value Type	Description
Streaming cluster	StormSlotAvailable	Integer	Number of available Storm slots Value range: 0 to 2147483646
	StormSlotAvailablePercentage	Percentage	Percentage of available Storm slots, that is, the proportion of the available slots to total slots Value range: 0 to 100
	StormSlotUsed	Integer	Number of used Storm slots Value range: 0 to 2147483646
	StormSlotUsedPercentage	Percentage	Percentage of the used Storm slots, that is, the proportion of the used slots to total slots Value range: 0 to 100
	StormSupervisorMemAverageUsage	Integer	Average memory usage of the Supervisor process of Storm Value range: 0 to 2147483646
	StormSupervisorMemAverageUsagePercentage	Percentage	Average percentage of the used memory of the Supervisor process of Storm to the total memory of the system Value range: 0 to 100
	StormSupervisorCPUAverageUsagePercentage	Percentage	Average percentage of the used CPUs of the Supervisor process of Storm to the total CPUs Value range: 0 to 6000
Analysis cluster	YARNAppPending	Integer	Number of pending tasks on YARN Value range: 0 to 2147483646
	YARNAppPendingRatio	Ratio	Ratio of pending tasks on YARN, that is, the ratio of pending tasks to running tasks on YARN Value range: 0 to 2147483646
	YARNAppRunning	Integer	Number of running tasks on YARN Value range: 0 to 2147483646
	YARNContainerAllocated	Integer	Number of containers allocated to YARN Value range: 0 to 2147483646
	YARNContainerPending	Integer	Number of pending containers on YARN Value range: 0 to 2147483646
	YARNContainerPendingRatio	Ratio	Ratio of pending containers on Yarn, that is, the ratio of pending containers to running containers on YARN Value range: 0 to 2147483646
	YARNCPUAllocated	Integer	Number of virtual CPUs (vCPUs) allocated to YARN Value range: 0 to 2147483646
	YARNCPUAvailable	Integer	Number of available vCPUs on YARN Value range: 0 to 2147483646
	YARNCPUAvailablePercentage	Percentage	Percentage of available vCPUs on YARN that is, the proportion of available vCPUs to total vCPUs Value range: 0 to 100
	YARNCPUPending	Integer	Number of pending vCPUs on YARN Value range: 0 to 2147483646
	YARNMemoryAllocated	Integer	Memory allocated to YARN, in MB Value range: 0 to 2147483646
	YARNMemoryAvailable	Integer	Available memory on YARN in MB Value range: 0 to 2147483646
	YARNMemoryAvailablePercentage	Percentage	Percentage of available memory on YARN that is, the proportion of available memory to total memory on YARN Value range: 0 to 100
	YARNMemoryPending	Integer	Pending memory on YARN Value range: 0 to 2147483646

NOTE:

When the value type is percentage or ratio in Table 1, the valid value can be accurate to percentile. The percentage metric value is a decimal value with a percent sign (%) removed. For example, 16.80 represents 16.80%.
Hybrid clusters support all metrics of analysis and streaming clusters.

Automation Script

When you add an automation script, you can configure related parameters by referring to Table 2.

**Table 2** Automation script configuration description
Parameter	Description
Name	Name of an automation script The value can contain only numbers, letters, spaces, hyphens (-), and underscores (_) and must not start with a space. The value can contain 1 to 64 characters. NOTE: A name must be unique in the same cluster. You can configure the same name for different clusters.
Script Path	Script path. The value can be an OBS file system path or a local VM path. An OBS file system path must start with s3a:// and end with .sh, for example, s3a://mrs-samples/xxx.sh. A local VM path must start with a slash (/) and end with .sh. For example, the path of the example script for installing the Zepelin is /opt/bootstrap/zepelin/zepelin_install.sh.
Execution Node	Select a type of the node where an automation script is executed. NOTE: If you select Master nodes, you can choose whether to run the script only on the active Master nodes by enabling or disabling the Active Master switch. If you enable it, the script runs only on the active Master nodes. If you disable it, the script runs on all master nodes. This function is disabled by default.
Parameter	Automation script parameter. The following predefined variables can be imported to obtain auto scaling information: ${mrs_scale_node_num}: Number of auto scaling nodes. The value is always positive. ${mrs_scale_type}: Scale-out/in type. The value can be scale_out or scale_in. ${mrs_scale_node_hostnames}: Host names of the auto scaling nodes. Use commas (,) to separate multiple host names. ${mrs_scale_node_ips}: IP address of the auto scaling nodes. Use commas (,) to separate multiple IP addresses. ${mrs_scale_rule_name}: Name of the triggered auto scaling rule. For a resource plan, this parameter is set to resource_plan.
Executed	Time for executing an automation script. The following four options are supported: Before scale-out, After scale-out, Before scale-in, and After scale-in. NOTE: Assume that the execution nodes include Task nodes. The automation script executed before scale-out cannot run on the Task nodes to be added. The automation script executed after scale-out can run on the added Task nodes. The automation script executed before scale-in can run on Task nodes to be deleted. The automation script executed after scale-in cannot run on the deleted Task nodes.
Action upon Failure	Whether to continue to execute subsequent scripts and scale-out/in after the script fails to be executed. NOTE: You are advised to set this parameter to Continue in the commissioning phase so that the cluster can continue the scale-out/in operation no matter whether the script is executed. If the script fails to be executed, view the log in /var/log/Bootstrap on the cluster VM. The scale-in operation cannot be rolled back. Therefore, the Action upon Failure can only be set to Continue after scale-in.

NOTE:

The automation script is triggered only during auto scaling. It is not triggered when the cluster node is manually scaled out or in.

Parent topic: Configuring Auto Scaling Rules

Previous topic: Configuring Automation Scripts

Next topic: Managing Data Connections

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot