Help Center/ MapReduce Service/ User Guide/ MRS Cluster O&M/ Adjusting the Capacity of an MRS Cluster/ MRS Task Node Auto Scaling/ Adding an Auto Scaling Policy for MRS Task Nodes
Updated on 2024-10-25 GMT+08:00
View PDF
Share

Adding an Auto Scaling Policy for MRS Task Nodes

Video Tutorial

This tutorial demonstrates how to configure AS when purchasing a cluster and how to add an AS policy to an existing cluster.

The UI may vary depending on the version. This tutorial is for reference only.

Configuring Auto Scaling During Cluster Creation

When you create a cluster, you can configure the auto scaling function in advanced configuration parameters.

Auto scaling policies can be configured during cluster creation only for analysis, streaming, and hybrid clusters.

  1. Log in to the MRS management console.
  2. When you buy a cluster containing task nodes, configure the cluster software and hardware information by referring to Manually Buying an MRS Cluster. Then, on the Set Advanced Options page, enable Analysis Task and configure or modify auto scaling rules and resource plans.

    Figure 1 Configuring auto scaling rules when creating a cluster

    You can configure the auto scaling rules by referring to the following scenarios:

Adding an Auto Scaling Policy for a Cluster

After a cluster is created, you can configure rules for the task node group in a cluster by node group or resource pool.

The node group policy and resource pool policy are mutually exclusive. You can configure either of them as needed.

MRS 3.1.5 or later supports the specified resource pool policy.

Item

By Node Group

By Resource Pool

Auto scaling object

All nodes in the task node group

Task nodes in the resource pool specified by an auto scaling policy

Resource pool ownership of added nodes

Default resource pool

Resource pool specified by the auto scaling policy

Scale-in object

Random scale-in of nodes in the task node group

Random scale-in of nodes in a resource pool specified by an auto scaling policy

Prerequisites

Procedure

  1. Log in to the MRS management console.
  2. Choose Active Clusters, select a running cluster, and click its name to go its details page.
  3. On the page that is displayed, click the Auto Scaling tab.

    You can configure policies by resource pool or node group as needed.

    • Auto scaling policies of different node groups are mutually exclusive. That is, you can enable auto scaling policies only for one node group.
    • An auto scaling rule adjusts the number of nodes, but also affects the actual price. Exercise caution when adding an auto scaling rule.

  4. Click Create Auto Scaling Policy to create an auto scaling policy.

    You can configure the auto scaling rules by referring to the following scenarios:

Scenario 1: Using Auto Scaling Rules Alone

Scenario where only auto scaling rules are configured: The number of nodes needs to be dynamically adjusted based on the YARN resource usage. When the available YARN memory is less than 20%, five nodes need to be added. When the available YARN memory is greater than 70%, five nodes need to be reduced. The number of nodes in a task node group ranges from 1 to 10.

  1. Go to the Auto Scaling page to configure auto scaling rules.

    • Configure the Default Range parameter.

      Enter a task node range, in which auto scaling is performed. This constraint applies to all scale-in and scale-out rules. The maximum value range allowed is 0 to 500.

      The value range in this example is 1 to 10.

    • Configure an auto scaling rule.

      To enable Auto Scaling, you must configure a scale-out or scale-in rule.

      1. Select Scale-Out or Scale-In.
      2. Click Add Rule.
        Figure 2 Adding a rule
      3. Configure Rule Name, If, Last For, Add, and Cooldown Period. For details about auto scaling metrics, see Automatic Scaling of Task Nodes in an MRS Cluster.
      4. Click OK.

        You can view, edit, or delete the rules you configured in the Scale-out or Scale-in area on the Auto Scaling page. You can click Add Rule to configure multiple rules.

  2. Click OK.

    If you want to configure an auto scaling rule for an existing cluster, select I agree to authorize MRS to scale out or in nodes based on the above rule.

Scenario 2: Using Resource Plans Alone

If the data volume changes regularly every day and you want to scale out or in a cluster before the data volume changes, you can create resource plans to adjust the number of Task nodes as planned in the specified time range.

For example: A real-time processing service sees a sharp increase in data volume from 7:00 to 13:00 on Monday, Tuesday, and Saturday. Assume that an MRS streaming cluster is used to process the service data. Five task nodes are required from 7:00 to 13:00 on Monday, Tuesday, and Saturday, while only two are required at other time.

  1. Go to the Auto Scaling page to configure a resource plan.

    A resource plan can be configured to adjust the number of nodes, which affects the actual price. Exercise caution when performing this operation.

  2. For example, the Default Range of node quantity is set to 2-2, indicating that the number of task nodes is fixed to 2 except the time range specified in the resource plan.
  3. Click Configure Node Range for Specific Time Range under Default Range or Add Resource Plan.
  4. Configure Effective On, Time Range, and Node Range.

    For example, set Effective On to Monday, Tuesday, and Saturday, Time Range to 07:00-13:00, and Node Range to 5-5. This indicates that the number of task nodes is fixed at 5 from 07:00 to 13:00.

    Click Configure Node Range for Specific Time Range to configure multiple resource plans.

    • Effective On is set to Daily by default. You can also select one or multiple days from Monday to Sunday.
    • If you do not set Node Range, its default value will be used.
    • If you set both Node Range and Time Range, the node range you set will be used during the time range you set, and the default node range will be used beyond the time range you set. If the time is not within the configured time range, the default range is used.

Scenario 3: Using Auto Scaling Rules and Resource Plans Together

If the data volume is not stable and the expected fluctuation may occur, the fixed Task node range cannot guarantee that the requirements in some service scenarios are met. In this case, it is necessary to adjust the number of Task nodes based on the real-time loads and resource plans.

For example: A real-time processing service sees an unstable increase in data volume from 7:00 to 13:00 on Monday, Tuesday, and Saturday. For example, 5 to 8 task nodes are required from 7:00 to 13:00 on Monday, Tuesday, and Saturday, and 2 to 4 are required beyond this period. You can set an auto scaling rule based on a resource plan. When the data volume exceeds the expected value, the number of Task nodes changes with resource loads, without exceeding the node range specified in the resource plan. When a resource plan is triggered, the number of nodes is adjusted within the specified node range with minimum affect. That is, increase nodes to the upper limit and decrease nodes to the lower limit.

  1. Go to the Auto Scaling page to configure auto scaling rules.

    An auto scaling rule adjusts the number of nodes, but also affects the actual price. Exercise caution when adding an auto scaling rule.

    • Default Range

      Enter a task node range, in which auto scaling is performed. This constraint applies to all scale-in and scale-out rules.

      For example, this parameter is set to 2-4 in this scenario.

    • Auto Scaling

      To enable Auto Scaling, you must configure a scale-out or scale-in rule.

      1. Select Scale-Out or Scale-In.
      2. Click Add Rule. The Add Rule page is displayed.
        Figure 3 Adding a rule
      3. Configure the Rule Name, If, Last for, Add, and Colddown Period parameters.
      4. Click OK.

        You can view, edit, or delete the rules you configured in the Scale-out or Scale-in area on the Auto Scaling page.

  2. Configure a resource plan.

    1. Click Configure Node Range for Specific Time Range under Default Range or Add Resource Plan.
    2. Configure Effective On, Time Range, and Node Range.

      For example, set Effective On to Monday, Tuesday, and Saturday, Time Range to 07:00-13:00, and Node Range to 5-8.

      Click Configure Node Range for Specific Time Range or Add Resource Plan to configure multiple resource plans.

      • Effective On is set to Daily by default. You can also select one or multiple days from Monday to Sunday.
      • If you do not set Node Range, its default value will be used.
      • If you set both Node Range and Time Range, the node range you set will be used during the time range you set, and the default node range will be used beyond the time range you set. If the time is not within the configured time range, the default range is used.