Creating a Node Scaling Policy

Prerequisites

Before using the node scaling function, you must install the CCE Cluster Autoscaler add-on of v1.13.8 or later in the cluster.

Constraints

• Auto scaling policies apply to node pools. When the number of nodes in a node pool is 0 and the scaling policy is based on CPU or memory usage, node scaling is not triggered.
• Node scale-in will cause PVC/PV data loss for the local PVs associated with the node. These PVCs and PVs cannot be restored or used again. In a node scale-in, the pod that uses the local PV is evicted from the node. A new pod is created and stays in the pending state. This is because the PVC used by the pod has a node label, due to which the pod cannot be scheduled.
• When Autoscaler is used, some taints or annotations may affect auto scaling. Therefore, do not use the following taints or annotations in clusters:
  • ignore-taint.cluster-autoscaler.kubernetes.io: The taint works on nodes. Kubernetes-native Autoscaler supports protection against abnormal scale outs and periodically evaluates the proportion of available nodes in the cluster. When the proportion of non-ready nodes exceeds 45%, protection will be triggered. In this case, all nodes with the ignore-taint.cluster-autoscaler.kubernetes.io taint in the cluster are filtered out from the Autoscaler template and recorded as non-ready nodes, which affects cluster scaling.
  • cluster-autoscaler.kubernetes.io/enable-ds-eviction: The annotation works on pods, which determines whether DaemonSet pods can be evicted by Autoscaler. For details, see Well-Known Labels, Annotations and Taints.

Procedure

1. Log in to the CCE console and click the cluster name to access the cluster console.
2. In the navigation pane, choose Nodes. On the Node Pools tab, locate the row containing the target node pool and click Auto Scaling.
   • If the auto scaling add-on has not been installed, configure add-on parameters based on service requirements, click Install, and wait until the add-on is installed. For details about add-on configurations, see CCE Cluster Autoscaler.
   • If the auto scaling add-on has been installed, directly configure auto scaling policies.

3. Configure auto scaling policies.

   Automatic Scale-out

   • Automatic scale-out: When workload pods in a cluster cannot be scheduled (pods remain in pending state), CCE automatically adds nodes to the slave node pool. If a node has been configured as an affinity node for pods, no node will not be automatically added when pods cannot be scheduled. Such auto scaling typically works with an HPA policy. For details, see Using HPA and CA for Auto Scaling of Workloads and Nodes.

     If this parameter is not selected, scaling can be performed only using custom scaling policies.

     

     The automatic scale-out configuration takes effect on all node pools in a cluster. After the configuration is modified, the auto scaling add-on will be restarted.

   • Customized Rule: Click Add Rule. In the dialog box displayed, configure parameters. You can add multiple node scaling rules, a maximum of one CPU usage-based rule, and one memory usage-based rule. The total number of rules cannot exceed 10.
     The following table lists custom rules.

     Table 1 Custom rules

     Rule Type	Configuration
     Metric-based	Trigger: Select CPU allocation rate or Memory allocation rate and enter a value. The value must be greater than the scale-in percentage configured in the auto scaling add-on. NOTE: Resource allocation (%) = Resources requested by pods in the node pool/Resources allocatable to pods in the node pool If multiple rules meet the conditions, the rules are executed in either of the following modes: If rules based on the CPU allocation rate and memory allocation rate are configured and two or more rules meet the scale-out conditions, the rule that will add the most nodes will be executed. If a rule based on the CPU allocation rate and a periodic rule are configured and they both meet the scale-out conditions, one of them will be executed randomly. The rule executed first (rule A) changes the node pool to the scaling state. As a result, the other rule (rule B) cannot be executed. After rule A is executed and the node pool status becomes normal, rule B will not be executed. If rules based on the CPU allocation rate and memory allocation rate are configured, the policy detection period varies with the processing logic of each loop of the Autoscaler add-on. A scale-out is triggered once the conditions are met, but it is constrained by other factors such as the cooldown period and node pool status. When the number of nodes in the cluster reaches the upper limit, or the CPU or memory usage reaches the upper limit of the autoscaler add-on, node scale-out will not be triggered. Action: Configure an action to be performed when the triggering condition is met. Custom: Add a specified number of nodes to a node pool. Auto calculation: When the trigger condition is met, nodes are automatically added and the allocation rate is restored to a value lower than the threshold. The formula is as follows: Number of nodes to be added = [Resource request of pods in the node pool/(Available resources of a single node x Target allocation rate)] – Number of current nodes + 1
     Periodic	Trigger Time: You can select a specific time every day, every week, every month, or every year. Action: specifies an action to be carried out when the trigger time is reached. A specified number of nodes will be added to the node pool.

   • Cooldown Period: a period during which the nodes added in the current node pool cannot be scaled in.

   Elastic Scale-in

   Automatic scale-in: Nodes in a cluster are automatically scaled in when the scale-in conditions are met.

   

   The automatic scale-in configuration takes effect on all node pools in a cluster. After the configuration is modified, the auto scaling add-on will be restarted.

   • Scale-in condition: When both requested cluster node CPU and memory resources are lower than a specified value (0.5 by default, which is 50%) for a consecutive period of time (10 minutes by default), cluster nodes will be scaled in. A node that is unavailable for more than 20 minutes will be automatically reclaimed.
   • Max. Nodes for Batch Deletion: maximum number of idle nodes that can be concurrently deleted. Default value: 10.
     Only idle nodes can be concurrently scaled in. Nodes that are not idle can only be scaled in one by one.

     

     During a node scale-in, if the pods on the node do not need to be evicted (such as DaemonSet pods), the node is idle. Otherwise, the node is not idle.

   • Check Interval: interval for rechecking a node that could not be removed before. Default value: 5 minutes.
   • Cooldown Period
     • How long after a scale-out that a scale-in evaluation resumes: Default value: 10 minutes.
       

       If both auto scale-out and scale-in exist in a cluster, set How long after a scale-out that a scale-in evaluation resumes to 0 minutes. This prevents the node scale-in from being blocked due to continuous scale-out of some node pools or retries upon a scale-out failure, which results in unexpected waste of node resources.

     • How long after the node deletion that a scale-in evaluation resumes: Default value: 10 minutes.
     • How long after a scale-in failure that a scale-in evaluation resumes: Default value: 3 minutes.

   AS Configuration

   • Nodes: The number of nodes in a node pool will always be within the range during auto scaling.
   • Node Pool Priority: A larger value indicates a higher priority.

   AS Object

   Specification selection: Configure whether to enable auto scaling for node flavors in a node pool.

4. Click OK.

Cooldown Period

The impact and relationship between the two cooldown periods configured for a node pool are as follows:

Cooldown Period During a Scale-out

This interval indicates the period during which nodes added to the current node pool after a scale-out cannot be deleted. This setting takes effect in the entire node pool.

Cooldown Period During a Scale-in

The interval after a scale-out indicates the period during which the entire cluster cannot be scaled in after the auto scaling add-on triggers a scale-out (due to the unschedulable pods, metrics, and scaling policies). This interval takes effect in the entire cluster.

The interval after a node is deleted indicates the period during which the cluster cannot be scaled in after the auto scaling add-on triggers a scale-in. This setting takes effect in the entire cluster.

The interval after a failed scale-in indicates the period during which the cluster cannot be scaled in after the auto scaling add-on triggers a scale-in. This setting takes effect in the entire cluster.

Example YAML

The following is a YAML example of a node scaling policy:

apiVersion: autoscaling.cce.io/v1alpha1
kind: HorizontalNodeAutoscaler
metadata:
  creationTimestamp: "2020-02-13T12:47:49Z"
  generation: 1
  name: xxxx
  namespace: kube-system
  resourceVersion: "11433270"
  selfLink: /apis/autoscaling.cce.io/v1alpha1/namespaces/kube-system/horizontalnodeautoscalers/xxxx
  uid: c2bd1e1d-60aa-47b5-938c-6bf3fadbe91f
spec:
  disable: false
  rules:
  - action:
      type: ScaleUp
      unit: Node
      value: 1
    cronTrigger:
      schedule: 47 20 * * *
    disable: false
    ruleName: cronrule
    type: Cron
  - action:
      type: ScaleUp
      unit: Node
      value: 2
    disable: false
    metricTrigger:
      metricName: Cpu
      metricOperation: '>'
      metricValue: "40"
      unit: Percent
    ruleName: metricrule
    type: Metric
  targetNodepoolIds:
  - 7d48eca7-3419-11ea-bc29-0255ac1001a8