Using HPA and the CCE Cloud Bursting Engine for CCI Add-on to Schedule CCE Workloads to CCI

Scenario

Assume that the traffic to an application deployed in a CCE cluster changes constantly and there are significant peaks and troughs. If the number of nodes is always kept based on the peak traffic, resources will be idle, and money will be wasted. To reduce costs, you can configure an HPA policy and install the CCE Cloud Bursting Engine for CCI to leverage serverless container resources. CCI features ultra-fast scaling and pay-per-use billing and can effectively handle scenarios where traffic changes greatly, reducing costs and improving efficiency.

Example Description

After the Cloud Native Cluster Monitoring add-on and the CCE Cloud Bursting Engine for CCI add-on are installed in the CCE cluster, create a workload in the cluster and configure an HPA policy that can forcibly schedule the workload to CCI 2.0. Then use hey to construct requests to access the workload and simulate traffic changes and observe the auto scaling of the workload.

Constraints

• Monitoring add-on dependency: HPA triggering highly depends on container monitoring metrics. When a workload is elastically scheduled to a CCI virtual node through the CCE Cloud Bursting Engine for CCI add-on, the native metrics-server of the Kubernetes cluster may fail to collect all pod metrics on the virtual node. You need to install the Cloud Native Cluster Monitoring add-on on the Add-ons page of the CCE cluster to ensure that the HPA policy can accurately obtain the CPU and memory usage of all local and CCI pods.
• Network connectivity: Ensure that the subnet of the VPC where the CCE cluster resides does not conflict with the subnet planned for CCI and the required VPC endpoints have been correctly configured. Otherwise, pods scheduled to CCI will fail to pull images or be added to the load balancer's backend server group.

Prerequisites

• You have created a CCE cluster and created nodes in the cluster.
• You have connected to the cluster using kubectl or CloudShell.
• You have installed the Cloud Native Cluster Monitoring add-on in the CCE cluster, set the deployment mode to Local Data Storage, and enabled the Metrics API.

Procedure

1. Log in to the CCE console.
2. Click the cluster name to access the cluster console. In the navigation pane, choose Add-ons. Select the CCE Cloud Bursting Engine for CCI add-on and click Install.

   

   • Select the add-on version. The latest version is recommended. Configure the specifications as needed. If you select Preset, CCE will configure the number of pods and resource quotas for the add-on based on the preset specifications. You can see the configurations on the console.
   • If the Networking option is enabled, pods in the CCE cluster can communicate with pods in CCI through Services. The Proxy component will be automatically deployed upon add-on installation. For details, see Networking.
   • Configure CCI Resource Pool Default Subnet and CCI 2.0 Resource Pool (bursting-node) Extended Subnet (Optional).

3. In the navigation pane, choose Workloads. Then create a Deployment using YAML. The following is an example YAML file:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       version: v1
     name: n1
     namespace: default
   spec:
     selector:
       matchLabels:
         app: n1
         version: v1
     template:
       metadata:
         labels:
           app: n1
           version: v1
           bursting.cci.io/burst-to-cci: 'enforce' # Configure a forcible scheduling policy to schedule pods to CCI.
       spec:
         containers:
           - name: container-1
             image: nginx:latest    # Specify a container image. If you use an open-source image, enter the image name. If you use an image in My Images, obtain the image path from SWR.
             resources:
               requests:
                 cpu: 500m
                 memory: 1Gi
               limits:
                 cpu: 500m
                 memory: 1Gi
         imagePullSecrets:       
           - name: default-secret
   ---

4. In the navigation pane, choose Services. Then create a Service using YAML. The following is an example YAML file:

   apiVersion: v1
   kind: Service
   metadata:
     name: n1
     namespace: default
     labels:
       app: n1
       version: v1
   spec:
     ports:
       - name: cce-service-0
         protocol: TCP    # Protocol used for accessing the Service. The value can be TCP or UDP.
         port: 80         # Port for accessing a Service.
         targetPort: 80   # Port used by a Service to access the target container. This port is closely related to the application running in the container created in step 2. In this example, the nginx image uses port 80 by default.
     selector:
       app: n1
       version: v1
     type: ClusterIP

5. In the navigation pane, choose Policies. Then create an HPA policy using YAML. The following is an example YAML file:

   kind: HorizontalPodAutoscaler
   apiVersion: autoscaling/v2
   metadata:
     name: hpa-n1
     namespace: default
     annotations:
       extendedhpa.metrics: '[{"type":"Resource","name":"cpu","targetType":"Utilization","targetRange":{"low":"45","high":"56"}}]'
   spec:
     scaleTargetRef:
       kind: Deployment
       name: n1
       apiVersion: apps/v1
     minReplicas: 1
     maxReplicas: 10
     metrics:
       - type: Resource
         resource:
           name: cpu
           target:
             type: Utilization          
             averageUtilization: 50

6. Use hey to construct requests to access the created Service for a stress test. In the following command, 30 million requests will be sent from 300 clients concurrently.

   hey -n 30000000 -c 300 "http://$<service_ip>/"

   $<service_ip>: indicates the ClusterIP of the Service created in 4.

   

   

   In this example, hey is used as the stress testing tool. You can also use other stress testing tools. Before starting the stress testing, ensure that the network environment where the tool is running can access the CCE cluster.

7. Observe the HPA and workload status.

   Before the stress testing:

   

   

   After the stress testing:

   

   

Common Issues

• Why are CCI pods not scaled in immediately after the service traffic drops?

  Symptom: Even though HPA metrics drop immediately after a stress test ends or traffic decreases, CCI pods remain running for a while before being deleted.

  Cause: This is the default protection mechanism (scale-in cooldown period) of Kubernetes HPA. To prevent thrashing (frequent pod creation and deletion) caused by instantaneous jitter of monitoring metrics, the system enforces a default 5-minute (300-second) observation period before triggering scale-in.

  Conclusion: It is normal for the pods to remain running during this period. After the cooldown period ends, the system automatically deletes the CCI pods and stops billing by the second.

• How are specifications for pods that are scheduled to CCI billed? How can I avoid unnecessary expenditures?

  Specification upgrade: CCI has fixed, standard specifications, such as 0.25C1G, 0.5C2G, and 1C4G. When a CCE pod is elastically scheduled to CCI, the system takes its highest declared requests and limits from the YAML file, then upgrades them to the nearest standard CCI specifications for billing.

  Configuration suggestion: When configuring HPA, you must declare requests (the calculation base for HPA scaling) in the YAML file. To avoid unexpected specification increases when a CCE pod is scheduled to CCI, set the requests and limits to the same value or ensure that the maximum value is equal to a specific CCI standard specification, for example, 0.25C1G or 1C2G.

  Example: If you configure only requests.cpu: 200m and do not specify the limits, you will be billed for 0.25 vCPUs (the vCPUs in the nearest specifications) after the pod is scheduled to CCI.

• Advanced practice: How do I avoid temporary access failures (502) during scale-in?

  Scenario: When a CCI pod is scaled in and deleted, the backend service process may exit immediately. However, if the pod's IP address has not deregistered from the load balancer yet, any remaining traffic routed to that IP address will fail.

  Suggestion: In a hybrid elastic architecture, configure graceful shutdown for workloads. Add a preStop hook in the Deployment lifecycle (for example, sleep 15). In this way, after receiving a termination signal, the container waits 15 seconds before exiting. This delay gives the load balancer enough time to asynchronously remove the pod's IP address, ensuring service continuity during scale-in.