Elastic Scaling of CCE Pods to CCI

Constraints

• Only CCE standard clusters using VPCs and CCE Turbo clusters (using virtual-kubelet 1.2.5 or later) are supported. Arm clusters are not supported. Add-on instances will not be deployed on Arm nodes, if any, running in the cluster.

• CCE pods that are scheduled to CCI supports ConfigMap, Secret, EmptyDir, DownwardAPI, Projected, and PersistentVolumeClaims volumes, and the DownwardAPI and Projected volumes can only be used in virtual-kubelet 1.3.25 and later versions.
  • emptyDir: Subpaths are not supported.
  • PersistentVolumeClaims: Only SFS and SFS Turbo cloud storage types and CSI storage classes are supported.
  • Projected: If a source of the serviceAccountToken type is configured, the token in the corresponding service-account-token secret is mounted after the pod is scheduled to CCI. The token is valid for a long time and has no expected audience. That is, the expirationSeconds and audience configurations do not take effect.
• DaemonSets and pods that use the HostNetwork mode cannot be scheduled to CCI.
• Pods deployed across CCE and CCI can only communicate through ClusterIP Services. CCE ClusterIP Services cannot be accessed in init-container.
• When you interconnect pods deployed across CCE and CCI with a LoadBalancer Service or ingress:
  1. Do not specify the health check port. In a CCE cluster, CCI containers and CCE containers use different backend ports registered with ELB. If you specify a health check port, some backend health checks will be abnormal.
  2. Ensure that the health check method will not impact service access if different clusters use a Service to connect to the listener of the same ELB load balancer.
  3. Allow traffic from the container port for 100.125.0.0/16 in the node security group when you interconnect pods deployed across CCE and CCI with a shared LoadBalancer Service or ingress.
  4. Use either the auto or enforce scheduling policy.
• The subnet where the cluster is located cannot overlap with 10.247.0.0/16. Otherwise, the subnet conflicts with the Service CIDR block in the CCI namespace.
• Before using virtual-kubelet, go to the CCI console to grant CCI with the permission to use CCE.
• After the virtual-kubelet add-on is installed, a namespace named "cce-burst-cluster ID" is created in CCI and managed by the add-on. Do not use this namespace when manually creating pods in CCI.
• The resource usage of the add-on varies according to the service volume scaled to CCI. The pods, secrets, ConfigMaps, PVs, and PVCs requested by the service occupy VM resources. You are advised to evaluate the service usage and apply for VMs based on the following specifications: For 1000 pods and 1000 ConfigMaps (300 KB), nodes with 2 vCPUs and 4 GiB memory are recommended. For 2000 pods and 2000 ConfigMaps, nodes with 4 vCPUs and 8 GiB memory are recommended. For 4000 pods and 4000 ConfigMaps, nodes with 8 vCPUs and 16 GiB memory are recommended.
• If scheduling the resources scaled to CCI failed, the virtual-kubelet node will be locked for half an hour, during which the resources cannot be scheduled to CCI. You can use kubectl to check the status of the virtual-kubelet node on the CCE cluster console. If the node has been locked, manually unlock it.

  

• The restrictions on using log management to collect the logs of pods scaled to CCI are as follows:
  • Only logs in container file paths can be collected.
  • Networking must be enabled when the add-on is installed because Service networking of virtual-kubelet is required.
  • Pods scaled to CCI do not support hot update of log policies. After a log policy is updated, redeploy the pods scaled to CCI to apply the modification.
  • Log collection consumes pod memory. It is a good practice to reserve 50 MB of memory for a pod to be associated with one log policy. If the pod is associated with multiple log policies, reserve additional 5 MB of memory each time a log policy is associated.
  • A log larger than 250 KB cannot be collected.
  • Logs cannot be collected from specified mount directories such as the system, device, cgroup, and tmpfs directories.
  • The names of the to-be-collected log files that are associated with the same log policy in the same container must be unique. If there are multiple files with the same name, the collector collects only the first one it detects.
  • If the name of a log file exceeds 190 characters, the log file will not be collected.

Calculation and Constraints on the Specifications of a Pod

There are some restrictions on pod specifications.

1. The number of vCPUs in a pod must be greater than 0.
2. After automatic resource adjustment, the number of vCPUs in the pod is 48 or 64 or ranges from 0.25 to 32, the memory is an integer multiple of 1 GiB and ranges from 1 GiB to 512 GiB, and the ratio of CPU to memory ranges from 1:2 to 1:8.

Automatic Resource Adjustment

If pods to be scaled to CCI do not meet the CCI specifications and the configured specifications are not higher than 32 vCPUs and 256 GiB of memory, virtual-kubelet will automatically adjust pod resources.

The rules are as follows:

1. Increase the number of vCPUs of each application container and initialization container (except BestEffort containers) in the pod to an integer multiple of 0.25 and set the memory to a value greater than or equal to 0.2 GiB.
2. Continuously adjust pod resources until the number of vCPUs of the pod is greater than 32 or the memory is greater than 256 GiB.
3. Increase the memory of the pod to an integer multiple of 1 GiB.
4. If the ratio of pod memory/vCPUs is smaller than 2, increase the pod memory to a value that is greater than or equal to twice the number of vCPUs as well as an integer multiple of 1 GiB. If the ratio of pod memory/vCPUs is greater than 8, increase the number of pod vCPUs to be greater than or equal to 1/8 of the memory as well as an integer multiple of 0.25.
5. The vCPU and memory increments caused by the adjustment are added to the first non-BestEffort application container.

Installing the Add-on

1. On the CCE console, click the cluster name to access the cluster. Click Add-ons in the navigation pane, locate virtual-kubelet on the right, and click Install.
2. In the Install Add-on page, select Networking to enable pod communications between CCE and CCI through Kubernetes Services.
   Figure 1 Selecting networking
   

3. Click Install.
   

   After networking is selected, a sidecar is injected into the pod running on CCI to support Service access. The number of running containers queried is one more than the defined number, which is normal.

Using Labels to Set Scaling Policies

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test
  namespace: default
  labels:
    virtual-kubelet.io/burst-to-cci: 'auto'    # Scale pods to CCI.
spec:
  replicas: 2
  selector:
    matchLabels:
      app: test
  template:
    metadata:
      labels:
        app: test
    spec:
      containers:
        - image: 'nginx:perl'
          name: container-0
          resources:
            requests:
              cpu: 250m
              memory: 512Mi
            limits:
              cpu: 250m
              memory: 512Mi
          volumeMounts: []
      imagePullSecrets:
        - name: default-secret

Add the following field to labels of a workload or pod in the YAML file:

virtual-kubelet.io/burst-to-cci: "auto"

The following values are supported:

• auto: The system automatically determines whether to scale pods to CCI based on the actual scoring result of the scheduler. TaintToleration preferentially schedules pods to CCE nodes.
• localPrefer: Pods are scheduled to CCI when cluster resources are insufficient.
• enforce: Pods are forcibly scheduled to CCI.
• off: Pods are not scheduled to CCI.

Using Profiles to Manage On- and Off-Cloud Pods

You can use profiles to configure and manage pods, associate profiles with pods using labelSelector, and configure the allocation policy of the associated pods to allocate or limit the number of on- and off cloud pods.

Constraints

• You can use the profile to configure and manage the pod policy. burst-to-cci is still compatible and is prior to profile.
• localPrefer cannot be set to local and cci at the same time.
• The auto and localPrefer policies can be associated with pods that have not been associated with profiles. The enforce policy cannot be associated with pods that have not been associated with profiles.
• Currently, when the localPrefer policy is configured in the profile, the configuration of the maximum number of local may become invalid in extreme scenarios to avoid global problems.
• When a Deployment is performing rolling upgrade, you are advised to set maxSurge to a value as small as possible (for example, 0) to prevent the scheduling volume of the region where maxNum is limited from being less than expected.
• A pod can be associated with only one profile, that is, the profile with the highest association degree. After a pod is created, if the label of the pod is modified, the pod does not match the original profile. In this case, the pod selects the profile with the highest association degree for association. You can evaluate the association degree according to the following instructions:
  • Calculate the sum of matchLabels and matchExpressions in labelSelector based on obejectLables in the profile. The profile with the largest sum has the highest association degree.
  • If two profiles are filtered out, select the one whose name has the smallest alphabetical order.
• log-agent cannot be used to collect workload logs managed by the profile.

Usage Description

apiVersion: scheduling.cci.io/v1
kind: ScheduleProfile
metadata:
  name: test-cci-profile
  namespace: default
spec:
  objectLabels:
    matchLabels:
      app: nginx
  strategy: localPrefer
  location:
    local: 
      maxNum: 20 # Currently, maxNum cannot be configured for local and cci at the same time.
      scaleDownPriority: 10
    cci: {}
status:
  phase: initialized
  restrict:
    local: 20 # restrict is configured based on location. That is, local and cci do not appear at the same time.
  used:
    local: 20
    cci: 0

Configuring cci maxNum and scaleDownPriority

apiVersion: scheduling.cci.io/v1
kind: ScheduleProfile
metadata:
  name: test-cci-profile
  namespace: default
spec:
  objectLabels:
    matchLabels:
      app: nginx
  strategy: localPrefer
  location:
    local: {}
    cci:
      maxNum: 20 # maxNum can be configured either for local or cci.
      scaleDownPriority: 10
status:
  phase: initialized
  restrict:
    cci: 20 # restrict is configured based on location. That is, local and cci do not appear at the same time.
  used:
    local: 0
    cci: 20

Uninstalling the Add-on

1. Log in to the CCE console, and switch to the cluster. In the navigation pane, choose Add-ons. On the Add-ons Installed tab page, click Uninstall under virtual kubelet.
2. In the dialog box displayed, click Yes to uninstall the add-on. (Uninstalling the add-on will clear all resources on CCI to ensure that no extra fees are incurred by residual resources.)

• When the virtual-kubelet add-on is uninstalled, a job is started in the cluster to clear resources. Therefore, ensure that at least one node in the cluster can be scheduled. If the add-on cannot be uninstalled because no node can be scheduled, uninstall the add-on again after a node can be scheduled.
• If you delete a cluster without uninstalling the virtual-kubelet add-on, resources on CCI will not be automatically cleared and may incur extra fees. Either of the following operations helps avoid that:
  • Uninstall the virtual-kubelet add-on before deleting the cluster.
  • After deleting the cluster, log in to the CCI console and delete the namespace named cce-burst-${CLUSTER_ID}.
  • When a cluster is hibernated, the running containers on CCI will not be automatically stopped. Instead, their running and billing continue. Therefore, if you want these containers to stop, ensure that the number of containers scaled out in CCI is reduced to 0 before the cluster is hibernated.