Dynamic Resource Oversubscription

Features

After dynamic resource oversubscription and elastic scaling are enabled in a node pool, oversubscribed resources change rapidly because the resource usage of high-priority applications changes in real time. To prevent frequent node scale-ins and scale-outs, do not consider oversubscribed resources when evaluating node scale-ins.

Hybrid deployment is supported, and CPU and memory resources can be oversubscribed. The key features are as follows:

• Offline jobs preferentially run on oversubscribed nodes.

  If both oversubscribed and non-oversubscribed nodes exist, the former will score higher than the latter and offline jobs are preferentially scheduled to oversubscribed nodes.

• Online jobs can use only non-oversubscribed resources if scheduled to an oversubscribed node.

  Offline jobs can use both oversubscribed and non-oversubscribed resources of an oversubscribed node.

• In the same scheduling period, online jobs take precedence over offline jobs.

  If both online and offline jobs exist, online jobs are scheduled first. When the node resource usage exceeds the upper limit and the node requests exceed 100%, offline jobs will be evicted.

• CPU/Memory resources can be isolation by kernel.

  CPU isolation: Online jobs can quickly preempt CPU resources of offline jobs and suppress the CPU usage of the offline jobs.

  Memory isolation: When system memory resources are used up and OOM Kill is triggered, the kernel evicts offline jobs first.

• kubelet offline jobs obey the following admission rules:

  After the pod is scheduled to a node, kubelet starts the pod only when the node resources can meet the pod request (predicateAdmitHandler.Admit). kubelet starts the pod when both of the following conditions are met:

  • The total request of pods to be started and online running jobs < allocatable nodes
  • The total request of pods to be started and online/offline running job < allocatable nodes+oversubscribed nodes
• Resource oversubscription and hybrid deployment can be configured separately.

  Enabling hybrid deployment of a node pool also enables oversubscription by default. Nodes are then labeled with both volcano.sh/colocation="true" and volcano.sh/oversubscription="true". To use hybrid deployment for both online and offline jobs without oversubscription, simply disable oversubscription in hybrid deployment settings. This will remove the volcano.sh/oversubscription="true" label.

  The following table lists the features that can be used after hybrid deployment or oversubscription is enabled.

  Hybrid Deployment	Oversubscription	Oversubscription Resource	Scenario for Evicting Offline Pods
  No	No	No	No
  Yes	No	No	The actual resource usage of a node exceeds the upper limit.
  No	Yes	Yes	The actual resource usage of a node exceeds the upper limit and the pod requests on the node exceed 100%.
  Yes	Yes	Yes	The actual resource usage of a node exceeds the upper limit.

Compatible kubelet Oversubscription

Specifications

• Cluster version
  • v1.19: v1.19.16-r4 or later
  • v1.21: v1.21.7-r0 or later
  • v1.23: v1.23.5-r0 or later
  • v1.25 or later
• Cluster type: CCE standard or CCE Turbo
• Node OS: EulerOS 2.9 (kernel-4.18.0-147.5.1.6.h729.6.eulerosv2r9.x86_64) or Huawei Cloud EulerOS 2.0
• Node type: ECS
• Volcano version: 1.7.0 or later

Constraints

• Before enabling oversubscription, ensure that the overcommit add-on is not enabled on Volcano.
• Modifying the label of an oversubscribed node does not affect the running pods.
• Running pods cannot be converted between online and offline services. To convert services, rebuild pods.
• If the label volcano.sh/oversubscription=true is configured for a node in the cluster, the oversubscription configuration must be added to the Volcano add-on. Otherwise, the scheduling of oversold nodes will be abnormal. Ensure that you have correctly configure labels because the scheduler does not check the add-on and node configurations. For details, see Table 1.
• To disable oversubscription, perform the following operations:
  • Remove the volcano.sh/oversubscription label from the oversubscribed node.
  • Set over-subscription-resource to false.
  • Modify the configmap of Volcano Scheduler named volcano-scheduler-configmap and remove the oversubscription add-on.
• If you have set cpu-manager-policy to statically bind CPU cores on a node, do not assign the QoS class of Guaranteed to offline pods. This is because offline pods may occupy the CPUs of online pods, leading to an online pod startup failure and offline pods failing to start even though they have been successfully scheduled. To prevent this, switch the pods to online pods if CPU core binding is required.
• If cpu-manager-policy is set to static CPU core binding on a node, do not bind CPU cores to all online pods. This is because doing so can cause online pods to occupy all available CPU or memory resources, leaving only a small number of oversubscribed resources.

Deployment Example

The following uses an example to describe how to deploy online and offline jobs in hybrid mode.

1. Configure a cluster with two nodes, one oversubscribed and the other non-oversubscribed.

   # kubectl get node
   NAME           STATUS   ROLES    AGE    VERSION
   192.168.0.173   Ready    <none>   4h58m   v1.19.16-r2-CCE22.5.1
   192.168.0.3     Ready    <none>   148m    v1.19.16-r2-CCE22.5.1

   • 192.168.0.173 is an oversubscribed node (with the volcano.sh/oversubscription=true label).
   • 192.168.0.3 is a non-oversubscribed node (without the volcano.sh/oversubscription=true label).

   # kubectl describe node 192.168.0.173
   Name:               192.168.0.173
   Roles:              <none>
   Labels:             beta.kubernetes.io/arch=amd64
                       ...
                       volcano.sh/oversubscription=true

2. Submit offline job creation requests. If resources are sufficient, all offline jobs will be scheduled to the oversubscribed node.
   The offline job template is as follows:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: offline
     namespace: default
   spec:
     replicas: 2
     selector:
       matchLabels:
         app: offline
     template:
       metadata:
         labels:
           app: offline
         annotations:
           volcano.sh/qos-level: "-1"       # Offline job label
       spec:
         schedulerName: volcano             # Volcano is used.
         priorityClassName: volcano-free         # volcano-free priorityClass
         containers:
           - name: container-1
             image: nginx:latest
             imagePullPolicy: IfNotPresent
             resources:
               requests:
                 cpu: 500m
                 memory: 512Mi
               limits:
                 cpu: "1"
                 memory: 512Mi
         imagePullSecrets:
           - name: default-secret

   Offline jobs are scheduled to the oversubscribed node.

   # kubectl get pod -o wide
   NAME                      READY   STATUS   RESTARTS  AGE     IP             NODE 
   offline-69cdd49bf4-pmjp8   1/1    Running   0         5s    192.168.10.178   192.168.0.173
   offline-69cdd49bf4-z8kxh   1/1    Running   0         5s    192.168.10.131   192.168.0.173

3. Submit online job creation requests. If resources are sufficient, the online jobs will be scheduled to the non-oversubscribed node.
   The online job template is as follows:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: online
     namespace: default
   spec:
     replicas: 2
     selector:
       matchLabels:
         app: online
     template:
       metadata:
         labels:
           app: online
       spec:
         schedulerName: volcano                 # Volcano is used.
         priorityClassName: volcano-production          # volcano-production priorityClass
         containers:
           - name: container-1
             image: resource_consumer:latest
             imagePullPolicy: IfNotPresent
             resources:
               requests:
                 cpu: 1400m
                 memory: 512Mi
               limits:
                 cpu: "2"
                 memory: 512Mi
         imagePullSecrets:
           - name: default-secret

   Online jobs are scheduled to the non-oversubscribed node.

   # kubectl get pod -o wide
   NAME                   READY   STATUS   RESTARTS  AGE     IP             NODE 
   online-ffb46f656-4mwr6  1/1    Running   0         5s    192.168.10.146   192.168.0.3
   online-ffb46f656-dqdv2   1/1    Running   0         5s    192.168.10.67   192.168.0.3

4. Improve the resource usage of the oversubscribed node and observe whether offline job eviction is triggered.
   Deploy online jobs to the oversubscribed node (192.168.0.173).

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: online
     namespace: default
   spec:
     replicas: 2
     selector:
       matchLabels:
         app: online
     template:
       metadata:
         labels:
           app: online
       spec:
          affinity:                             # Submit an online job to an oversubscribed node.
           nodeAffinity:
             requiredDuringSchedulingIgnoredDuringExecution:
               nodeSelectorTerms:
               - matchExpressions:
                 - key: kubernetes.io/hostname
                   operator: In
                   values:
                   - 192.168.0.173
         schedulerName: volcano                 # Volcano is used.
         priorityClassName: volcano-production          # volcano-production priorityClass
         containers:
           - name: container-1
             image: resource_consumer:latest
             imagePullPolicy: IfNotPresent
             resources:
               requests:
                 cpu: 700m
                 memory: 512Mi
               limits:
                 cpu: 700m
                 memory: 512Mi
         imagePullSecrets:
           - name: default-secret

   Submit the online or offline jobs to the oversubscribed node (192.168.0.173) at the same time.

   # kubectl get pod -o wide
   NAME                     READY   STATUS   RESTARTS  AGE     IP             NODE 
   offline-69cdd49bf4-pmjp8  1/1     Running    0      13m   192.168.10.178   192.168.0.173 
   offline-69cdd49bf4-z8kxh  1/1     Running     0      13m   192.168.10.131   192.168.0.173 
   online-6f44bb68bd-b8z9p  1/1     Running     0     3m4s   192.168.10.18   192.168.0.173 
   online-6f44bb68bd-g6xk8  1/1     Running     0     3m12s   192.168.10.69   192.168.0.173

   Check the oversubscribed node with IP address 192.168.0.173. It is found that resources are oversubscribed, where there are 2343m CPU cores and 3073653200 MiB of memory. Additionally, the CPU allocation rate exceeded 100%.

   # kubectl describe node 192.168.0.173
   Name:              192.168.0.173
   Roles:              <none>
   Labels:              …
                       volcano.sh/oversubscription=true
   Annotations:         …                  
                       volcano.sh/oversubscription-cpu: 2343
                       volcano.sh/oversubscription-memory: 3073653200
                       …
   Allocated resources:
     (Total limits may be over 100 percent, i.e., overcommitted.)
     Resource               Requests      Limits
     --------               --------        ------
     cpu                    4750m (121%)  7350m (187%)
     memory                 3760Mi (61%)  4660Mi (76%)
                            …

   Increase the CPU usage of online jobs on the node. Offline job eviction is triggered.

   # kubectl get pod -o wide
   NAME                     READY   STATUS   RESTARTS  AGE     IP             NODE 
   offline-69cdd49bf4-bwdm7  1/1    Running   0       11m   192.168.10.208  192.168.0.3 
   offline-69cdd49bf4-pmjp8   0/1    Evicted    0       26m   <none>         192.168.0.173
   offline-69cdd49bf4-qpdss   1/1     Running   0       11m   192.168.10.174  192.168.0.3  
   offline-69cdd49bf4-z8kxh   0/1     Evicted    0       26m   <none>        192.168.0.173
   online-6f44bb68bd-b8z9p   1/1     Running   0       24m   192.168.10.18  192.168.0.173
   online-6f44bb68bd-g6xk8   1/1     Running   0       24m   192.168.10.69  192.168.0.173

Handling Suggestions

• After kubelet of the oversubscribed node is restarted, the resource view of Volcano Scheduler is not synchronized with that of kubelet. As a result, OutOfCPU occurs in some newly scheduled jobs, which is normal. After a period of time, Volcano Scheduler can properly schedule online and offline jobs.
• After online and offline jobs are submitted, you are not advised to dynamically change the job type (adding or deleting annotation volcano.sh/qos-level: "-1") because the current kernel does not support the change of an offline job to an online job.
• CCE collects the resource usage (CPU/memory) of all pods running on a node based on the status information in the cgroups system. The resource usage may be different from the monitored resource usage, for example, the resource statistics displayed by running the top command.
• You can add oversubscribed resources (such as CPU and memory) at any time.

  You can reduce the oversubscribed resource types only when the resource allocation rate does not exceed 100%.

• If an offline job is deployed on a node ahead of an online job and the online job cannot be scheduled due to insufficient resources, configure a higher priorityClass for the online job than that for the offline job.
• If there are only online jobs on a node and the eviction threshold is reached, the offline jobs that are scheduled to the current node will be evicted soon. This is normal.