Priority-based Preemptive Scheduling on Virtual GPUs

After this function is enabled, the Volcano Scheduler supports preemption of different types of resources across GPUs. This means that high-priority tasks can preempt resources of the same or different types of tasks on different GPUs.

1. Log in to the CCE console and click the cluster name to access the cluster console. The Overview page is displayed.
2. In the navigation pane, choose Settings. Then, click the Scheduling tab. In the Default Cluster Scheduler area, select Volcano for Default Scheduler and click Try Now in Expert mode.
3. In the window that slides out from the right, set enable_topology_aware_preemption to true to enable preemption of different types of resources across GPUs.

   ...
   descheduler_enable: 'false'
   deschedulingInterval: 10m
   enable_scale_in_score: true
   enable_topology_aware_preemption: true  # Enable topology-aware preemption.
   enable_workload_balancer: false
   oversubscription_method: nodeResource
   ...

4. Click Confirm Settings in the lower right corner. In the displayed dialog box, confirm the modification and click Save.

Assume that there is a node in the cluster and the node has a GPU card with 8 GiB of memory. There are two workloads priority-low-gpu and priority-high-gpu. Both of them request 8 GiB of memory. Initially, priority-low-gpu is deployed in the cluster and consumes all available GPU resources. When priority-high-gpu is deployed, the scheduler detects the priority difference and evicts priority-low-gpu to free up resources. As a result, priority-high-gpu is scheduled and runs successfully, while priority-low-gpu enters a pending state. Once GPU resources become available again, priority-low-gpu is rescheduled.

1. Use kubectl to access the cluster.
2. Create a YAML file for a low-priority workload. You can enter a file name as needed. If no priority is specified, the default value is 0.

   vim low-priority.yaml

   Example file content:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: priority-low-gpu
     namespace: default
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: priority-low-gpu
     template:
       metadata:
         labels:
           app: priority-low-gpu
       spec:
         schedulerName: volcano
         containers:
         - image: <your_image_address>     # Replace it with your image address.
           name: container-0
           resources:
             requests:
               cpu: 250m
               memory: 512Mi
               volcano.sh/gpu-mem.128Mi: '64' # Request 8 GiB of memory. It must be the same as the value of limits.volcano.sh/gpu-mem.128Mi.
             limits:
               cpu: 250m
               memory: 512Mi
               volcano.sh/gpu-mem.128Mi: '64' # Request 8 GiB of memory. The value is a multiple of 128 MiB, that is, 64 × 128/1024 = 8.
         imagePullSecrets:
         - name: default-secret

3. Create the workload.

   kubectl apply -f low-priority.yaml

   If information similar to the following is displayed, the workload has been created:

   deployment.apps/priority-low-gpu created

4. Check whether the pod has been created.

   kubectl get pod -n default

   If information similar to the following is displayed, the pod has been created, and the GPU memory on the GPU node has been used up:

   NAME                               READY   STATUS    RESTARTS   AGE
   priority-low-gpu-6bdb4d7cb-pmtc2   1/1     Running   0          21s

5. Create a YAML file for a high-priority workload.

   vim high-priority.yaml

   Example file content:

   apiVersion: scheduling.k8s.io/v1  # Create a high-priority priority class. The value is 1000.
   description: high priority
   kind: PriorityClass
   metadata:
     name: high-priority
   preemptionPolicy: PreemptLowerPriority
   value: 1000        # A larger value indicates a higher priority.
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: priority-high-gpu
     namespace: default
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: priority-high-gpu
     template:
       metadata:
         labels:
           app: priority-high-gpu
       spec:
         priorityClassName: high-priority  # Use the high-priority priority class.
         schedulerName: volcano
         containers:
         - image: <your_image_address>     # Replace it with your image address.
           name: container-0
           resources:
             requests:
               cpu: 250m
               memory: 512Mi
               volcano.sh/gpu-mem.128Mi: '64' # Request 8 GiB of memory.
               volcano.sh/gpu-core.percentage: '50' # Request 50% of the compute.
             limits:
               cpu: 250m
               memory: 512Mi
               volcano.sh/gpu-mem.128Mi: '64' # Request 8 GiB of memory.
               volcano.sh/gpu-core.percentage: '50' # Request 50% of the compute.
         imagePullSecrets:
         - name: default-secret

6. Create the workload.

   kubectl apply -f high-priority.yaml

   If information similar to the following is displayed, the workload has been created. As the GPU memory is exhausted, the cluster cannot schedule the high-priority workload. In such cases, resource preemption is triggered. The lower-priority pods are evicted to free up resources, allowing high-priority pods to be scheduled. This preemption typically takes approximately 30 seconds.

   deployment.apps/priority-high-gpu created

7. Check whether the resource preemption is successful.

   kubectl get pod -n default

   If information similar to that shown below is displayed, it indicates that resource preemption is successful. Once sufficient GPU resources become available, the scheduler will reschedule the priority-low-gpu-6bdb4d7cb-pmtc2 pod.

   NAME                                READY   STATUS    RESTARTS   AGE
   priority-low-gpu-6bdb4d7cb-pmtc2    1/1     Pending   0          21s
   priority-high-gpu-8adb4d6bd-stka1   1/1     Running   0          21s