Help Center/ Cloud Container Engine/ User Guide/ Scheduling/ Volcano Scheduling/ Heterogeneous Resource Network Topology-aware Scheduling/ Multi-dimensional Gang Scheduling
Updated on 2026-05-26 GMT+08:00
View PDF
Share

Multi-dimensional Gang Scheduling

Background

Gang scheduling is a scheduling algorithm that schedules correlated processes or threads to run simultaneously on different processors. It meets the scheduling requirements of "All or nothing" in the scheduling process and avoids the waste of cluster resources caused by arbitrary scheduling of pods. Gang is mainly used in scenarios that require multi-process collaboration, such as AI and big data scenarios. Gang scheduling effectively resolves pain points such as resource waiting or deadlocks in distributed training jobs, thereby significantly improving the utilization of cluster resources.

As AI tasks evolve, they can be further divided into subtasks in different dimensions, such as multiple shards in a distributed training job or different roles in a distributed inference job, so new multi-dimensional pod gang scheduling is required within and between subtasks. Multiple pods in a subtask can be started only after the minimum resources are met. This prevents the subtask from running invalidly when only some pods are started. When there are multiple subtasks, the task can run normally only after a certain number or proportion of subtasks are started. To handle this, Volcano provides multi-dimensional gang scheduling. During scheduling, the gang scheduling capability is provided for tasks in different dimensions to meet the overall resource scheduling requirements of these tasks.

Function Introduction

Multi-dimensional gang scheduling provides the following capabilities:

  • Task-level gang scheduling

    During scheduling, the system checks whether the number of scheduled pods in a task reaches the minimum number of running pods. Pods in a task are scheduled to the cluster only when the minimum number of running pods in the task is met. Otherwise, the task will not be scheduled.

  • Subtask-level gang scheduling

    Each subtask must comply with the gang scheduling principle. A subtask is scheduled only when all pods in the subtask meet the gang scheduling conditions. Otherwise, the subtask will not be scheduled. When cluster resources are insufficient, subtasks must be scheduled as a whole to maintain the integrity of subtasks and prevent subtasks from running invalidly due to insufficient pods, thereby avoiding resource waste.

  • Gang scheduling between subtasks

    During scheduling, the number of subtasks of different types that can be scheduled in a task is monitored. Pods of the task can be scheduled only when the minimum number of subtasks that can be scheduled is met. Otherwise, task scheduling will not be performed. When cluster resources are insufficient, elastic scheduling by subtask can be performed to ensure that different types of subtasks started in a task meet the minimum deployment quantity and proportion requirements, ensuring effective task running.

Volcano provides PodGroup to group the pods for a task. You can divide the pods within a PodGroup into multiple subgroups by configuring the required fields. In this process, you can define the number of pods for gang scheduling in each subgroup and the minimum number of subgroups of different types. During scheduling, each subgroup complies with its own gang scheduling constraints. You can set PodGroups for the tasks and use Volcano to perform multi-dimensional gang scheduling for the workloads.

apiVersion: scheduling.volcano.sh/v1beta1
kind: PodGroup
metadata:
  name: pg-test1
  namespace: default
spec:
  subGroupPolicy:                          
  - name: subgroup-test
    subGroupSize: 2                          # The minimum number of pods that can be scheduled in a subgroup
    minSubGroups: 4                          # The minimum number of subgroups that can be scheduled
    labelSelector:                           # Labels that pods in a group need to match
        matchLabels:
          task-type: test  
    matchLabelKeys:                          # Criteria for grouping pods. Pods with the same label value are grouped into the same subgroup.
    - group                                             
  minMember: 8                    # The minimum number of pods that can be scheduled in a PodGroup

The involved parameters are described as follows:

  • minMember: specifies the minimum number of pods for running a workload.
  • subGroupPolicy: specifies the grouping policy, which is used to group pods in a job into different subgroups.
    • name: specifies name of the grouping policy.
    • subGroupSize: specifies the minimum number of pods in a subgroup.
    • minSubGroups: specifies the minimum number of subgroups in the current grouping policy.
    • labelSelector: specifies the labels that the grouped pods need to match.
    • matchLabelKeys: specifies the criteria for grouping pods. Pods are grouped based on the values of their labels. Pods with the same label value are grouped into the same subgroup.

Volcano provides the partition definition and gang scheduling constraint definition in the job definition. Volcano automatically creates and manages PodGroups and schedules Volcano jobs based on the multi-dimensional gang scheduling policy.

Prerequisites

The Volcano Scheduler add-on must be installed in the cluster, and the add-on version must be 1.21.1 or later.

Configuring Gang Scheduling

After Volcano is installed, you can enable or disable Gang scheduling on the Scheduling page. This function is enabled by default.

  1. Log in to the CCE console and click the cluster name to access the cluster console.
  2. In the navigation pane, choose Settings. Then click the Scheduling tab.
  3. Select Volcano Scheduler. In the AI task performance enhanced scheduling pane, select whether to enable Gang.

    Enabling this function helps enhance the cluster service throughput and improve service performance.

  4. Click Confirm.

    After the configuration, you can use gang scheduling in workloads or Volcano jobs.

Using Multi-dimensional Gang Scheduling for Workload Deployment

This following describes how to use the multi-dimensional gang scheduling capability in the provided scenarios.

Creating a Volcano Job

When creating a training task using a Volcano job, you can define different gang scheduling policies for training shards through the Volcano job for elastic deployment in different dimensions. Based on the total number of pods specified for the training task, you can use partitionPolicy to define the grouping policy for different partitions of the task and the minimum number of partitions to be started, ensuring that both the partitions and the pods in the partitions are scheduled. When creating a Volcano job, you only need to specify minAvailable and partitionPolicy. The Volcano scheduler will automatically create and manage PodGroups based on partitionPolicy. The following is an example:

apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
  name: sample
spec:
  schedulerName: volcano
  minAvailable: 5           # Optional. The minimum number of pods in a job (gang scheduling condition)
  tasks:
    - name: "ps"
      replicas: 2           # Mandatory. The number of pods in a task
      partitionPolicy:     # Optional. All pods in the current task are grouped.
        totalPartitions: 2 # Mandatory. The number of partitions into which a group of pods in a task is divided
        partitionSize: 1    # Mandatory. The number of pods in each partition (gang scheduling condition). The product of totalPartitions and partitionSize must be equal to the value of replicas.
        minPartitions: 1    # Optional. The gang scheduling condition for the number of partitions
      template:
        spec:
          containers:
            - image: busybox
              command: ['sh', '-c', 'echo "Hello, Kubernetes!" && sleep 3600']
              imagePullPolicy: IfNotPresent
              name: ps
              resources:
                requests:
                  cpu: "500m"
          restartPolicy: OnFailure
    - name: "worker"
      replicas: 8           # Mandatory. The number of pods in a task
      partitionPolicy:      # Optional. All pods in the current task are grouped. The network topology affinity scheduling capability is set based on the partitions.
        totalPartitions: 4  # Mandatory. The number of partitions into which a group of pods in a task is divided
        partitionSize: 2    # Mandatory. The number of pods in each group. The product of totalPartitions and partitionSize must be equal to the value of replicas.
        minPartitions: 2    # Optional. The gang scheduling condition for the number of partitions
      template:
        spec:
          containers:
            - image: busybox
              command: ['sh', '-c', 'echo "Hello, Kubernetes!" && sleep 3600']
              imagePullPolicy: IfNotPresent
              name: worker
              resources:
                requests:
                  cpu: "500m"
          restartPolicy: OnFailure

In the example, gang scheduling must meet the following three dimensions:

  • partitionSize: In a Parameter Server task, each partition can be scheduled only when at least one pod in the partition is available for scheduling. In a Worker task, each partition can be scheduled only when at least two pods in the partition are available for scheduling.
  • minPartitions: specifies the minimum number of partitions in each task. In a Parameter Server task, at least one partition must be available for scheduling. In a Worker task, at least two partitions must be available for scheduling.
  • minAvailable: specifies the minimum number of pods in a job. A job requires at least five pods to be scheduled.

If cluster resources are insufficient and only five pods can be scheduled, pod scheduling is preferentially performed based on the constraints of multi-dimensional gang scheduling to ensure that the number of pods scheduled for each partition meets the requirements. The Parameter Server task starts a partition, and the Worker task starts two partitions, meeting the minimum startup requirements of the entire task.

Run the kubectl get pods command to view the workload information. Information similar to the following is displayed:

NAME              READY   STATUS    RESTARTS   AGE
sample-ps-0       1/1     Running   0          34s
sample-ps-1       0/1     Pending   0          34s
sample-worker-0   1/1     Running   0          34s
sample-worker-1   1/1     Running   0          34s
sample-worker-2   1/1     Running   0          34s
sample-worker-3   1/1     Running   0          34s
sample-worker-4   0/1     Pending   0          61s
sample-worker-5   0/1     Pending   0          61s
sample-worker-6   0/1     Pending   0          61s
sample-worker-7   0/1     Pending   0          61s

Creating a Kthena ModelServing Inference Workload

When a distributed inference workload has both the prefill and decode roles, each role can have multiple pods. By default, the pods for a prefill or decode role must meet the gang scheduling capability. That means all pods in a role instance must either be scheduled or none of them are scheduled. The prefill and decode roles each can have multiple instances, which can be combined in different ways. For example, in a 4P2D scenario, resources can be allocated to workload pods only when the minimum requirement of 2P1D is met.

Kthena's ModelServing allows you to schedule inference workloads through Volcano. In ModelServing, you can directly define the gang scheduling requirements of inference workloads in different dimensions and implement multi-dimensional gang scheduling through Volcano. When creating a ModelServing workload, you only need to specify gangPolicy and Role. The ModelServing controller will automatically create and manage PodGroups based on the gangPolicy and Role definitions. The following is an example:

apiVersion: workload.serving.volcano.sh/v1alpha1
kind: ModelServing
metadata:
  name: sample
  namespace: default
spec:
  schedulerName: volcano
  replicas: 1
  template:
    restartGracePeriodSeconds: 60
    gangPolicy:
      minRoleReplicas:    # Optional. The gang scheduling condition for the minimum number of replicas for each role
        prefill: 2       
        decode: 2
    roles:
      - name: prefill
        replicas: 4       # Mandatory. The number of role replicas
        entryTemplate:
          spec:
            containers:
              - name: leader
                image: busybox
                command: ['sh', '-c', 'echo "Hello, Kubernetes!" && sleep 3600']
                resources:
                  limits:
                    cpu: "0.5"
                  requests:
                    cpu: "0.5"
        workerReplicas: 1    # Optional. The number of worker pods for the prefill role
        workerTemplate:
          spec:
            containers:
              - name: worker
                image: busybox
                command: ['sh', '-c', 'echo "Hello, Kubernetes!" && sleep 3600']
                resources:
                  limits:
                    cpu: "0.5"
                  requests:
                    cpu: "0.5"
      - name: decode
        replicas: 4  
        entryTemplate:
          spec:
            containers:
              - name: leader
                image: busybox
                command: ['sh', '-c', 'echo "Hello, Kubernetes!" && sleep 3600']
                resources:
                  limits:
                    cpu: "0.5"
                  requests:
                    cpu: "0.5"
        workerReplicas: 1    # Optional. The number of worker pods in a role replica
        workerTemplate:
          spec:
            containers:
              - name: worker
                image: busybox
                command: ['sh', '-c', 'echo "Hello, Kubernetes!" && sleep 3600']
                resources:
                  limits:
                    cpu: "0.5"
                  requests:
                    cpu: "0.5"

In the example, gang scheduling must meet the following two dimensions:

  • workerReplicas: specifies the minimum number of worker pods in each role replica. A role replica can be scheduled only when at least entryReplica (1) + workerReplicas pods are available.
  • minRoleReplicas: specifies the minimum number of replicas for each role. In the example, prefill and decode each require at least two replicas (2P2D) to be scheduled.

The minimum number of pods for the ModelServing workload is calculated as follows: Minimum number of role replicas × Number of role pods. You do not need to set this parameter separately.

When resources are insufficient, you can see that two replicas are started for the prefill role and two replicas for the decode role, and each replica contains two pods.

NAME                   READY   STATUS    RESTARTS   AGE
sample-0-decode-0-0    1/1     Running   0          8s
sample-0-decode-0-1    1/1     Running   0          8s
sample-0-decode-1-0    1/1     Running   0          8s
sample-0-decode-1-1    1/1     Running   0          7s
sample-0-decode-2-0    0/1     Pending   0          7s
sample-0-decode-2-1    0/1     Pending   0          7s
sample-0-decode-3-0    0/1     Pending   0          7s
sample-0-decode-3-1    0/1     Pending   0          7s
sample-0-prefill-0-0   1/1     Running   0          8s
sample-0-prefill-0-1   1/1     Running   0          8s
sample-0-prefill-1-0   1/1     Running   0          8s
sample-0-prefill-1-1   1/1     Running   0          8s
sample-0-prefill-2-0   0/1     Pending   0          8s
sample-0-prefill-2-1   0/1     Pending   0          8s
sample-0-prefill-3-0   0/1     Pending   0          8s
sample-0-prefill-3-1   0/1     Pending   0          8s
Parent Topic: Heterogeneous Resource Network Topology-aware Scheduling