Multi-dimensional and Multi-tier Network Topology-aware Scheduling

Background

In large-scale training jobs, it is common to split a big job into multiple shards, such as shards of model parallelism and replica groups of data parallelism. Compared with the communication between shards, each shard has higher requirements for the internal network communication performance.

In distributed inference tasks, there may also be subtasks with different roles, such as the common prefill and decode roles, and there are frequent communication between tasks of these roles.

In large model training or distributed inference scenarios, the entire training or inference task consumes a large number of resources, which cannot be met within a single network performance domain (hypernode). Although constrained network topology-aware scheduling for the entire task can ensure that the task is deployed in multiple specified network performance domains, it cannot ensure that each subtask is deployed in the same network performance domain. As a result, subtasks may be deployed across network performance domains, significantly affecting communication efficiency.

To address this, Volcano provides multi-dimensional and multi-tier network topology-aware affinity scheduling based on network topology-aware scheduling. During scheduling, Volcano detects the network topology of the cluster and preferentially schedules defined subtasks to the same network performance domain. This minimizes the internal communication overhead of subtasks while meeting the overall resource requirements of the task. This type of scheduling effectively addresses network communication performance issues in large-scale training and distributed inference jobs.

Function Introduction

Volcano provides a task grouping mechanism and allows you to configure independent network topology constraints for different groups based on the network topology constraints of the task.

After network topology-aware scheduling is enabled, the scheduler complies with both the network topology constraints of each subtask and the network topology constraints of the task. In this way, refined topology affinity scheduling for each subtask is implemented while meeting the overall network communication performance requirements of the job.

Volcano provides the capability of defining groups of pods within a task through the PodGroup resource object. By configuring the subGroupPolicy field, you can divide the pods in a PodGroup into multiple subgroups. In subGroupPolicy, network topology constraints can be independently configured for each subgroup. During scheduling, each subgroup complies with its own network topology constraints.

You can set subGroupPolicy for the associated PodGroups to implement multi-dimensional and multi-tier network topology-aware affinity scheduling for the task using Volcano. Volcano jobs and Kthena ModelServing workloads have built-in group definitions and network topology constraints. They can automatically create and manage PodGroups and comply with network topology constraints during scheduling.

apiVersion: scheduling.volcano.sh/v1beta1
kind: PodGroup
metadata:
  name: pg-test1
  namespace: default
spec:
  networkTopology:                           # Configure network topology constraints for PodGroups.
      mode: hard
      highestTierAllowed: 2
  subGroupPolicy:                          
  - name: subgroup-test              
    subGroupSize: 2                          # Number of subgroup pods
    networkTopology:                         # Configure network topology constraints for subgroups.
      mode: hard
      highestTierAllowed: 1
    labelSelector:                           # Labels that pods in a group need to match
       matchLabels:
          task-type: test  
    matchLabelKeys:                          # Criteria for grouping pods. Pods with the same value are grouped into the same subgroup.
    - group      

Network Topology Constraint Configuration

In a Volcano job, you can configure the networkTopology field to describe the network topology constraints during job deployment. This ensures that job pods are scheduled to the optimal performance domain that meets the specified conditions, significantly improving computing efficiency and communication performance. The specific constraint configuration is as follows.

spec:
  networkTopology:
    mode: hard
    highestTierAllowed: 2

Prerequisites

The Volcano Scheduler add-on must be installed in the cluster, and the add-on version must be 1.21.1 or later.

Constraints

• Network topology-aware scheduling does not support preemption. When resources are insufficient, the scheduler does not proactively preempt the resources that have been allocated to pods.
• If a group affinity policy is configured, network topology constraints in soft mode are not supported.
• Hyperjob workloads do not support network topology-aware scheduling. Even if networkTopology or group affinity policies are configured in the JobTemplate of a hyperjob, these configurations will be ignored and do not take effect.
• The hypernode auto discovery does not support clusters with multiple network topologies (for example, a cluster with both A2 and A3 topologies).
• You are advised not to change the network topology configuration items when the job is running. This is because such changes may cause the hypernode structure in the cluster to be rebuilt, which in turn may cause load fluctuations or scheduling exceptions in the Volcano Scheduler process.
• If the network topology structure or configuration items of a cluster are changed during scheduling, Volcano cannot ensure the correctness of the scheduling result.

Enabling Network Topology

1. Log in to the CCE console and click the cluster name to access the cluster console.
2. In the navigation pane, choose Settings and click the Scheduling tab. Select Volcano, find the expert mode, and click Try Now.

   

3. Go to the expert mode page and enable the network-topology-aware add-on in the YAML file to implement network topology-aware scheduling.

   ...
   default_scheduler_conf:
     actions: allocate, backfill, preempt
     tiers:
     - plugins:
       - name: priority
       - enableJobStarving: false
         enablePreemptable: false
         name: gang
       - name: conformance
     - plugins:
       - enablePreemptable: false
         name: drf
       - name: predicates
       - name: nodeorder
     - plugins:
       - name: cce-gpu-topology-predicate
       - name: cce-gpu-topology-priority
       - name: xgpu
          name: network-topology-aware # Enable the network-topology-aware plugin.
         weight: 10 # The overall policy weight of the plugin
     - plugins:
       - name: nodelocalvolume
       - name: nodeemptydirvolume
       - name: nodeCSIscheduling
       - name: networkresource
   ...

   Modify the default_controller_conf configuration to enable the hypernode auto discovery function. Configure other parameters based on your requirements.

   ...
   default_controller_conf:
     networkTopologyDiscovery:
       - source: label # Name of the hypernode auto discovery plugin, which cannot be changed
         enabled: true # Whether to enable the hypernode auto discovery plugin. true indicates that the plugin is enabled, and false indicates that the plugin is disabled (existing hypernode instances will not be deleted).
         config:
           networkTopologyTypes:
             testtopology:                        # Name of the network topology hierarchy. Up to 20 characters are allowed. Only one topology can be configured.
               - nodeLabel: test-tier-3             # Node label name corresponding to the tier-3 network topology. That means the hypernode tier is 3.
               - nodeLabel: test-tier-2             # Node label name corresponding to the tier-2 network topology. That means the hypernode tier is 2.
               - nodeLabel: test-tier-1             # Node label name corresponding to the tier-1 network topology. That means the hypernode tier is 1.
               - nodeLabel: kubernetes.io/hostname # Node label name corresponding to the tier-0 network topology (node level). The label name can only be matched based on the node name and cannot be modified.

   Click Save.

4. Click Confirm Settings in the lower right corner. In the displayed dialog box, confirm the modification and click Save.

Implementing Multi-dimensional and Multi-tier Network Topology-aware Affinity Scheduling

To use multi-dimensional and multi-tier network topology-aware affinity scheduling, you need to define the hypernodes and network topology based on the cluster network environment. For details about how to define and generate hypernodes, see Network Topology-aware Scheduling.

The following example uses a two-tier network topology.