Multi-tier Network Topology-aware Scheduling

Function Introduction

Distributed AI training/inference tasks are gradually becoming the core workloads of the computing infrastructure. However, due to their high dependency on data exchange between subtasks, they face bottlenecks in network transmission performance, making it difficult to significantly improve computing efficiency. The main cause of this issue is the complex network topology (typically involving multiple layers of switches) and diverse network types (such as IB, RoCE, and NVSwitch) in data centers. A longer communication path across multi-layer switches increases the latency, lowers the throughput, and reduces the efficiency. Conversely, a shorter communication path improves communication performance. Therefore, scheduling highly related workloads to the same high-performance domain can effectively reduce cross-switch communication costs, accelerate data exchange between nodes, and significantly improve the overall efficiency of distributed training/inference tasks.

To address this issue, Volcano defines a new CRD called hypernode to uniformly describe the multi-tier network topology of a cluster and mask network type differences. As shown in the figure, each hypernode represents a network topology performance domain, which can be mapped to a switch or ToR. Multiple hypernodes can be connected hierarchically to form a tree structure, accurately representing the complex network topology in a data center.

In this structure, the communication efficiency between nodes depends on the hypernode hierarchy span between them. In this example:

• node0 and node1 belong to hypernode0 (tier 1), with the highest communication efficiency.
• node0 and node2 belong to hypernode4 (tier 2), with reduced communication efficiency.
• node0 and node4 belong to hypernode6 (tier 3), with the lowest communication efficiency.

Compared with the traditional topology of using node labels to represent the network topology, hypernodes have the following advantages:

• Unified semantics: hypernodes provide a standardized way to describe network topology, avoiding the semantic inconsistency issues of the manual label method.
• Hierarchy: hypernodes support a tree-like hierarchical structure, allowing for more precise representation of the network topology of multi-layer switches.
• Easy management: You can manually create hypernodes or use automatic network topology discovery to maintain hypernodes for automatic synchronization and management of resource status.

To further reduce the burden of managing network topology information, Volcano provides hypernode auto discovery. Volcano can automatically discover the network topology in a cluster and automatically creates, updates, or deletes hypernode Custom Resources (CRs) based on the discovery results. The auto discovery function has the following core advantages:

• Automatic management: automatically discovers and maintains hypernode information from node network topology information labels, eliminating the need for manual operations.
• Real-time synchronization: synchronizes network topology changes (such as changes of node network topology information labels) in real time to ensure that the hypernode information is consistent with the actual network status.

This auto discovery mechanism allows you to focus on job scheduling configuration without worrying about the complexity of hypernode creation and maintenance, greatly simplifying the deployment and management of network topology-aware scheduling.

Network Topology Constraint Configuration

In a Volcano job, you can configure the networkTopology field to describe the network topology constraints during job deployment. This ensures that job pods are scheduled to the optimal performance domain that meets the specified conditions, significantly improving computing efficiency and communication performance. The specific constraint configuration is as follows.

spec:
  networkTopology:
    mode: hard
    highestTierAllowed: 2

Prerequisites

The Volcano Scheduler add-on must be installed in the cluster, and the add-on version must be 1.21.1 or later.

Constraints

• Network topology-aware scheduling does not support preemption. When resources are insufficient, the scheduler does not proactively preempt the resources that have been allocated to pods.
• If a group affinity policy is configured, network topology constraints in soft mode are not supported.
• Hyperjob workloads do not support network topology-aware scheduling. Even if networkTopology or group affinity policies are configured in the JobTemplate of a hyperjob, these configurations will be ignored and do not take effect.
• The hypernode auto discovery does not support clusters with multiple network topologies (for example, a cluster with both A2 and A3 topologies).
• You are advised not to change the network topology configuration items when the job is running. This is because such changes may cause the hypernode structure in the cluster to be rebuilt, which in turn may cause load fluctuations or scheduling exceptions in the Volcano Scheduler process.
• If the network topology structure or configuration items of a cluster are changed during scheduling, Volcano cannot ensure the correctness of the scheduling result.

How to Use the Add-on

1. Log in to the CCE console and click the cluster name to access the cluster console.
2. In the navigation pane, choose Settings and click the Scheduling tab. Select Volcano, find the expert mode, and click Try Now.

   

3. On the page for configuring the expert mode, modify default_controller_conf in the YAML file as follows to enable the hypernode auto discovery function: Configure the parameters based on your requirements.

   ...
   default_controller_conf:
     networkTopologyDiscovery:
       - source: label # Name of the hypernode auto discovery plugin, which cannot be changed
         enabled: true # Whether to enable the function. true indicates that the function is enabled, and false indicates that the function is disabled (existing hypernode instances will not be deleted).
         config:
           networkTopologyTypes:
             testtopology:                        # Name of the network topology hierarchy. Up to 20 characters are allowed. Only one topology can be configured.
               - nodeLabel: test-tier-3             # Node label name corresponding to the tier-3 network topology. That means the hypernode tier is 3.
               - nodeLabel: test-tier-2             # Node label name corresponding to the tier-2 network topology. That means the hypernode tier is 2
               - nodeLabel: test-tier-1             # Node label name corresponding to the tier-1 network topolog. That means the hypernode tier is 1
               - nodeLabel: kubernetes.io/hostname # Node label name corresponding to the tier-0 network topology (node level). The label name can only be matched based on the node name and cannot be modified.
   ...

   In addition, add the plugin named network-topology-aware to default_scheduler_conf, as shown below. Configure other parameters based on your requirements.

   ...
   default_scheduler_conf:
     actions: allocate, backfill, preempt
     configurations: ''
     metrics:
       interval: 30s
       type: ''
     tiers:
       - plugins:
           - name: priority
           - enableJobStarving: false
             enablePreemptable: false
             enableReclaimable: false
             name: gang
           - name: conformance
           - enableHierarchy: true
             name: capacity
       - plugins:
           - enablePreemptable: false
             enableReclaimable: false
             name: drf
           - name: predicates
           - name: nodeorder
       - plugins:
           - name: cce-gpu-topology-predicate
           - name: cce-gpu-topology-priority
           - name: xgpu
           - name: network-topology-aware # Add this line to enable network topology-aware scheduling.
             arguments:
               weight: 10                       # The overall policy weight of the plugin. This parameter is optional, and the default value is 1.
               hypernode.binpack.cpu: 5         # CPU bin packing weight at the hypernode level. This parameter is optional, and the default value is 1.
               hypernode.binpack.memory: 5      # Memory bin packing weight at the hypernode level. This parameter is optional, and the default value is 1.
               hypernode.binpack.resources: nvidia.com/gpu, huawei.com/ascend-1980      #: Name of the custom resource to be considered in the bin packing policy. This parameter is optional and not specified by default.
               hypernode.binpack.resources.nvidia.com/gpu: 10                           # Hypernode-level bin packing weight of the custom resource nvidia.com/gpu. This parameter is optional, and the default value is 1.
               hypernode.binpack.resources.huawei.com/ascend-1980: 10                   # Hypernode-level bin packing weight of the custom resource huawei.com/ascend-1980. This parameter is optional, and the default value is 1.
               hypernode.binpack.normal-pod.enable: true      # Whether to enable the multi-tier hypernode bin packing for pods without network topology constraints configured. This parameter is optional, and the default value is true.
               hypernode.binpack.normal-pod.fading: 0.8       # Weight of the tier when the bin packing scores of hypernodes at each tier are calculated for pods without network topology constraints. This parameter is optional, and the default value is 0.8. The weight is calculated using the formula: math.Pow(fading, tier-1).
       - plugins:
           - name: nodelocalvolume
           - name: nodeemptydirvolume
           - name: nodeCSIscheduling
           - name: networkresource
   ...

4. Create eight nodes and add the network topology labels to them. The following figure shows the multi-tier network topology.

   

   Table 1 Mapping between nodes, IP addresses, and hypernodes

   Node Name	IP Address	Labels
   node0	192.168.5.17	test-tier-2=HyperNode2, test-tier-1=HyperNode0
   node1	192.168.5.224	test-tier-2=HyperNode2, test-tier-1=HyperNode0
   node2	192.168.5.235	test-tier-2=HyperNode2, test-tier-1=HyperNode0
   node3	192.168.5.239	test-tier-2=HyperNode2, test-tier-1=HyperNode0
   node4	192.168.5.240	test-tier-2=HyperNode2, test-tier-1=HyperNode1
   node5	192.168.5.251	test-tier-2=HyperNode2, test-tier-1=HyperNode1
   node6	192.168.5.83	test-tier-2=HyperNode2, test-tier-1=HyperNode1
   node7	192.168.5.95	test-tier-2=HyperNode2, test-tier-1=HyperNode1

   This example is for demonstration only. The hypernodes to be created must be based on the actual topology of the cluster. The nodes that are not in a network topology do not need network topology-related node labels. Otherwise, the created hypernodes may not meet the expectation.

   kubectl label node 192.168.5.17  test-tier-2=HyperNode2 test-tier-1=HyperNode0 # node0
   kubectl label node 192.168.5.224 test-tier-2=HyperNode2 test-tier-1=HyperNode0 # node1
   kubectl label node 192.168.5.235 test-tier-2=HyperNode2 test-tier-1=HyperNode0 # node2
   kubectl label node 192.168.5.239 test-tier-2=HyperNode2 test-tier-1=HyperNode0 # node3
   kubectl label node 192.168.5.240 test-tier-2=HyperNode2 test-tier-1=HyperNode1 # node4
   kubectl label node 192.168.5.251 test-tier-2=HyperNode2 test-tier-1=HyperNode1 # node5
   kubectl label node 192.168.5.83  test-tier-2=HyperNode2 test-tier-1=HyperNode1 # node6
   kubectl label node 192.168.5.95  test-tier-2=HyperNode2 test-tier-1=HyperNode1 # node7

5. Query the hypernodes automatically discovered by Volcano.

   kubectl get hypernodes

   Information similar to the following is displayed:

   NAME                                      TIER   TIERNAME      NODECOUNT   AGE
   hypernode-testtopology-tier1-8****   1      test-tier-1   4           6s
   hypernode-testtopology-tier1-l****   1      test-tier-1   4           6s
   hypernode-testtopology-tier2-v****   2      test-tier-2   2           6s

6. Create a Volcano job workload that contains eight pods. Each pod exclusively occupies the NPU resources of a node. At least four pods must be scheduled, and these pods must be on the same tier-1 hypernode. An example is provided as follows:

   apiVersion: batch.volcano.sh/v1alpha1
   kind: Job
   metadata:
     name: network-topology-job
     namespace: default
   spec:
     schedulerName: volcano
     minAvailable: 4         # Gang scheduling condition. At least four pods must be successfully scheduled.
     networkTopology:        # Configure network topology constraints.
       mode: hard
       highestTierAllowed: 1 # Allow placement on a tier=1 hypernode.
     tasks:
       - replicas: 8 # A total of eight pods
         name: t0
         template:
           spec:
             containers:
               - name: container1
                 image: nginx:latest
                 resources:
                   requests:
                     huawei.com/ascend-1980: 16 # Each pod exclusively occupies the NPU resources of a single node.
                   limits:
                     huawei.com/ascend-1980: 16

7. Check the scheduling result. In this example, four pods are scheduled to hypernode1 and are in the Running state. The other four pods are in the Pending state because no node that meets the network topology constraints can be found.

   kubectl get pod -o wide

   Information similar to the following is displayed:

   NAME                        READY   STATUS    RESTARTS   AGE   IP             NODE            NOMINATED NODE   READINESS GATES
   network-topology-job-t0-0   1/1     Running   0          6s    172.19.3.3     192.168.5.239   <none>           <none>
   network-topology-job-t0-1   1/1     Running   0          6s    172.19.2.131   192.168.5.224   <none>           <none>
   network-topology-job-t0-2   1/1     Running   0          6s    172.19.1.142   192.168.5.235   <none>           <none>
   network-topology-job-t0-3   1/1     Running   0          6s    172.19.1.24    192.168.5.17    <none>           <none>
   network-topology-job-t0-4   0/1     Pending   0          6s    <none>         <none>          <none>           <none>
   network-topology-job-t0-5   0/1     Pending   0          6s    <none>         <none>          <none>           <none>
   network-topology-job-t0-6   0/1     Pending   0          6s    <none>         <none>          <none>           <none>
   network-topology-job-t0-7   0/1     Pending   0          6s    <none>         <none>          <none>           <none>