Multi-tier Network Topology-aware Bin Packing

Function Introduction

Through multi-tier network topology-aware scheduling and group affinity scheduling, Volcano has effectively resolved network communication performance issues of distributed AI training/inference jobs in data centers. However, as new workloads are created and old workloads are deleted, resource fragmentation on each hypernode becomes increasingly severe. Although the cluster has sufficient available resources, they are scattered across hypernodes. As a result, new jobs that require high communication performance cannot be scheduled or can only be scheduled in a scattered manner, reducing the cluster resource utilization. Conventional bin packing scheduling algorithms mainly consider node-level resource usage. By assigning tasks to as few nodes as possible, these algorithms effectively mitigate resource fragmentation on nodes but cannot address resource fragmentation on hypernodes.

For this reason, Volcano introduces the multi-tier network topology-aware bin packing capability. During scheduling, Volcano considers the resource usage of hypernodes at each tier and schedules workloads to hypernodes with higher resource usage. This helps alleviate hypernode resource fragmentation and improve overall resource utilization.

When scheduling workloads with network topology constraints, the scheduler preferentially places them on the current hypernode with higher resource usage to enhance the resource utilization of the network performance domain. In the figure below, there are two tier-1 hypernodes in the cluster, with hypernode1 already partially occupied by other tasks. A Volcano job is delivered, and the pods are evenly split into two partitions, with each partition configured with a network topology constraint that restricts deployment to hypernodes at tier 1.

• Without bin packing, pods in partition1 are scheduled to hypernode1, and those in partition2 is scheduled to hypernode2.
• With bin packing, pods in both partition1 and partition2 will be preferentially scheduled to hypernode1, so that hypernode2 can serve as a fully idle hypernode for other tasks.

In addition, for workloads without network topology constraints, the scheduler provides multi-tier hypernode bin packing. This means these workloads are preferentially scheduled to the nodes with high resource usages on the hypernodes at all tiers. As shown in the figure below, assume that there are eight nodes and seven hypernodes in the cluster, with four hypernodes at tier 1, two hypernodes at tier 2, and one hypernode at tier 3. The resources on node0, node2, and node4 are occupied by other tasks. A Volcano job is delivered, and this job requires two pods and has no network topology constraints.

• If multi-tier hypernode bin packing is not enabled, the two pods may be scheduled to any two nodes of node1, node3, node5, node6, and node7. As a result, other larger jobs with network topology constraints may fail to be scheduled.
• If this capability is enabled, the two pods are preferentially scheduled to node1 and node3. In this case, hypernode3 and hypernode5 have more resources for other larger jobs with network topology constraints.

To use this capability, you need to enable the network-topology-aware plugin for Volcano and configure the parameters below. The value of the fading parameter must be greater than or equal to 0. Otherwise, the default value 0.8 is used.

• When the value of fading is 0, the bin packing weight of hypernodes at each tier is 0.
• When the value of fading is greater than 0 and less than 1, the bin packing weight of lower-tier hypernodes is greater than that of higher-tier hypernodes.
• When the value of fading is 1, the bin packing weight of hypernodes at each tier is 1.
• When the value of fading is greater than 1, the bin packing weight of higher-tier hypernodes is greater than that of lower-tier hypernodes.

...
default_scheduler_conf:
  tiers:
    - plugins:
        - name: network-topology-aware # Add this line to enable network topology-aware scheduling.
          arguments:
            weight: 10                       # The overall policy weight of the plugin. This parameter is optional, and the default value is 1.
            hypernode.binpack.cpu: 5         # CPU bin packing weight at the hypernode level. This parameter is optional, and the default value is 1.
            hypernode.binpack.memory: 5      # Memory bin packing weight at the hypernode level. This parameter is optional, and the default value is 1.
            hypernode.binpack.resources: nvidia.com/gpu, huawei.com/ascend-1980      #: Name of the custom resource to be considered in the bin packing policy. This parameter is optional and not specified by default.
            hypernode.binpack.resources.nvidia.com/gpu: 10                  # Hypernode-level bin packing weight of the custom resource nvidia.com/gpu. This parameter is optional, and the default value is 1.
            hypernode.binpack.resources.huawei.com/ascend-1980: 10         # Hypernode-level bin packing weight of the custom resource huawei.com/ascend-1980. This parameter is optional, and the default value is 1.
            hypernode.binpack.normal-pod.enable: true      # Whether to enable the multi-tier hypernode bin packing for pods without network topology constraints configured. This parameter is optional, and the default value is true.
            hypernode.binpack.normal-pod.fading: 0.8       # Weight of the tier when the bin packing scores of hypernodes at each tier are calculated for pods without network topology constraints. This parameter is optional, and the default value is 0.8. The weight is calculated using the formula: math.Pow(fading, tier-1).
...

Prerequisites

The Volcano Scheduler add-on must be installed in the cluster, and the add-on version must be 1.21.1 or later.

Constraints

• Network topology-aware scheduling does not support preemption. When resources are insufficient, the scheduler does not proactively preempt the resources that have been allocated to pods.
• If a group affinity policy is configured, network topology constraints in soft mode are not supported.
• Hyperjob workloads do not support network topology-aware scheduling. Even if networkTopology or group affinity policies are configured in the JobTemplate of a hyperjob, these configurations will be ignored and do not take effect.
• The hypernode auto discovery does not support clusters with multiple network topologies (for example, a cluster with both A2 and A3 topologies).
• You are advised not to change the network topology configuration items when the job is running. This is because such changes may cause the hypernode structure in the cluster to be rebuilt, which in turn may cause load fluctuations or scheduling exceptions in the Volcano Scheduler process.
• If the network topology structure or configuration items of a cluster are changed during scheduling, Volcano cannot ensure the correctness of the scheduling result.

Use Case

1. Log in to the CCE console and click the cluster name to access the cluster console.
2. In the navigation pane, choose Settings and click the Scheduling tab. Select Volcano, find the expert mode, and click Try Now.

   

3. On the page for configuring the expert mode, modify default_controller_conf in the YAML file as follows to enable the hypernode auto discovery function: Configure the parameters based on your requirements.

   ...
   default_controller_conf:
     networkTopologyDiscovery:
       - source: label # Name of the hypernode auto discovery plugin, which cannot be changed
         enabled: true # Whether to enable the hypernode auto discovery plugin. true indicates that the plugin is enabled, and false indicates that the plugin is disabled (existing hypernode instances will not be deleted).
         config:
           networkTopologyTypes:
             testtopology:                        # Name of the network topology hierarchy. Up to 20 characters are allowed. Only one topology can be configured.
               - nodeLabel: test-tier-3             # Node label name corresponding to the tier-3 network topology. That means the hypernode tier is 3.
               - nodeLabel: test-tier-2             # Node label name corresponding to the tier-2 network topology. That means the hypernode tier is 2.
               - nodeLabel: test-tier-1             # Node label name corresponding to the tier-1 network topolog. That means the hypernode tier is 1.
               - nodeLabel: kubernetes.io/hostname # Node label name corresponding to the tier-0 network topology (node level). The label name can only be matched based on the node name and cannot be modified.
   ...

   In addition, add the plugin named network-topology-aware to default_scheduler_conf, as shown below. Configure other parameters based on your requirements.

   ...
   default_scheduler_conf:
     actions: allocate, backfill, preempt
     configurations: ''
     metrics:
       interval: 30s
       type: ''
     tiers:
       - plugins:
           - name: priority
           - enableJobStarving: false
             enablePreemptable: false
             enableReclaimable: false
             name: gang
           - name: conformance
           - enableHierarchy: true
             name: capacity
       - plugins:
           - enablePreemptable: false
             enableReclaimable: false
             name: drf
           - name: predicates
           - name: nodeorder
       - plugins:
           - name: cce-gpu-topology-predicate
           - name: cce-gpu-topology-priority
           - name: xgpu
           - name: network-topology-aware # Add this line to enable network topology-aware scheduling.
             arguments:
               weight: 10                  # The overall policy weight of the plugin. The default value is 1.
               hypernode.binpack.cpu: 5         # CPU bin packing weight at the hypernode level. The default value is 1.
               hypernode.binpack.memory: 5 # Memory bin packing weight at the hypernode level. The default value is 1.
               hypernode.binpack.resources: nvidia.com/gpu, huawei.com/ascend-1980 # Name of the custom resource to be considered in the bin packing policy. By default, this parameter is not specified.
               hypernode.binpack.resources.nvidia.com/gpu: 10                      # Hypernode-level bin packing weight of nvidia.com/gpu. By default, this parameter is not specified.
   hypernode.binpack.resources.huawei.com/ascend-1980: 10 # Hypernode-level bin packing weight of huawei.com/ascend-1980. By default, this parameter is not specified.
               hypernode.binpack.normal-pod.enable: true     # Whether to enable the multi-tier hypernode bin packing for pods without network topology constraints configured. The default value is true.
               hypernode.binpack.normal-pod.fading: 0.8       # Weight of the tier when the bin packing scores of hypernodes at each tier are calculated for pods without network topology constraints. The default value is 0.8. The weight is calculated using the formula: math.Pow(fading, tier-1).
       - plugins:
           - name: nodelocalvolume
           - name: nodeemptydirvolume
           - name: nodeCSIscheduling
           - name: networkresource
   ...

4. Create eight nodes and add the network topology labels to them. The following figure shows the multi-tier network topology.

   

   This example is for demonstration only. The hypernodes to be created must be based on the actual topology of the cluster. The nodes that are not in a network topology do not need network topology-related node labels. Otherwise, the created hypernodes may not meet the expectation.

   kubectl label node 192.168.5.17  test-tier-3=HyperNode6 test-tier-2=HyperNode4 test-tier-1=HyperNode0 # node0
   kubectl label node 192.168.5.224 test-tier-3=HyperNode6 test-tier-2=HyperNode4 test-tier-1=HyperNode0 # node1
   kubectl label node 192.168.5.235 test-tier-3=HyperNode6 test-tier-2=HyperNode4 test-tier-1=HyperNode1 # node2
   kubectl label node 192.168.5.239 test-tier-3=HyperNode6 test-tier-2=HyperNode4 test-tier-1=HyperNode1 # node3
   kubectl label node 192.168.5.240 test-tier-3=HyperNode6 test-tier-2=HyperNode5 test-tier-1=HyperNode2 # node4
   kubectl label node 192.168.5.251 test-tier-3=HyperNode6 test-tier-2=HyperNode5 test-tier-1=HyperNode2 # node5
   kubectl label node 192.168.5.83  test-tier-3=HyperNode6 test-tier-2=HyperNode5 test-tier-1=HyperNode3 # node6
   kubectl label node 192.168.5.95  test-tier-3=HyperNode6 test-tier-2=HyperNode5 test-tier-1=HyperNode3 # node7

   Table 1 Mapping between nodes, IP addresses, and hypernodes

   Node Name	IP Address	Labels
   node0	192.168.5.17	test-tier-3=HyperNode6 test-tier-2=HyperNode4 test-tier-1=HyperNode0
   node1	192.168.5.224	test-tier-3=HyperNode6 test-tier-2=HyperNode4 test-tier-1=HyperNode0
   node2	192.168.5.235	test-tier-3=HyperNode6 test-tier-2=HyperNode4 test-tier-1=HyperNode1
   node3	192.168.5.239	test-tier-3=HyperNode6 test-tier-2=HyperNode4 test-tier-1=HyperNode1
   node4	192.168.5.240	test-tier-3=HyperNode6 test-tier-2=HyperNode5 test-tier-1=HyperNode2
   node5	192.168.5.251	test-tier-3=HyperNode6 test-tier-2=HyperNode5 test-tier-1=HyperNode2
   node6	192.168.5.83	test-tier-3=HyperNode6 test-tier-2=HyperNode5 test-tier-1=HyperNode3
   node7	192.168.5.95	test-tier-3=HyperNode6 test-tier-2=HyperNode5 test-tier-1=HyperNode3

5. Query the hypernodes automatically discovered by Volcano.

   kubectl get hypernodes

   Information similar to the following is displayed:

   NAME                                      TIER   TIERNAME      NODECOUNT   AGE
   hypernode-testtopology-tier1-5****   1      test-tier-1   2           9s
   hypernode-testtopology-tier1-7****   1      test-tier-1   2           10s
   hypernode-testtopology-tier1-g****   1      test-tier-1   2           10s
   hypernode-testtopology-tier1-l****   1      test-tier-1   2           9s
   hypernode-testtopology-tier2-5****   2      test-tier-2   2           9s
   hypernode-testtopology-tier2-t****   2      test-tier-2   2           10s
   hypernode-testtopology-tier3-c****   3      test-tier-3   2           10s

6. Create three pods, schedule them to node0, node2, and node4, and occupy all NPU resources of these nodes. The following is an example:

   apiVersion: v1
   kind: Pod
   metadata:
     name: node-pod-1
   spec:
     containers:
     - name: nginx
       image: nginx:latest
       resources:
         requests:
           huawei.com/ascend-1980: 16
         limits:
           huawei.com/ascend-1980: 16
     nodeName: 192.168.5.17 # node0
   ---
   apiVersion: v1
   kind: Pod
   metadata:
     name: node-pod-2
   spec:
     containers:
     - name: nginx
       image: nginx:latest
       resources:
         requests:
           huawei.com/ascend-1980: 16
         limits:
           huawei.com/ascend-1980: 16
     nodeName: 192.168.5.235 # node2
   ---
   apiVersion: v1
   kind: Pod
   metadata:
     name: node-pod-3
   spec:
     containers:
     - name: nginx
       image: nginx:latest
       resources:
         requests:
           huawei.com/ascend-1980: 16
         limits:
           huawei.com/ascend-1980: 16
     nodeName: 192.168.5.240 # node4

7. Create a Volcano job that requires two pods. Each pod exclusively occupies the NPU resources of a node, and there is no network topology constraint. An example is provided as follows:

   apiVersion: batch.volcano.sh/v1alpha1
   kind: Job
   metadata:
     name: network-topology-job
     namespace: default
   spec:
     schedulerName: volcano
     tasks:
       - replicas: 2
         name: t0
         template:
           spec:
             containers:
               - name: container1
                 image: nginx:latest
                 resources:
                   requests:
                     huawei.com/ascend-1980: 16
                   limits:
                     huawei.com/ascend-1980: 16

8. Check the scheduling result. In this example, the resources of node0, node2, and node4 are occupied. Therefore, the two pods are scheduled to node1 and node3, respectively.

   kubectl get pod -o wide

   Information similar to the following is displayed:

   NAME                        READY   STATUS    RESTARTS   AGE   IP             NODE            NOMINATED NODE   READINESS GATES
   network-topology-job-t0-0   1/1     Running   0          5s    172.19.2.132   192.168.5.224   <none>           <none>
   network-topology-job-t0-1   1/1     Running   0          5s    172.19.3.4     192.168.5.239   <none>           <none>
   node-pod-1                  1/1     Running   0          16s   172.19.1.25    192.168.5.17    <none>           <none>
   node-pod-2                  1/1     Running   0          16s   172.19.1.143   192.168.5.235   <none>           <none>
   node-pod-3                  1/1     Running   0          16s   172.19.2.4     192.168.5.240   <none>           <none>