Hypernode Topology-aware Affinity Scheduling

Prerequisites

The Volcano Scheduler add-on must be installed in the cluster, and the add-on version must be 1.21.1 or later.

Constraints

• Network topology-aware scheduling does not support preemption. When resources are insufficient, the scheduler does not proactively preempt the resources that have been allocated to pods.
• If a group affinity policy is configured, network topology constraints in soft mode are not supported.
• Hyperjob workloads do not support network topology-aware scheduling. Even if networkTopology or group affinity policies are configured in the JobTemplate of a hyperjob, these configurations will be ignored and do not take effect.
• The hypernode auto discovery does not support clusters with multiple network topologies (for example, a cluster with both A2 and A3 topologies).
• You are advised not to change the network topology configuration items when the job is running. This is because such changes may cause the hypernode structure in the cluster to be rebuilt, which in turn may cause load fluctuations or scheduling exceptions in the Volcano Scheduler process.
• If the network topology structure or configuration items of a cluster are changed during scheduling, Volcano cannot ensure the correctness of the scheduling result.

Use Case

1. Log in to the CCE console and click the cluster name to access the cluster console.
2. In the navigation pane, choose Settings and click the Scheduling tab. Select Volcano, find the expert mode, and click Try Now.

   

3. On the page for configuring the expert mode, modify default_controller_conf in the YAML file as follows to enable the hypernode auto discovery function: Configure the parameters based on your requirements.

   ...
   default_controller_conf:
     networkTopologyDiscovery:
       - source: label   # Name of the hypernode auto discovery plugin, which cannot be changed
         enabled: true   # Whether to enable the hypernode auto discovery plugin. The value cannot be changed once configured.
         config:
           networkTopologyTypes:
             testtopology:                           # Name of the network topology hierarchy. Up to 20 characters are allowed. Only one topology can be configured. This parameter is configurable.
               - nodeLabel: volcano.sh/hypernode     # Tier-1 network topology, corresponding to the hypernode. This parameter cannot be changed.
               - nodeLabel: kubernetes.io/hostname   # Tier-0 network topology, corresponding to the node. This parameter cannot be changed.
   ...

   After the configuration is updated, run the following command:

   kubectl get hypernodes

   If a hypernode whose TIERNAME is volcano.sh/hypernode is displayed in the command output, the hypernode has been created, and Volcano has detected the hypernode topology information.

   In addition, add the network-topology-aware plugin to the default_scheduler_conf configuration in the YAML file on the CCE expert mode page. The following is an example:

   ...
   default_scheduler_conf:
     actions: allocate, backfill, preempt
     configurations: ''
     metrics:
       interval: 30s
       type: ''
     tiers:
       - plugins:
           - name: priority
           - enableJobStarving: false
             enablePreemptable: false
             enableReclaimable: false
             name: gang
           - name: conformance
           - enableHierarchy: true
             name: capacity
       - plugins:
           - enablePreemptable: false
             enableReclaimable: false
             name: drf
           - name: predicates
           - name: nodeorder
       - plugins:
           - name: cce-gpu-topology-predicate
           - name: cce-gpu-topology-priority
           - name: xgpu
           - name: network-topology-aware # Add this line to enable network topology-aware scheduling.
             arguments:
               weight: 10                       # The overall policy weight of the plugin. This parameter is optional, and the default value is 1.
               hypernode.binpack.cpu: 5         # CPU bin packing weight at the hypernode level. This parameter is optional, and the default value is 1.
               hypernode.binpack.memory: 5      # Memory bin packing weight at the hypernode level. This parameter is optional, and the default value is 1.
               hypernode.binpack.resources: nvidia.com/gpu, huawei.com/ascend-1980      #: Name of the custom resource to be considered in the bin packing policy. This parameter is optional and not specified by default.
               hypernode.binpack.resources.nvidia.com/gpu: 10                           # Hypernode-level bin packing weight of the custom resource nvidia.com/gpu. This parameter is optional, and the default value is 1.
               hypernode.binpack.resources.huawei.com/ascend-1980: 10                   # Hypernode-level bin packing weight of the custom resource huawei.com/ascend-1980. This parameter is optional, and the default value is 1.
               hypernode.binpack.normal-pod.enable: true      # Whether to enable the multi-tier hypernode bin packing for pods without network topology constraints configured. This parameter is optional, and the default value is true.
               hypernode.binpack.normal-pod.fading: 0.8       # Weight of the tier when the bin packing scores of hypernodes at each tier are calculated for pods without network topology constraints. This parameter is optional, and the default value is 0.8. The weight is calculated using the formula: math.Pow(fading, tier-1).
       - plugins:
           - name: nodelocalvolume
           - name: nodeemptydirvolume
           - name: nodeCSIscheduling
           - name: networkresource
   ...

4. Submit the Volcano job defined below to deliver four pods. Every two pods form a group, and each group can only be scheduled to the same tier-1 hypernode.

   apiVersion: batch.volcano.sh/v1alpha1
   kind: Job
   metadata:
     name: vcjob-test
   spec:
     tasks:
       - replicas: 4
         name: "worker"
         partitionPolicy:
           totalPartitions: 2
           partitionSize: 2
           networkTopology:
             mode: hard
             highestTierAllowed: 1
         template:
           spec:
             containers:
               - name: test
                 image: busybox
                 command: ['sh', '-c', 'echo "Hello, Kubernetes!" && sleep 3600']
                 imagePullPolicy: IfNotPresent
                 resources:
                   requests:
                     cpu: 1
                     "huawei.com/ascend-1980": 16
                   limits:
                     cpu: 1
                     "huawei.com/ascend-1980": 16

5. Check the job status.

   kubectl get pod -o wide

   Information similar to the following is displayed:

   NAME                  READY   STATUS    RESTARTS   AGE   IP             NODE            NOMINATED NODE   READINESS GATES
   vcjob-test-worker-0   1/1     Running   0          13s   172.19.3.181   192.168.5.95    <none>           <none>
   vcjob-test-worker-1   1/1     Running   0          13s   172.19.0.3     192.168.5.25    <none>           <none>
   vcjob-test-worker-2   1/1     Running   0          13s   172.19.1.3     192.168.5.231   <none>           <none>
   vcjob-test-worker-3   1/1     Running   0          13s   172.19.3.42    192.168.5.239   <none>           <none>

6. Check the hypernode of the nodes where the four pods are running.

   kubectl describe node 192.168.5.95  | grep volcano.sh/hypernode
   kubectl describe node 192.168.5.25  | grep volcano.sh/hypernode
   kubectl describe node 192.168.5.231 | grep volcano.sh/hypernode
   kubectl describe node 192.168.5.239 | grep volcano.sh/hypernode

   Information similar to the following is displayed:

   volcano.sh/hypernode=hypernode-2
   volcano.sh/hypernode=hypernode-2
   volcano.sh/hypernode=hypernode-1
   volcano.sh/hypernode=hypernode-1

   The first group of pods (worker-0 and worker-1) are scheduled to hypernode-2 (volcano.sh/hypernode = hypernode-2), and the second group of pods (worker-2 and worker-3) are scheduled to hypernode-1 (volcano.sh/hypernode = hypernode-1), which meets the scheduling requirements.