Pod Rescheduling by Partition

Function Introduction

During distributed AI training/inference in data centers, job instances have features like strong collaborative dependency, long-term running, and exclusive resource usage (such as NPUs and GPUs). However, data centers are prone to faults such as a node breakdown or network partition. If a cluster is faulty and affects the running of some pods, restarting the entire job is costly. As a result, the entire job may be stuck in a cycle of running, failure, and restart, and cannot be completed successfully. Migrating only the affected pods is less costly, but these pods may be isolated from other pods in the same group. This can cause outdated running information and invalid topology constraints. As a result, the entire group is in the state of occupying resources but not running properly, and cluster resources are wasted.

To address this issue, Volcano provides the capability of rescheduling pods by partition. When some pods are evicted or become inactive due to a cluster fault, the pods in the partition are proactively rescheduled to other suitable nodes to accelerate job recovery and avoid wasting cluster resources.

The following figure shows an example. The cluster consists of eight idle nodes, four tier-1 hypernodes, two tier-2 hypernodes, and one tier-3 hypernode. A Volcano job is running in the cluster. The job has two partitions. Each partition is scheduled to the same tier-1 hypernode. pod0 and pod1 are in partition1 and run on node0 and node1, respectively. pod2 and pod3 are in partition2 and run on node2 and node3, respectively. If node3 fails, pod3 on it becomes inactive.

Figure 1 Pod rescheduling not supported

• If pod rescheduling is not supported, the newly created pod3 will remain in the Pending state because no node meets the network topology constraints. As a result, pod2 occupies the resources of node2 but cannot run properly, wasting cluster resources. You can restart the entire job, but pod0 and pod1 will need to tbe restarted as well. This results in high job recovery costs.
• If pod rescheduling is supported, pod2 in the same partition can be proactively terminated, and the newly created pod2 and pod3 can be scheduled to node6 and node7 on hypernode3. In this way, the entire job can be efficiently restored.

Figure 2 Pod rescheduling supported

Volcano provides the RestartPartition action in job lifecycle management policies, and pods in the entire partition can be restarted or rescheduled in case of a specific event. For example, add the following configuration to a job to restart and reschedule the entire partition when a pod in this partition is evicted or fails:

spec:
  policies:
  - action: RestartPartition
    event: PodEvicted
  - action: RestartPartition
    event: PodFailed

Prerequisites

The Volcano Scheduler add-on must be installed in the cluster, and the add-on version must be 1.21.1 or later.

Constraints

• Network topology-aware scheduling does not support preemption. When resources are insufficient, the scheduler does not proactively preempt the resources that have been allocated to pods.
• If a group affinity policy is configured, network topology constraints in soft mode are not supported.
• Hyperjob workloads do not support network topology-aware scheduling. Even if networkTopology or group affinity policies are configured in the JobTemplate of a hyperjob, these configurations will be ignored and do not take effect.
• The hypernode auto discovery does not support clusters with multiple network topologies (for example, a cluster with both A2 and A3 topologies).
• You are advised not to change the network topology configuration items when the job is running. This is because such changes may cause the hypernode structure in the cluster to be rebuilt, which in turn may cause load fluctuations or scheduling exceptions in the Volcano Scheduler process.
• If the network topology structure or configuration items of a cluster are changed during scheduling, Volcano cannot ensure the correctness of the scheduling result.

Use Case

1. Log in to the CCE console and click the cluster name to access the cluster console.
2. In the navigation pane, choose Settings and click the Scheduling tab. Select Volcano, find the expert mode, and click Try Now.

   

3. On the page for configuring the expert mode, modify default_controller_conf in the YAML file as follows to enable the hypernode auto discovery function: Configure the parameters based on your requirements.

   ...
   default_controller_conf:
     networkTopologyDiscovery:
       - source: label # Name of the hypernode auto discovery plugin, which cannot be changed
         enabled: true # Whether to enable the hypernode auto discovery plugin. true indicates that the plugin is enabled, and false indicates that the plugin is disabled (existing hypernode instances will not be deleted).
         config:
           networkTopologyTypes:
             testtopology:                        # Name of the network topology hierarchy. Up to 20 characters are allowed. Only one topology can be configured.
               - nodeLabel: test-tier-3             # Node label name corresponding to the tier-3 network topology. That means the hypernode tier is 3.
               - nodeLabel: test-tier-2             # Node label name corresponding to the tier-2 network topology. That means the hypernode tier is 2.
               - nodeLabel: test-tier-1             # Node label name corresponding to the tier-1 network topology. That means the hypernode tier is 1.
               - nodeLabel: kubernetes.io/hostname # Node label name corresponding to the tier-0 network topology (node level). The label name can only be matched based on the node name and cannot be modified.
   ...

4. Create eight nodes and add the network topology labels to them. The following figure shows the multi-tier network topology.

   This example is for demonstration only. The hypernodes to be created must be based on the actual topology of the cluster. The nodes that are not in a network topology do not need network topology-related node labels. Otherwise, the created hypernodes may not meet the expectation.

   kubectl label node 192.168.5.17  test-tier-3=HyperNode6 test-tier-2=HyperNode4 test-tier-1=HyperNode0 # node0
   kubectl label node 192.168.5.224 test-tier-3=HyperNode6 test-tier-2=HyperNode4 test-tier-1=HyperNode0 # node1
   kubectl label node 192.168.5.235 test-tier-3=HyperNode6 test-tier-2=HyperNode4 test-tier-1=HyperNode1 # node2
   kubectl label node 192.168.5.239 test-tier-3=HyperNode6 test-tier-2=HyperNode4 test-tier-1=HyperNode1 # node3
   kubectl label node 192.168.5.240 test-tier-3=HyperNode6 test-tier-2=HyperNode5 test-tier-1=HyperNode2 # node4
   kubectl label node 192.168.5.251 test-tier-3=HyperNode6 test-tier-2=HyperNode5 test-tier-1=HyperNode2 # node5
   kubectl label node 192.168.5.83  test-tier-3=HyperNode6 test-tier-2=HyperNode5 test-tier-1=HyperNode3 # node6
   kubectl label node 192.168.5.95  test-tier-3=HyperNode6 test-tier-2=HyperNode5 test-tier-1=HyperNode3 # node7

   Table 1 Mapping between nodes, IP addresses, and hypernodes

   Node Name	IP Address	Labels
   node0	192.168.5.17	test-tier-3=HyperNode6 test-tier-2=HyperNode4 test-tier-1=HyperNode0
   node1	192.168.5.224	test-tier-3=HyperNode6 test-tier-2=HyperNode4 test-tier-1=HyperNode0
   node2	192.168.5.235	test-tier-3=HyperNode6 test-tier-2=HyperNode4 test-tier-1=HyperNode1
   node3	192.168.5.239	test-tier-3=HyperNode6 test-tier-2=HyperNode4 test-tier-1=HyperNode1
   node4	192.168.5.240	test-tier-3=HyperNode6 test-tier-2=HyperNode5 test-tier-1=HyperNode2
   node5	192.168.5.251	test-tier-3=HyperNode6 test-tier-2=HyperNode5 test-tier-1=HyperNode2
   node6	192.168.5.83	test-tier-3=HyperNode6 test-tier-2=HyperNode5 test-tier-1=HyperNode3
   node7	192.168.5.95	test-tier-3=HyperNode6 test-tier-2=HyperNode5 test-tier-1=HyperNode3

5. Query the hypernodes automatically discovered by Volcano.

   kubectl get hypernodes

   Information similar to the following is displayed:

   NAME                                      TIER   TIERNAME      NODECOUNT   AGE
   hypernode-testtopology-tier1-2****   1      test-tier-1   2           8s
   hypernode-testtopology-tier1-4****   1      test-tier-1   2           8s
   hypernode-testtopology-tier1-j****   1      test-tier-1   2           8s
   hypernode-testtopology-tier1-l****   1      test-tier-1   2           8s
   hypernode-testtopology-tier2-m****   2      test-tier-2   2           8s
   hypernode-testtopology-tier2-n****   2      test-tier-2   2           8s
   hypernode-testtopology-tier3-c****   3      test-tier-3   2           8s

6. Create a Volcano job that requires four pods, with each pod exclusively occupying all NPU resources of a node. Configure a grouping policy to evenly divide the four pods into two partitions and schedule the pods in each partition to a hypernode whose tier is 1 at most. In addition, configure a lifecycle management policy with event set to PodEvicted and action set to RestartPartition, and enable the capability of rescheduling the pods in case of a fault. An example is provided as follows:

   apiVersion: batch.volcano.sh/v1alpha1
   kind: Job
   metadata:
     name: network-topology-job
     namespace: default
   spec:
     schedulerName: volcano
     policies:
     - action: RestartPartition   # When a PodEvicted event occurs, all pods in the partition that the pods belong to are restarted.
       event: PodEvicted
     minAvailable: 4         # Gang scheduling condition. At least four pods must be successfully scheduled.
     tasks:
       - replicas: 4 # A total of four pods
         name: t0
         partitionPolicy:          # Configure a group affinity scheduling policy.
           totalPartitions: 2      # Split the task's pods into two partitions.
           partitionSize: 2        # Each partition contains two pods.
           networkTopology:        # Configure network topology constraints at the partition level.
             mode: hard            # The hard mode is used and must be met.
             highestTierAllowed: 1 # Allow placement on a tier=1 hypernode.
         template:
           spec:
             containers:
               - name: container1
                 image: nginx:latest
                 resources:
                   requests:
                     huawei.com/ascend-1980: 16 # Each pod exclusively occupies the NPU resources of a single node.
                   limits:
                     huawei.com/ascend-1980: 16

7. Check the scheduling result. Now, the first partition of pods has been scheduled to node0 and node1 of hypernode0, and the other group of pods has been scheduled to node2 and node3 of hypernode1.

   kubectl get pod -o wide

   Information similar to the following is displayed:

   NAME                        READY   STATUS    RESTARTS   AGE   IP             NODE            NOMINATED NODE   READINESS GATES
   network-topology-job-t0-0   1/1     Running   0          6s    172.19.1.36    192.168.5.17    <none>           <none>
   network-topology-job-t0-1   1/1     Running   0          6s    172.19.2.143   192.168.5.224   <none>           <none>
   network-topology-job-t0-2   1/1     Running   0          6s    172.19.3.19    192.168.5.239   <none>           <none>
   network-topology-job-t0-3   1/1     Running   0          6s    172.19.1.162   192.168.5.235   <none>           <none>

8. Evict the pod on node3 to simulate a node fault.

   kubectl drain 192.168.5.239 --ignore-daemonsets

9. Check the scheduling result again. The two pods in the second partition have been scheduled to node6 and node7 of hypernode3, maintaining the network topology consistency with other pods in the partitions.

   kubectl get pod -o wide

   Information similar to the following is displayed:

   NAME                        READY   STATUS    RESTARTS   AGE     IP             NODE            NOMINATED NODE   READINESS GATES
   network-topology-job-t0-0   1/1     Running   0          2m51s   172.19.1.29    192.168.5.17    <none>           <none>
   network-topology-job-t0-1   1/1     Running   0          2m51s   172.19.2.136   192.168.5.224   <none>           <none>
   network-topology-job-t0-2   1/1     Running   0          80s     172.19.3.143   192.168.5.95    <none>           <none>
   network-topology-job-t0-3   1/1     Running   0          82s     172.19.4.12    192.168.5.83    <none>           <none>

10. Restore node3.

    kubectl uncordon 192.168.5.239