Help Center/ Cloud Container Engine/ User Guide/ Scheduling/ Volcano Scheduling/ Heterogeneous Resource Network Topology-aware Scheduling/ Network Topology-aware Scheduling
Updated on 2026-06-16 GMT+08:00
View PDF
Share

Network Topology-aware Scheduling

Overview

With the rapid emergence and widespread application of various AI foundation models, distributed AI training/inference jobs (including data parallelism, tensor parallelism, pipeline parallelism, expert parallelism, and PD disaggregation) have gradually become the core workloads of the computing infrastructure. Typically, the subtasks of these jobs run on multiple compute nodes, and the entire computing process heavily relies on data exchange between subtasks, such as gradient aggregation and model distribution. In this case, the network transmission performance between nodes often becomes a bottleneck, significantly affecting the training efficiency. The core cause of this issue is the complex network topology (typically involving multiple layers of switches) and diverse network types (such as IB, RoCE, and NVSwitch) in data centers. A longer communication path across multi-layer switches increases the latency and lowers the throughput. Conversely, a shorter communication path improves communication performance. Therefore, scheduling compute tasks with strong communication dependencies (such as shards of model parallelism and replica groups of data parallelism) to the same high-performance domain (a network area with the shortest communication path and optimal bandwidth) can effectively reduce communication costs, accelerate data exchange, and significantly improve the overall computing efficiency of distributed AI training/inference jobs.

To handle this, Volcano introduced network topology-aware scheduling, which takes the cluster's network topology into account during scheduling and places a group of pods with strong communication dependencies into the same network-performance domain. This effectively resolves network communication performance issues for large-scale data center AI training jobs.

Multi-tier Network Topology

To uniformly describe the multi-tier network topology of a cluster, mask network type differences, and provide standardized APIs for upper-layer components such as the scheduler, Volcano introduces a new Custom Resource Definition (CRD) called hypernode to represent the network topology. As shown in the figure, each hypernode represents a network topology performance domain, which can be mapped to a switch or Top-of-Rack (ToR). Multiple hypernodes can be connected hierarchically to form a tree structure, accurately representing the complex network topology in a data center.

In addition, Volcano provides a hypernode auto discovery mechanism. With this mechanism, Volcano can automatically discover network topology structures within clusters and creates, updates, or deletes hypernodes based on the discovery results. This ensures the consistency between hypernodes and the actual network status and also reduces the management burden of network topology information.

Multi-dimensional Job Grouping

In large-scale AI training/inference scenarios, a job may contain multiple shards, and each shard contains multiple compute tasks. Compared with communication between shards, each shard has higher requirements on the internal network communication performance. Take the prefill role and decode role as an example. The task communication within a role is more frequent than that between roles. Therefore, only network topology-aware affinity scheduling at the job level is not enough because it cannot ensure that tasks of a shard are deployed in the same network performance domain. Shards may be deployed in different network performance domains, affecting communication efficiency.

To address this issue, Volcano introduces the partitionPolicy field in Volcano jobs, including configuration items such as totalPartitions and partitionSize, to describe and detect the multi-dimensional grouping of jobs. The Volcano Scheduler performs job-level and group-level affinity scheduling based on these configurations to schedule tasks with tight communication requirements to the same high-performance network domain, thereby improving communication efficiency.

Multi-tier Network Topology-aware Scheduling

In multi-tier network topology-aware scheduling, workloads with high communication performance requirements are scheduled to the same hypernode to improve the network communication performance. For details, see Multi-tier Network Topology-aware Scheduling.

Multi-dimensional and Multi-tier Network Topology-aware Scheduling

Based on the multi-tier network topology-aware scheduling, the multi-dimensional grouping is introduced to further divide workloads into multiple groups. Each group can be independently scheduled to the same hypernode, implementing fine-grained topology affinity scheduling. This capability significantly improves scheduling flexibility. For details, see Multi-dimensional and Multi-tier Network Topology-aware Scheduling.

Multi-tier Network Topology-aware Bin Packing

Based on the awareness of the network topology performance domain of each hypernode, workloads are preferentially scheduled to hypernodes with a high resource allocation rate to address resource fragmentation. For details, see Multi-tier Network Topology-aware Bin Packing.

Multi-dimensional Gang Scheduling

Gang scheduling is performed from three dimensions: pod, group (PodGroup), and partition (subgroup). This type of scheduling avoids deadlocks and resource waste caused by resource contention. For details, see Multi-dimensional Gang Scheduling.

Fault-triggered Pod Rescheduling

When a node breaks down or there is a network partition, the system can reschedule the entire partition of pods to continuously meet the network topology affinity requirements of the partition and quickly restore the jobs. For details, see Pod Rescheduling by Partition.

Hypernode Topology-aware Scheduling

In large-scale AI training and inference scenarios, CCE allows you to connect hypernodes as container nodes to accelerate AI computing tasks. You can use Volcano jobs to schedule and manage related tasks in a unified manner. For details, see Hypernode Topology-aware Affinity Scheduling.

Hyperjobs will be taken offline soon. You are advised to use Volcano jobs.

Parent Topic: Heterogeneous Resource Network Topology-aware Scheduling