Configuring Supernode Affinity Group Instances
Description
During foundation model training, various parallelism strategies (such as model parallelism and MoE expert parallelism) are typically adopted to improve training efficiency and scalability through distributed computing. Among them, the all-reduce communication in model parallelism and the all-to-all communication in MoE expert parallelism have extremely high requirements for inter-card interconnect bandwidth. Constrained by hardware networking limits, these communication-intensive phases often become bottlenecks in training performance.
Huawei Ascend's next-generation hardware (such as the Snt9b23 and Snt9e) interconnects NPUs across multiple compute nodes with high bandwidth to form a supernode. This inter-node networking topology within a supernode is referred to as a hyperplane, which can significantly improve the communication efficiency of AI tasks. Based on supernode hardware, models can adjust the parameters of parallelism strategies (such as model parallelism or MoE expert parallelism) on a larger scale, thereby leveraging higher bandwidth to accelerate training.
ModelArts provides a supernode affinity feature that acts as a scheduling policy. This feature organizes AI training jobs to align with the hardware network of compute resources, maximizing the high bandwidth and low latency of supernodes for better training efficiency. Algorithm engineers can effortlessly use supernode hardware through simple setups.
Principles
The figure shows that each task in a training job is referred to as an instance. ModelArts uses affinity groups to schedule training jobs on supernodes and maximize their ultra-high bandwidth. An affinity group is a group of instances that you want to connect through a hyperplane in a training job. Instances in the same affinity group run on the same hyperplane.
To boost training efficiency, place PUs that need high bandwidth in the same affinity group.
For Tensor Parallelism (TP), in Snt9b hardware, TP is often set to 8, linking eight PUs per node with HCCS. In supernodes, the number of PUs with high-speed interconnection increases, allowing for larger TP domains.
- Prerequisite: In distributed training, each instance must use all PUs on the node.
If TP is 32, then 32 PUs form one TP domain, and each node has 16 PUs. Here, two instances must share the same hyperplane, forming an affinity group.
- Configuration: If TP is 32 (where every 32 PUs form a TP domain), 2 instances (16 Pus per instance) must be scheduled to the same hyperplane; therefore, you need to set Supernode Affinity Group Instances to 2.
- Resource scheduling condition: The training job runs only if the resource pool's available supernodes meet these conditions:
- Hardware differences and scheduling strategies.
The following introduces the corresponding scheduling strategies for different chip types:
- Snt9b23 affinity scheduling
In Snt9b23 hardware resource pools, each supernode features an independent hyperplane, and hyperplanes across different supernodes are always distinct. Consequently, the boundary of a supernode is equivalent to the boundary of a hyperplane. All instances within the same affinity group are guaranteed to be scheduled to the same hyperplane, which is equivalent to ensuring they are all scheduled to the same supernode. By default, communication between affinity groups on Snt9b23 occurs via the parameter plane.
Figure 2 Snt9b23 supernodes and hyperplanes
Figure 3 Supernode affinity scheduling for Snt9b23 training jobs
If the training job needs eight instances, with Supernode Affinity Group Instances set to 2. Here are the available resources:
- Resource pool A
Supernode 1: maximum nodes: 8; available nodes: 6
Supernode 2: maximum nodes: 8; available nodes: 2
- Resource pool B
Supernode 3: maximum nodes: 8; available nodes: 7
Supernode 4: maximum nodes: 8; available nodes: 1
Both pools have eight available nodes, meeting the condition 3.a. However, supernode 3 in pool B can allocate three affinity groups, while supernode 4 can allocate none. The condition 3.b is not met. Thus, only pool A can run the training job. If the job goes to pool B, it will wait in the queue due to insufficient resources.
- Resource pool A
- Snt9e affinity scheduling
In the Snt9e hardware resource pool, supernodes can share the same hyperplane when created. Therefore, multiple supernodes may exist under the same hyperplane, and the NPUs across these supernodes are interconnected with high speed. All instances within the same affinity group are guaranteed to be scheduled to the same hyperplane, though this may span across different supernodes in this scenario.
Figure 4 Specifying a hyperplane network
Figure 5 Snt9e supernodes and hyperplanes
Figure 6 Supernode affinity scheduling for Snt9e training jobs
- Snt9b23 affinity scheduling
Constraints
- The affinity group feature only works with dedicated resource pools that have Ascend Snt9b23 and Snt9e supernodes.
- In distributed training, each instance must use all PUs on the node.
Configuration Method
When you create a training job on the ModelArts console, selecting a dedicated resource pool and an Snt9b23 or Snt9e flavor allows you to set Supernode Affinity Group Instances during Step 5: Configuring Resources.
Notes for setting Supernode Affinity Group Instances:
- The default value is 1, and the maximum value cannot exceed the maximum number of nodes on a single supernode.
- You must set the number of instances as an integral multiple of Supernode Affinity Group Instances. Otherwise, the training job cannot be created.
- After choosing a dedicated resource pool, you can check its supernode specifications, like total nodes, total PUs, available nodes, and available PUs. This helps you decide on Supernode Affinity Group Instances.
- When selecting a resource queue, ensure that the number of instances in the supernode affinity group does not exceed the number of nodes per instance in that queue.
What is your overall rating for this page?
Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot