Scheduling Policy

Overview

In Huawei Cloud resource management scenarios, users configuring inference services may need to optimize scheduling policies to enhance service performance or resource utilization. For instance, to ensure high availability, users typically select a high-availability scheduling policy, which requires service replicas (pods) to be distributed as evenly as possible across different nodes to minimize the impact of individual failures. However, in practice, users may find that existing scheduling configuration options lack the flexibility required for specific business needs, such as scenarios needing finer control over scheduling priorities, or those requiring a compact scheduling policy to maximize resource utilization. Faced with this challenge, users often wonder: How can scheduling policies be configured more flexibly on Huawei Cloud to adapt to diverse service requirements?

To address this, ModelArts offers multiple scheduling policies for real-time inference services in dedicated resource pools. By flexibly adjusting the distribution logic of service instances across cluster nodes, these policies satisfy three core service demands: highly reliable operations, efficient resource utilization, and custom deployment control.

Supported Scheduling Policies

Inference services support collaboration of various scheduling policies.

• HA scheduling: Pods within the same replica are placed together, but pods from different replicas are distributed across multiple nodes instead of one node. This reduces the risk of a single node failure affecting the service and helps maintain high availability.
• Compact scheduling: Pods within the same replica are placed together, and different replicas are packed onto high-load nodes using bin packing to reduce resource fragmentation.
• High performance scheduling: Multiple pods under the same deployment replica are forced to schedule to nodes under the same hyperplane network, thereby reducing communication latency and improving inference speed.
• Affinity scheduling: node affinity and anti-affinity.
• Single-replica resource instance scheduling: Pod groups of single-replica resource instances are strictly constrained to be scheduled within the same network performance domain.

High-Availability Scheduling

Concepts

High-availability scheduling is the default recommended policy for real-time inference services. Its core objective is to eliminate individual failure and guarantee continuous service availability. The scheduling policy automatically adapts the deployment granularity based on the resource pool type. When a supernode type resource pool (Snt9b23 or A5) is selected, the system operates at the hyperplane network granularity, where pods of the same replica are centrally deployed within the same hyperplane network, and pods of different replicas are distributed across different hyperplane networks. If another node type resource pool is used, the system operates at the node granularity, where pods of the same replica are centrally deployed on the same node, and pods of different replicas are distributed across different nodes. This prevents overall service unavailability caused by a single node or cabinet failure, while also supporting lossless switching during rolling updates to ensure zero workload interruption.

Configuration entry

Log in to the ModelArts console and choose Model Inference > Real-Time Inference. When deploying a real-time service, select HA scheduling in Resource Settings > Scheduling Policy. For details about how to deploy a real-time service, see Configuring Deployment Settings.

Constraints

• Only one of the three scheduling policies can be selected: high-availability scheduling, high utilization scheduling, and high-performance scheduling. By default, high-availability scheduling is enabled.
• If both HA scheduling and Affinity Scheduling are enabled, Affinity Scheduling takes precedence.
• When using supernode type (Snt9b23, A5) resource pools, the system schedules based on the hyperplane network granularity. This requires KubeInfer plugin version 7.6.3 or higher, and Volcano plugin version 1.22.40 or higher.

High-Utilization Scheduling

Concepts

The core objective of high-availability scheduling is to maximize resource utilization and reduce deployment costs. The scheduling policy automatically adapts the deployment granularity based on the resource pool type. When a supernode type resource pool (Snt9b23 or A5) is selected, the system operates at the hyperplane network granularity: multiple pods under the same deployment replica are scheduled to the same hyperplane network as much as possible, while pods of different deployment replicas use bin packing to preferentially schedule pods to hyperplane networks with higher resource consumption. If another node type resource pool is used, the system operates at the node granularity, utilizing bin packing to preferentially schedule pods to nodes with higher resource consumption. This reduces idle resource fragmentation across nodes and improves cluster resource utilization.

Use cases

This mode works well for situations needing moderate stability and lower costs, like non-core tests, offline inference transitions, and edge deployments with few resources. Common uses are checking model functions, low-traffic tests, and light inference on edge devices.

Configuration entry

Log in to the ModelArts console and choose Model Inference > Real-Time Inference. When deploying a real-time service, select Compact scheduling in Resource Settings > Scheduling Policy. For details about how to deploy a real-time service, see Configuring Deployment Settings.

Constraints

• Only one of the three scheduling policies can be selected: high-availability scheduling, high utilization scheduling, and high-performance scheduling. By default, high-availability scheduling is enabled.

• If both Compact scheduling and Affinity Scheduling are enabled, Affinity Scheduling takes precedence.
• When using supernode type (Snt9b23, A5) resource pools, the system schedules based on the hyperplane network granularity. This requires KubeInfer plugin version 7.6.3 or higher, and Volcano plugin version 1.22.40 or higher.

High-Performance Scheduling

Concepts

The core objective of high-performance scheduling is to minimize communication latency and enhance inference speed. The scheduling policy strictly constrains multiple pods under the same deployment replica to be scheduled within the same hyperplane network.

Use cases

Suitable for multi-node P/D disaggregation scenarios to reduce data transmission latency between prefill and decode nodes.

Configuration entry

Log in to the ModelArts console and choose Model Inference > Real-Time Inference. When deploying a real-time service, select High Performance Scheduling in Resource Settings > Scheduling Policy. For details about how to deploy a real-time service, see Configuring Deployment Settings.

Constraints

• Only one of the three scheduling policies can be selected: high-availability scheduling, high utilization scheduling, and high-performance scheduling. By default, high-availability scheduling is enabled.
• When both High Performance Scheduling and Affinity Scheduling are enabled simultaneously, both take effect if the node affinity and anti-affinity configurations in Affinity Scheduling are set to strong affinity; however, if they are set to weak affinity, the High Performance Scheduling setting takes precedence.
• High-performance scheduling can be enabled for supernode resource pools (node type Snt9b23 or A5), requiring KubeInfer plugin version 7.6.3 or higher, and Volcano plugin version 1.22.40 or higher.

Affinity Scheduling

Concepts

Affinity scheduling allows you to control where service replicas run by using node affinity or anti-affinity rules. This helps meet specific deployment needs. You can configure the node affinity type and strength to flexibly schedule workloads in a resource pool. If no nodes are specified, the pods will be randomly scheduled according to the default cluster scheduling policy.

Configuration entry

Log in to the ModelArts console and choose Model Inference > Real-Time Inference. When deploying a real-time service, select Affinity Scheduling in Unit Settings > More Settings. In the dialog box displayed on the right, configure the affinity scheduling policy. For details about how to deploy a real-time service, see Configuring Deployment Settings.

In the Add Node list, select the nodes that meet the preceding configuration rules. When a supernode type resource pool (Snt9b23 or A5) is selected, the node list is displayed in a tree structure. If the node type is Snt9b23, the hierarchy is supernode instance and node; if the node type is A5, the hierarchy is supernode instance, cabinet, and node.

In the affinity scheduling policy, a single account (not an IAM user) can add a maximum of 20 nodes by default. To increase the number of API keys that can be created, submit a service ticket.

When you choose a pre-warmed model, only the pre-warmed nodes appear on the affinity scheduling page. Unwarmed nodes do not show. A message appears stating: The selected model is pre-warmed and will deploy automatically on the best node. Specifying a node might cause warmup to fail.

Single-replica Resource Instance Scheduling Constraint

Concepts

Pod groups of unit replica resource instances are strictly constrained to be scheduled within the same network performance domain. The constraint scope can be set to either hyperplane network or cabinet.

• Hyperplane network: Pod groups of unit replica resource instances are strictly constrained to be scheduled within the same hyperplane network.
• Cabinet: Pod groups of unit replica resource instances are strictly constrained to be scheduled within the same cabinet, offering higher network communication efficiency compared to a hyperplane network. This is available when the unit instance specification is A5.

Use cases

This policy is suitable for large-scale EP deployment scenarios where a single P/D instance may contain multiple pods, minimizing data transmission latency between multiple Pods within the same instance.

Configuration entry

Log in to the ModelArts console and choose Model Inference > Real-Time Inference. When deploying a real-time service, set the number of resource instances in a single replica to a value greater than 1 in the unit settings, select Single-replica Resource Instance Scheduling Constraint, and then select the constraint scope. For details about how to deploy a real-time service, see Configuring Deployment Settings.

Constraints

• After you select Single-replica Resource Instance Scheduling Constraint, you can only select either Hyperplane Network or Cabinet. By default, Single-replica Resource Instance Scheduling Constraint is disabled.
• HA scheduling, Compact scheduling, and High Performance Scheduling are deployment dimensions and apply to all instances in a deployment. Affinity Scheduling is a unit dimension and applies to all instances in a single unit. Single-replica Resource Instance Scheduling Constraint are unit replica dimensions and apply to all resource instances of a single unit replica.
• Single-replica Resource Instance Scheduling Constraint can be enabled when the unit instance specification is Snt9b23 or A5. When the unit instance specification is Snt9b23, Hyperplane Network is available for selection; when the unit instance specification is A5, both Hyperplane Network and Cabinet are available for selection.