Inference Deployment Overview

ModelArts helps you quickly deploy AI models as ready-to-use inference services and offers APIs to easily incorporate these services into your custom applications.

ModelArts allows you to deploy a model as a real-time service that provides a real-time test UI and monitoring capabilities. This service provides a callable API. Real-time inference is used in situations that need fast responses, like online intelligent customer service and autonomous driving decisions.

Product Architecture

ModelArts inference deployment adopts a cloud-native microservices architecture designed to provide high-availability, high-concurrency, and low-latency model inference services. Its core architectural layers are as follows:

Access layer: Provides standard RESTful APIs, supports HTTPS encrypted transmission, features a built-in high-performance gateway, and supports multiple load-balancing scheduling algorithms for traffic distribution.
Service layer:
Inference engine: Supports various backend engines (such as vLLM, Triton Inference Server, TensorFlow Serving, PyTorch Serve, MindSpore Serving, etc.) to accommodate models from different frameworks.

Scheduling and management: Handles instance elastic scaling, health checks, version management, and gray releases.
Resource layer: Interfaces with underlying GPU/NPU bare metal servers and container engines (CCE) to deliver elastic compute provisioning.
Monitoring layer: Real-time monitoring of key metrics including QPS, latency, GPU utilization, error rates, time to first token (TTFT), and throughput.

Core Capabilities

Resource scheduling optimization
Traditional scheduling approaches struggle to meet the complex orchestration requirements of multi-role distributed inference. PD-disaggregated deployments demand high levels of role-coordinated scheduling and strict resource ratios, which can easily lead to inefficient task allocation and deployment complexity. KubeInfer supports multi-unit inference instance resource orchestration, leveraging Volcano gang scheduling and elastic gang scheduling to achieve unified multi-role scheduling and elastic degradation, helping you deploy inference services across multi-server scales. Additionally, it offers four differentiated scheduling policies: high-availability, high-utilization, high-performance, and affinity scheduling. These meet needs for reliability, efficient resource use, fast communication, and custom control.
Elastic inference
Supports both manual and automatic scaling (triggered by schedules or metrics) at the deployment dimension to improve resource utilization. In prefill-decode disaggregation scenarios, it provides elastic degradation (P-to-D resource reallocation) capabilities: during a fault, P-role replicas are reduced to free up resources that immediately prioritize D-role demands, maintaining service continuity in a degraded state and automatically restoring full replicas once resources recover. Rolling upgrades combined with graceful shutdown enable seamless transitions during resource expansion or contraction, ensuring service continuity and stability.
Request scheduling optimization
Intelligent routing policies: The platform provides six fine-grained routing policies (round robin, source IP hash, least connections, minimum TTFT, overall loads, and SLO priority) to flexibly distribute traffic according to service requirements and ensure timely request processing. Static load balancing supports round robin, source IP hash, and least connections policies. Intelligent routing is supported exclusively in dedicated resource pools, and only one policy can be selected per deployment.
High reliability
Four-tier automatic rebuild: Supports automatic rebuild across four levels: pod level, unit replica level, unit level, and deployment-replica level, automatically selecting the rebuild granularity based on the fault's impact scope. Combined with automatic restart (scheduling migration following hardware faults) and health-check self-healing mechanisms, this minimizes fault impact. Graceful shutdown: Ensures that in-flight requests complete smoothly during service decommissioning, upgrades, and scale-in, preventing forced interruptions.
Ease of use
Out-of-the-box experience: Pre-integrates a wide range of model assets (such as the DeepSeek, GLM, and Qwen3 series) supporting one-click deployment for rapid inference service deployment. It supports both custom deployments of built-in models and direct pulling of models from the vLLM-Ascend open-source community to meet diverse scenario requirements.

Terms

**Table 1** Inference deployment modes
Inference Deployment Mode	Definition	Characteristics	Use Case
Basic mode (single-node deployment)	This is the simplest and most basic operation mode, where only one AI accelerator card on a single node is used for inference. One inference unit completes all tasks. It handles all computing, pre-processing, and post-processing on the same instance. This setup is lightweight, simple, and easy to maintain. Only one inference unit is needed to provide the service.	Only one inference unit needs to be configured, which independently hosts the entire inference service.	The structure is simple, requiring no complex distributed communication configuration, and the debugging threshold is low. This mode is preferred for small-parameter-scale model training, small-scale inference services, and personal developer debugging and learning. It is ideal when the model size and data volume are small enough to fit into the video memory of a single PU.
Multi-role separation (multi-node PD co-location)	Multi-node prefill-decode (PD) co-location refers to deploying both the prefill and decode stages of LLM inference onto the same set of compute nodes (such as NPUs/GPUs) to share KV cache resources. This mode is suitable for resource-constrained scenarios or when architecture simplification is required. When using the vLLM framework in a co-location scenario, you are advised to set the first unit as the vLLM master node and place the remaining worker nodes in other units.	Multiple units are configured, and the prefill and decode units are deployed on the same compute node.	In this mode, KV cache resources are shared. This approach is suitable for scenarios with limited resources or where architectural simplicity is desired.
Multi-role separation (multi-node PD disaggregation)	PD disaggregation is an optimized technical architecture designed specifically for the inference phase of LLMs. Its core principle is to decouple the two main stages of LLM inference: the compute-intensive prefill phase and the memory-intensive decode phase. These stages are then executed independently on separate compute units, such as GPUs or AI chips.	Multiple units are configured, with prefill and decode units deployed on different physical nodes to completely isolate resources. This approach offers optimal performance but at a higher cost.	This mode is suitable for large-scale inference tasks, scenarios requiring high availability and fault tolerance, and scenarios where resource isolation and optimization are necessary.

**Table 2** Inference deployment terms
Term	Definition
Service	A service is the resource carrier of the inference platform. Its core function is to deploy existing models in the production environment and provide real-time prediction and logical inference capabilities for external systems.
Deployment	A deployment is used to manage the deployment and update of models. It supports rolling update and rollback to ensure high availability and load balancing of models.
Deployment replica	A deployment can manage multiple replicas to ensure high availability and load balancing of models. A deployment contains one or more replicas. Each replica consists of a group of independently running units, and all replicas have the same configuration.
Unit	A unit is the smallest logical deployment component for inference services. It is a collection of containers that perform different functions in the system. For example, in PD-disaggregated deployment mode, units P and D represent prefill and decode, respectively. In basic mode, a deployment replica contains one unit. In multi-role separation mode, a deployment replica can contain multiple units, and each unit has its own image and specifications.
Pod	A pod is the smallest deployable unit in Kubernetes. Pods are the actual containers executing the inference tasks.

Figure 1 Relationship between deployment, deployment replica, unit, and pod
Click to enlarge

Features Overview

**Table 3** Features supported by new-version inference
Category	Supported Features
Basic information	Synchronous inference is supported in various scenarios, including single-node deployment, multi-node PD co-location, and multi-node PD disaggregation.
Basic information	Supports asynchronous inference.
Image settings	Supports the following SWR images: private images, shared images, and SWR enterprise edition images.
	Supports ModelArts preset images.
	Allows you to register images using ModelArts image management.
	Allows you to select a pre-warmed image. (This function is restricted. To use it, submit a service ticket.)
Model settings	Allows you to mount OBS, PFS, and SFS Turbo during model configuration.
Model settings	Allows you to select a warm-up model during model configuration.
Other settings	Allows you to set environment variables and boot commands.
	Allows you to perform a health check on services.
	Supports affinity scheduling: Single-node deployment supports affinity scheduling and allows selecting the IP of an affinity node. Multi-node multi-PU deployment also supports affinity scheduling, where roles can be configured to select the IP of an affinity node.
	Supports graceful shutdown.
	Supports automatic restart upon faults.
	Allows users to set traffic limiting for inference services.
	Allows you to set the request size limit and request timeout for inference services.
	Supports rolling upgrade and allows you to set the maximum invalid instances and extra instances of an inference service by percentage.
Access protocols and authentication modes	Allows you to set protocols for inference services: WebSocket, Server-Sent Events (SSE), and HTTP/HTTPS.
Access protocols and authentication modes	Allows you to set the authentication mode for inference services: none, IAM token, and API key.
Log management	Allows you to view real-time logs of inference services.
Log management	Allows you to view LTS logs of inference services.
Request scheduling policy	Allows you to configure static load balancing policies, including the polling, source IP hash, and least connections.
Request scheduling policy	Allows you to configure the minimum TTFT, overall loads, and SLO priority for inference services.
Access network	Allows you to access a real-time service through a public network.
	Allows you to query, create, approve, modify, and delete private network access requests.
	Allows you to access a real-time service through a private network using Elastic Load Balance (ELB).
Traffic weight	Allows you to configure traffic mirroring and traffic weights.
Monitoring	Allows you to view service-level metrics, including the request QPS, number of requested services, number of current connections, service request latency, CPU usage, NPU usage, memory usage, GPU usage, and service network traffic rate.
	Supports custom monitoring metrics.
	Allows you to view metrics related to LLMs, including TTFT, time per output token (TPOT), and number of input and output tokens.
Authorization management	Allows you to create API keys and bind or unbind API keys to or from specific services.
Scaling	Supports manual scaling, allowing you to manually change the number of instances.
Scaling	Supports automatic scaling, allowing you to set scaling rules to automatically change the number of instances.

Inference Service Deployment Process

Prepare an inference image. For details, see Preparing an Inference Image.
Prepare model and code files in advance and upload them to OBS, OBS parallel file systems, or SFS Turbo.
Create a resource pool. For details, see Creating a Dedicated Resource Pool.
Configure real-time service information. For details, see Configuring Service Information.
Deploy a real-time service. For details, see Deploying a Real-Time Inference Service Using a Single Node, Deploying a Real-Time Inference Service Using Multi-Node PD Co-location, or Deploying a Real-Time Inference Service Using Multi-Node PD Disaggregation.
Test and debug the real-time service. For details, see Testing a Real-Time Service.

Accessing an Inference Service

If a real-time service is in the Running state, it has been deployed. This service provides a standard RESTful API for calling. When accessing a real-time service, you can choose the authentication method, access channel, and transmission protocol that best suit your needs. These three elements make up your access requests and can be mixed and matched without any interference. For example, you can use different authentication methods for different access channels and transmission protocols.

Figure 2 Authentication method, access channel, and transmission protocol
Click to enlarge

ModelArts allows you to access real-time services in the following authentication modes (HTTPS requests are used as examples):

No authentication: No authentication is required.
Token-based authentication: Use Huawei Cloud Identity and Access Management (IAM) for authentication. The validity period of a token is 24 hours. When using a token for authentication, cache it to prevent frequent calls.
API key authentication: API key authentication provides a straightforward method for securing APIs with basic access control needs. Create an API key in the Huawei Cloud console and include it in the request header for API calls.

ModelArts allows you to call APIs to access real-time services in the following ways (HTTPS requests are used as examples):

Accessing a Real-Time Service Through a Public Network: By default, ModelArts inference uses the public network to access real-time services. A standard, callable RESTful API is provided after deployment of a real-time service.
Accessing a Real-Time Service Through a Private Network: ModelArts offers private network connection. When you create a private network connection request, it automatically sets up a VPCEP to connect your VPC with the real-time inference service securely.
Accessing a Real-Time Service Through ELB: ELB allows secure and fast access to ModelArts real-time services from the internet.

Real-time service APIs are accessed using HTTPS by default. Additionally, the following transmission protocols are also supported:

Accessing a Real-Time Service Using WebSocket: WebSocket simplifies data exchange between the client and server and allows the server to proactively push data to the client. In the WebSocket API, if the initial handshake between the client and server is successful, a persistent connection can be established between them and bidirectional data transmission can be performed.
Accessing a Real-Time Service Using Server-Sent Events: Server-Sent Events (SSE) primarily facilitates unidirectional real-time communication from the server to the client, such as streaming LLM responses. In contrast to WebSockets, which provide bidirectional real-time communication, SSE is designed to be more lightweight and simpler to implement.

Parent Topic: Inference Deployment (New Version)

Previous topic: Inference Deployment (New Version)

Next topic: Scenario-based Deployment