Deploying a Real-Time Inference Service Using Multi-Node PD Disaggregation
Real-time services have two versions. The new version is recommended.
Overview
In ultra-large-scale LLM inference scenarios (10B to 100B+ parameters), traditional single-node deployment faces pain points such as insufficient VRAM, high time-to-first-token (TTFT) latency, low concurrent throughput, and poor resource utilization. Consequently, it cannot meet the high-concurrency, low-latency, and high-stability demands of enterprise-grade generative AI (text generation, multimodal generation).
Multi-node PD disaggregation is designed specifically for generative scenarios such as LLMs, text-to-image generation, and long-context dialogues. It is perfectly suited for the following core scenarios:
- Ultra-large-scale model inference (34B/70B/175B+ parameters): Scenarios where the VRAM of a single node cannot accommodate the complete model weights.
- High-concurrency request scenarios: Scenarios requiring boosted inference throughput and reduced TTFT.
- Resource utilization optimization: Scenarios that need to align with the differentiated computing demands of the prefill phase (compute-intensive) and the decode phase (concurrency-intensive).
Core Principles
Generative inference for LLMs is divided into two core phases, each with significantly different compute and VRAM requirements:
- Prefill: Processes the user's input prompt and calculates the input sequence's KV cache all at once. This phase requires high VRAM and high compute, but has short execution time and low concurrency.
- Decode: Generates the output token-by-token based on the KV cache. This phase requires low VRAM and low compute, but has long execution time and high concurrency.
Deployment mode comparison:
- Multi-node prefill-decode (PD) co-location (multi-role separation – co-location): PD co-location refers to deploying both the prefill and decode stages of LLM inference onto the same set of compute nodes (such as NPUs/GPUs) to share KV cache resources. This mode is suitable for resource-constrained scenarios or when architecture simplification is required. When using the vLLM framework in a co-location scenario, you are advised to set the first unit as the vLLM master node and place the remaining worker nodes in other units. For details about multi-node PD co-location, see Deploying a Real-Time Inference Service Using Multi-Node PD Co-location.
- Multi-node PD disaggregation (multi-role separation – isolation): In the multi-role separation mode, the prefill and decode units are deployed on different physical nodes to achieve complete resource isolation. This mode delivers optimal performance but incurs higher costs. This topic focuses on multi-node PD disaggregation.
Based on the ModelArts multi-role separation deployment mode, the two phases are split across different nodes to achieve multi-node collaboration:
- Unit splitting: Creates two types of independent inference units, prefill units (high-compute nodes) and decode units (high-concurrency nodes).
- Intelligent routing: The platform automatically distributes requests. A user request is first processed by the prefill unit to generate the KV cache, and is then forwarded to the decode unit to complete the token-by-token generation.
- Resource isolation: The two types of units are scheduled and scaled independently, allowing resources to be matched precisely on demand.
- Multi-node collaboration: Relying on the distributed capabilities of dedicated resource pools, the multi-node cluster communicates via a high-speed internal network to complete cross-node PD task collaboration.
Constraints
- Deployment mode constraints: The multi-role separation mode must be selected, consisting of at least one prefill unit and one decode unit.
- Resource specification constraints: Supports dedicated resource pools (including heterogeneous ones) and public resource pools. Physical pools that have already created logical subpools are currently not supported. When deploying on a heterogeneous resource pool, the instance specifications of the prefill and decode units must match the specifications of the node pool.
- Image/Model requirements: The model image must be adapted to the multi-role separation architecture and support independent deployment of prefill and decode; otherwise, the deployment will fail. The model must also support KV cache separation.
- Call mode constraints: Both synchronous and asynchronous calls are supported.
Prerequisites
- You have prepared data as instructed in Preparations.
- Your account is not in arrears to ensure available resources for running services.
- You have configured the inference service information as instructed in Configuring Service Information.
- The current account has the real-time inference deployment permission and dedicated resource pool scheduling permission. For details, see Configuring Agency Authorization for ModelArts with One Click.
Notes
Resource pools allocate quota for real-time services even when they are Abnormal or Stopped. If the quota is insufficient and no more services can be deployed, delete some abnormal services to release resources.
- Quota calculation:
The quota stays the same when you deploy real-time services with a dedicated resource pool. It only changes if you create, modify, or delete a resource pool.
- Usage metering:
Deploying real-time services in a dedicated resource pool is not metered. Only the usage of the dedicated resource pool itself is metered.
- Mounting SFS Turbo:
Before mounting an SFS Turbo file system to a real-time service, associate the dedicated resource pool network with the file system.
Configuring Basic Information
You can also click the service name to go to the service details page and configure deployment information on the Deploy tab page or by clicking Add Deployment in the upper right corner of the service details page.
On the Deploy Real-Time Service page, configure basic information, resource settings, model settings, unit settings, deployment management settings, and advanced settings.
| Parameter | Description | Example |
|---|---|---|
| Deployment Name | Name of the current deployment, which is used to identify and manage the deployment configuration of the real-time service. Enter a name as prompted. Only letters, digits, hyphens (-), and underscores (_) are supported. Max length: 128 characters | service |
| Description (Optional) | Brief description of the deployment. | / |
Configuring Resource Settings
| Parameter | Description | Example |
|---|---|---|
| Resource Pool Type | Both dedicated resource pools and public resource pools support multi-role separation deployment.
| Dedicated Resource Pool |
| Resource Pool | If Resource Pool Type is set to Dedicated Resource Pool, click Select Resource Pool, select the corresponding specifications in the dedicated resource pool specifications area, and click OK. The physical pools with logical subpools created are not supported. If no dedicated resource pool is available, create one. You can choose a heterogeneous resource pool for deploying real-time services. If you use a heterogeneous dedicated resource pool, ensure the real-time service's instance specifications match the pool's specifications. | / |
| Deployment Replicas | A deployment instance consists of units capable of independently completing an inference task. Specify the number of instance replicas for the deployment. The value ranges from 1 to 128. If there is one replica, only one service instance runs. This is a standard single-node setup. With three replicas, three identical service instances run simultaneously. This spreads out requests, improves handling multiple tasks, and provides basic high availability. | 2 to 4 |
| Scheduling Policy | Two scheduling policies are available: HA scheduling and Compact scheduling. HA scheduling is enabled by default.
Affinity scheduling is set in More Settings of the Unit Settings. For details, see Table 2. | HA scheduling |
| Scheduling Priority | Resource scheduling priority for service deployment. When Service Resource Pool Settings is set to Dedicated resource pool, set this parameter. The platform handles jobs by prioritizing them from highest to lowest. Range: 1 (lowest) to 3 (highest). If multiple jobs have the same priority, they are scheduled in the order they were submitted. The final schedule depends on available resources. When resources are enough and priorities match, the first jobs submitted are scheduled first. | 1 |
Configuring Model Settings
| Category | Parameter | Description | Example |
|---|---|---|---|
| Model Source | Platform asset | When deploying a real-time service, you can select either Platform asset or Custom Model as the model source. When selecting Platform asset, the model originates from the ModelArts asset center. Click a card to select a model. You can choose from either Preset Models or My Models:
Supported model assets depend on the resource pool specifications. Select a dedicated resource pool or switch to the public resource pool and try again. | Platform asset |
| Custom Model | When selecting a custom model, you can select the model storage type. Set the model storage address and mount path. Supported storage types for custom models: OBS buckets, OBS parallel file systems, SFS Turbo, and pre-warmed models. For details about the parameters, see Storage Mounting. | Custom Model |
Configuring Prefill Inference Units
For multi-role PD disaggregation, set the inference service deployment mode to Multi-role.
Multi-role: Requires configuring multiple inference units as needed, with each unit corresponding to a specific role within the inference deployment instance. Adding or scaling roles requires adding new units. Multiple units are combined to form a complete inference deployment instance, suitable for scenarios such as PD co-location or disaggregation.
xPyD general ratio rules
x = Number of prefill units; y = Number of decode units; x + y = Total number of units
Example: 1P2D (1 prefill unit + 2 decode units) is the standard configuration for mainstream online production environments.
For details about the inference unit parameters, see the table below. During the configuration, you can click Preview Deployment in the upper right corner to view the topology of the current deployment, as shown in Viewing the Service Deployment Topology.
You can click Clone to copy the configuration of an existing inference unit, or click Delete to delete an inference unit that is no longer used.
| Parameter | Description | Example |
|---|---|---|
| role-0 | The default value is role-0, which can be modified. Enter up to 16 characters. Only lowercase letters, digits, and hyphens (-) are supported, and it must start and end with a letter or digit. | role-prefill |
| Unit Replicas | For multi-role separation, you can set or adjust the number of unit replicas to improve the unit throughput and response speed. The value ranges from 1 to 100. Set this parameter based on the number of concurrent requests. | 1 |
| Specification Type > Preset | Public resource pools only support preset specifications. Dedicated resource pools support preset specifications and custom specifications. Select an available unit instance specification and set the number of instances per replica. Total resource requirements of the inference unit = Unit instance specifications x Number of resource instances. For example, if the unit instance specification provides eight accelerators and you need 32 in total, enter 4. Prefill instance type: High-compute GPU/NPU specifications recommended for compute-intensive prefill. NOTE: Actual consumption will be slightly higher than the selected specification due to system overhead. | / |
| Specification Type > Custom | Only dedicated resource pools support custom specifications. Choose a custom specification if the preset ones do not fit your needs. GPU virtualization is enabled for this resource pool. Resources will be strictly allocated based on request volume. Configure specific vCPUs and memory. For details, see Viewing Details About a Real-Time Service.
When deploying a real-time service, you may select a heterogeneous dedicated resource pool—one that contains multiple architectures or specifications. For example, a resource pool might include node pool 1 (x86 architecture, CPU-only, 8 vCPUs, 32 GB), node pool 2 (x86 architecture, CPU-only, 8 vCPUs, 64 GB), and node pool 3 (Arm architecture, NPU-accelerated, 192 vCPUs, 1536 GB). When deploying a real-time service using such a heterogeneous resource pool, you can first select the node pool specifications, for example, node pool 1 with CPU-only specifications, and then select the job specifications required for deploying the service, such as the number of CPUs, memory, or accelerators required for deploying the real-time service. If you have node pools with the same specification, like node pool 1 and node pool 2, and you choose node pool 1 during setup, the service may still be deployed in either node pool 1 or node pool 2. This depends on which pool has better resource usage at the time. When deploying multiple inference units, you cannot use heterogeneous node pools. All inference units must be in the same node pool. | / |
| Image Type | Select the source of the inference image.
| / |
| Environment Variables (Optional) | Inject environment variables into the pod. To ensure data security, do not enter sensitive information, such as plaintext passwords, in environment variables. Environment variables are key-value pairs. For example, the key is A and the value is AAAA. Only letters, digits, underscores (_), hyphens (-), and periods (.) are supported. They cannot start with a digit and cannot exceed 64 characters. The value of an environment variable cannot be in HTML format, such as <p>, <^>, and <...>. You can upload an Excel file to batch import environment variables. Only .xlsx and .xls files are supported, and up to 100 parameters can be uploaded. To ensure correct parsing, fill in the file strictly according to the template format. To download the template, click Download Template. Click Local Upload. In the dialog box that is displayed, click Add File to import the local Excel file. Check the parsed environment variable key values. If the check result is To be modified, modify the values as required and upload the file again. After the check result is Passed, select the key values and click OK to complete the batch upload of environment variables. | / |
| Mount File Storage | Supports mounting file storage and specifying artifact dump path. Supported storage types for file storage: OBS buckets, OBS parallel file systems, SFS Turbo, and pre-warmed models. Supported storage type for artifact dumping: OBS parallel file system You can add up to 15 storage paths for file storage and 10 storage paths for artifact dump. Artifact dump cannot share the same OBS parallel file system with file or model mounting. For details about the storage type, storage address, and mount path, see Storage Mounting. | / |
| Health Check | Configure probes to monitor the model health. It can only be configured when the health check API is configured in the inference image. Otherwise, the model deployment fails. The following probes are supported:
For details about the three types of probes involved in health check, see Real-Time Service Health Check. | / |
| Boot Command | Set the service boot command. | / |
| More Settings > Automatic Rebuild | When enabled, if a pod restarts due to deployment configuration changes or failures, the platform will automatically rebuild it using the selected policy. If disabled, the platform will not intervene. For details, see Auto Rebuild upon a Real-Time Service Fault. | / |
| More Settings > Auto Restart | When enabled, if an NPU, switch, or hardware fault is detected, services on the faulty node are automatically rescheduled. Some capabilities are only supported by Snt9b and Snt9b2 resources. For details, see Auto Restart upon a Real-Time Service Fault. To ensure service continuity, configure multiple instances. | / |
| More Settings > Graceful Shutdown | Enabling this feature allows you to configure shutdown timeouts and commands. This prevents in-progress requests from being forcefully interrupted, thereby enhancing the availability and stability of the system. If you have configured health checks and set a large sleep value in this command, it will result in a longer restart or stop time for the container when the health check fails.
| / |
| More Settings > Affinity Scheduling | You can configure the node affinity type and strength to flexibly schedule workloads in a resource pool. If no nodes are specified, the pods will be randomly scheduled according to the default cluster scheduling policy. After this function is enabled, you can refine the pod deployment policy.
In the Add Node list, select the nodes that meet the preceding configuration rules. When you choose a pre-warmed model, only the pre-warmed nodes appear on the affinity scheduling page. Unwarmed nodes do not show. A message appears stating: The selected model is pre-warmed and will deploy automatically on the best node. Specifying a node might cause warmup to fail. | / |
| More Settings > Container User ID | If this option is selected, enter the user ID and user group ID (optional). | / |
| Authentication Credential | Displays when OBS is used for mounting and local storage acceleration is not enabled, or System Log Reporting is selected in Table 5. A secret is used to verify identity and authorize access. In the information security and identity authentication fields, a secret is a key mechanism to ensure that only authorized users can access the system, resources, or services.
| / |
Configuring Decode Inference Units
Click Add Inference Unit or click Clone to copy the existing inference unit settings and configure the decode inference units.
For details about the inference unit parameters, see the table below. During the configuration, you can click Preview Deployment in the upper right corner to view the topology of the current deployment, as shown in Viewing the Service Deployment Topology.
| Parameter | Description | Example |
|---|---|---|
| role-1. | The default value is role-1, which can be modified. Enter up to 16 characters. Only lowercase letters, digits, and hyphens (-) are supported, and it must start and end with a letter or digit. | role-decode |
| Unit Replicas | For multi-role separation, you can set or adjust the number of unit replicas to improve the unit throughput and response speed. The value ranges from 1 to 100. Set this parameter based on the number of concurrent requests. | 2 |
| Specification Type > Preset | Dedicated resource pools support preset specifications and custom specifications. Select an available unit instance specification and set the number of instances per replica. Total resource requirements of the inference unit = Unit instance specifications x Number of resource instances. For example, if the unit instance specification provides eight accelerators and you need 32 in total, enter 4. For decode inference units, you are advised to select low-compute CPU/NPU specifications to better adapt to decode streaming generation. NOTE: Actual consumption will be slightly higher than the selected specification due to system overhead. | / |
| Specification Type > Custom | Only dedicated resource pools support custom specifications. Choose a custom specification if the preset ones do not fit your needs. GPU virtualization is enabled for this resource pool. Resources will be strictly allocated based on request volume. Configure specific vCPUs and memory. For details, see Viewing Details About a Real-Time Service.
When deploying multiple inference units, you cannot use heterogeneous node pools. All inference units must be in the same node pool. | / |
| Image Type | Select the source of the inference image.
| Custom Images > SWR |
| Environment Variables (Optional) | Inject environment variables into the pod. To ensure data security, do not enter sensitive information, such as plaintext passwords, in environment variables. Environment variables are key-value pairs. For example, the key is A and the value is AAAA. Only letters, digits, underscores (_), hyphens (-), and periods (.) are supported. They cannot start with a digit and cannot exceed 64 characters. The value of an environment variable cannot be in HTML format, such as <p>, <^>, and <...>. You can upload an Excel file to batch import environment variables. Only .xlsx and .xls files are supported, and up to 100 parameters can be uploaded. To ensure correct parsing, fill in the file strictly according to the template format. To download the template, click Download Template. Click Local Upload. In the dialog box that is displayed, click Add File to import the local Excel file. Check the parsed environment variable key values. If the check result is To be modified, modify the values as required and upload the file again. After the check result is Passed, select the key values and click OK to complete the batch upload of environment variables. | / |
| Mount File Storage | Supports mounting file storage and specifying artifact dump path. Supported storage types for file storage: OBS buckets, OBS parallel file systems, SFS Turbo, and pre-warmed models. Supported storage type for artifact dumping: OBS parallel file system You can add up to 15 storage paths for file storage and 10 storage paths for artifact dump. Artifact dump cannot share the same OBS parallel file system with file or model mounting. For details about the storage type, storage address, and mount path, see Storage Mounting. | Object Storage Service – Bucket |
| Health Check | Configure probes to monitor the model health. It can only be configured when the health check API is configured in the inference image. Otherwise, the model deployment fails. The following probes are supported:
For details about the three types of probes involved in health check, see Real-Time Service Health Check. | / |
| Boot Command | Set the service boot command. | / |
| More Settings > Automatic Rebuild | When enabled, if a pod restarts due to deployment configuration changes or failures, the platform will automatically rebuild it using the selected policy. If disabled, the platform will not intervene. For details, see Auto Rebuild upon a Real-Time Service Fault. | / |
| More Settings > Auto Restart | When enabled, if an NPU, switch, or hardware fault is detected, services on the faulty node are automatically rescheduled. Some capabilities are only supported by Snt9b and Snt9b2 resources. For details, see Auto Restart upon a Real-Time Service Fault. To ensure service continuity, configure multiple instances. | / |
| More Settings > Graceful Shutdown | Enabling this feature allows you to configure shutdown timeouts and commands. This prevents in-progress requests from being forcefully interrupted, thereby enhancing the availability and stability of the system. If you have configured health checks and set a large sleep value in this command, it will result in a longer restart or stop time for the container when the health check fails.
| / |
| More Settings > Affinity Scheduling | You can configure the node affinity type and strength to flexibly schedule workloads in a resource pool. If no nodes are specified, the pods will be randomly scheduled according to the default cluster scheduling policy. After this function is enabled, you can refine the pod deployment policy.
In the Add Node list, select the nodes that meet the preceding configuration rules. When you choose a pre-warmed model, only the pre-warmed nodes appear on the affinity scheduling page. Unwarmed nodes do not show. A message appears stating: The selected model is pre-warmed and will deploy automatically on the best node. Specifying a node might cause warmup to fail. | / |
| More Settings > Container User ID | If this option is selected, enter the user ID and user group ID (optional). | / |
| Authentication Credential | Displays when OBS is used for mounting and local storage acceleration is not enabled, or System Log Reporting is selected in Table 5. A secret is used to verify identity and authorize access. In the information security and identity authentication fields, a secret is a key mechanism to ensure that only authorized users can access the system, resources, or services.
| / |
Configuring Deployment Management Settings
The following table describes the deployment management settings.
| Parameter | Description |
|---|---|
| Container Protocol | The network transmission protocol used by the container. Configure it according to the API definitions in your actual setup. If Service Protocol is set to HTTP or HTTPS in Table 2, Container Protocol can be set to HTTP or HTTPS. If Service Protocol is set to WSS or WS in Table 2, the container protocol is the same as the service protocol by default and is not displayed on the console. |
| Container Port | The port number that the image listens on. Requests are sent to the instance through this port. The port number must be the same as the port number in your image. |
| More Settings > Deployment Timeout (Minutes) | The wait time before a single service times out. This value includes both the deployment and startup time. Set this parameter properly. The system will cancel the deployment if it exceeds the time limit. |
| More Settings > Max Surge Replicas (%) | The maximum number of replicas that can be created above the target count during a rolling upgrade When percentages are used, the instance count is rounded up. Example: When Max Surge Replicas (%) is set to 1% during a rolling upgrade of four instances, you can add one new instance at a time. When Max Surge Replicas (%) is set to 100% during a rolling upgrade of two instances, you can add two new instances immediately. For details about rolling upgrade, see Rolling Upgrades for Real-Time Service Deployment. |
| More Settings > Max Unavailable Replicas (%) | The maximum number of replicas that can be unavailable relative to the target count during a rolling update When percentages are used, the instance count is rounded down. Example:
If Max Unavailable Replicas equals the target replica count, there is a risk of downtime (Min Available Replicas = Total Replicas – Max Unavailable Replicas). Example: Setting Max. Invalid Instances to 100 allows the service stay running when an instance fails because of quick recovery. Yet, the service will not be able to perform predictions, potentially leading to interruptions. To ensure a hitless upgrade, set Max. Invalid Instances to 1%. (This requires extra resources. Make sure you have enough, or the update will fail.) For details about rolling upgrade, see Rolling Upgrades for Real-Time Service Deployment. |
Configuring Advanced Settings
| Parameter | Description |
|---|---|
| Key Settings | Inference services support secret mounting. This uses encryption to protect sensitive data and securely mount it to containers. To protect your data, do not enter sensitive information in plaintext.
Choose whether to select Associate Image User Group ID. The user group ID of the user who starts the image can be associated. After the association, the secret mounting security is improved. |
| Intelligent Routing Policy | If intelligent routing is enabled, set intelligent routing policies. The following policies are supported:
For details, see Intelligent Routing Policy. |
| Custom Metric Collection | If this function is enabled, you need to input the metric collection port details. Custom metrics will send data to AOM's Prometheus instances for viewing on the AOM console. To query custom metrics, see Viewing Performance Metrics of a Real-Time Service on AOM and Viewing Performance Metrics of a Real-Time Service on ModelArts. Constraints
Data Format of Custom Metrics The format of custom metrics data must comply with the open metrics specifications. That is, the format of each metric must be: <metric_name>{<tag_name>=<tag_value>,...} <sample_value> [timestamp_in_millisecond] The following shows an example (the comment starts with #, which is optional): # HELP http_requests_total The total number of HTTP requests.
# TYPE http_requests_total gauge
html_http_requests_total{method="post",code="200"} 1656 1686660980680
html_http_requests_total{method="post",code="400"} 2 1686660980681 |
| System Log Reporting | Displays only for NPU dedicated resource pools. When enabled, system logs are mounted to a fixed parallel file system for O&M engineers to analyze. System logs are stored for 30 days and will be automatically deleted after that. The mount path of system logs cannot be changed. |
Confirming Settings
On the Deploy Real-Time Service > Confirmation page, confirm the configuration information and click Confirm Deployment.
Deploying a service generally requires a period of time, which may be several minutes or tens of minutes depending on the amount of your data and resources.
You can go to the real-time service list to check if the deployment is complete. Once the service status changes from Deploying to Running, the service is deployed.
After a real-time service is deployed, it is started immediately.
Follow-Up Operations
- For details about how to test a real-time service, compare the model effect in real time, and debug a real-time service using CloudShell, see Testing a Real-Time Service.
- For details about how to view real-time service details, see Viewing Details About a Real-Time Service.
- For details about how to modify, stop, and delete a real-time service, see Managing the Lifecycle of a Real-Time Service Deployment.
What is your overall rating for this page?
Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot
