Deploying a Real-Time Inference Service Using a Single Node

Overview

In a single-node deployment, one inference unit completes all tasks. It handles all computing, pre-processing, and post-processing on the same instance. This setup is lightweight, simple, and easy to maintain. Only one inference unit is needed to provide the service.

Constraints

• Deployment limit: A user can create up to 20 real-time services.
• Deployment mode: Only the basic mode (non-multi-role separation/PD disaggregation) is supported.
• Node limit: Single-node deployment is required. Cross-node deployment and distributed scaling are not supported.
• Performance upper limit: The GPU, CPU, and memory of a single node are limited. High concurrency may cause out-of-memory (OOM) and increase latency.
• Low availability: If a node is faulty, the service is interrupted, and no automatic switchover is available.
• O&M restrictions: P/D ratio optimization and intelligent O&M are not supported.
• Scaling constraints: Only horizontal scaling by adding replicas (on the same node or nodes of the same specifications) is supported; role separation is not permitted.

Prerequisites

• You have prepared data as instructed in Preparations.
• Your account is not in arrears to ensure available resources for running services.
• You have configured the inference service information as instructed in Configuring Service Information.

Notes

Resource pools allocate quota for real-time services even when they are Abnormal or Stopped. If the quota is insufficient and no more services can be deployed, delete some abnormal services to release resources.

• Quota calculation:

  The quota stays the same when you deploy real-time services with a dedicated resource pool. It only changes if you create, modify, or delete a resource pool.

• Usage metering:

  Deploying real-time services in a dedicated resource pool is not metered. Only the usage of the dedicated resource pool itself is metered.

• Mounting SFS Turbo:

  Before mounting an SFS Turbo file system to a real-time service, associate the dedicated resource pool network with the file system.

Configuring Deployment Settings

1. Log in to the ModelArts console and choose Model Inference > Real-Time Inference. On the service list page, choose More > Add Deployment on the right of the service name.
   Figure 1 Adding a deployment
   

   You can also click the service name to go to the service details page and configure deployment information on the Deploy tab page or by clicking Add Deployment in the upper right corner of the service details page.

   Figure 2 Adding a deployment
   

2. On the Deploy Real-Time Service page, set basic information, resource settings, model settings, unit settings, deployment management settings, and advanced settings, and click Confirm Deployment. The Confirmation page is displayed, as shown in Confirming Settings.

• Basic information

  Table 1 Basic information

  Parameter	Description	Example
  Deployment Name	Name of the current deployment, which is used to identify and manage the deployment configuration of the real-time service. Enter a name as prompted. Only letters, digits, hyphens (-), and underscores (_) are supported. Max length: 128 characters	service
  Description (Optional)	Brief description of the deployment.	/

• Resource settings

  Table 2 Resource settings

  Parameter	Description	Example
  Resource Pool Type	Public Resource Pool Public resource pool for deploying the real-time service. The public resource pool supplies shared compute clusters assigned according to job parameters. Each job operates with its own isolated resources. This option offers cost-effective and flexible solutions for tasks like development and testing. Choosing the public resource pool might leave fewer resources available because of its limits. If this happens, join the queue and wait your turn. Dedicated Resource Pool Dedicated resource pool for deploying the real-time service. The resources provided in a dedicated resource pool are exclusive and more controllable. Use dedicated resource pools for core production services to secure exclusive resources.	Dedicated Resource Pool
  Resource Pool	If Resource Pool Type is set to Dedicated Resource Pool, click Select Resource Pool, select the corresponding specifications in the dedicated resource pool specifications area, and click OK. The physical pools with logical subpools created are not supported. If no dedicated resource pool is available, create one. You can choose a heterogeneous resource pool for deploying real-time services. If you use a heterogeneous dedicated resource pool, ensure the real-time service's instance specifications match the pool's specifications.	/
  Deployment Replicas	A deployment instance consists of units capable of independently completing an inference task. Specify the number of instance replicas for the deployment. The value ranges from 1 to 128. If there is one replica, only one service instance runs. This is a standard single-node setup. With three replicas, three identical service instances run simultaneously. This spreads out requests, improves handling multiple tasks, and provides basic high availability. In single-node setups, the number of replicas depends on the node's memory and hardware. Too many replicas can cause memory shortages, OOM errors, compute bottlenecks, and service freezes. We recommend deploying 1 to 4 replicas in single-node setups.	1
  Scheduling Policy	Two scheduling policies are available: HA scheduling and Compact scheduling. HA scheduling is enabled by default. HA scheduling: Pods from different replicas will be distributed across different nodes as evenly as possible, while multiple pods under the same replica will prioritize scheduling to the same node to ensure high availability for inference services. If both HA scheduling and Affinity Scheduling are enabled, Affinity Scheduling takes precedence. Compact scheduling: Enable bin packing for cluster workloads. The scheduler will prioritize placing pods on nodes with higher resource consumption to reduce idle resource fragmentation and improve cluster resource utilization. When both Compact scheduling and Affinity Scheduling are enabled, Affinity Scheduling takes precedence. Affinity scheduling is set in More Settings of the Unit Settings. For details, see Table 4.	HA scheduling
  Scheduling Priority	Resource scheduling priority for service deployment. When Service Resource Pool Settings is set to Dedicated resource pool, set this parameter. The platform handles jobs by prioritizing them from highest to lowest. Range: 1 (lowest) to 3 (highest). If multiple jobs have the same priority, they are scheduled in the order they were submitted. The final schedule depends on available resources. When resources are enough and priorities match, the first jobs submitted are scheduled first.	1

• Model settings

  Table 3 Model source parameters

  Category	Parameter	Description	Example
  Model Source	Platform asset	When deploying a real-time service, you can select either Platform asset or Custom Model as the model source. When selecting Platform asset, the model originates from the ModelArts asset center. Click a card to select a model. You can choose from either Preset Models or My Models: Preset Models: These are inference model assets built into ModelArts that you can select and use directly. My Models: These are models generated by importing a local model or by completing model pre-training or model fine-tuning on ModelArts. Select your model and click OK. Supported model assets depend on the resource pool specifications. Select a dedicated resource pool or switch to the public resource pool and try again.	Platform asset
  	Custom Model	When selecting a custom model, you can select the model storage type. Set the model storage address and mount path. Supported storage types for custom models: OBS buckets, OBS parallel file systems, SFS Turbo, and pre-warmed models. For details about the parameters, see Storage Mounting.	Custom Model

• Unit settings

  Table 4 Inference unit parameters

  Parameter	Description	Example
  Basic mode	Inference service deployment supports Basic mode and Multi-role. Basic mode: Requires configuring only one inference unit, which independently hosts the entire inference service. Select Basic mode for single-node deployment. For details about multi-role separation deployment, see Deploying a Real-Time Inference Service Using Multi-Node PD Co-location.	Basic mode
  Unit Replicas	If the deployment mode is Basic mode, the number of unit replicas is 1 by default and cannot be changed.	1
  Unit Name	Enter a custom unit name ranging from 0 to 16 characters. Only lowercase letters, digits, and hyphens (-) are supported, and it must start and end with a letter or digit.	role-0
  Specification Type	Select the hardware resource type when using a dedicated resource pool. Public resource pools only support preset specifications.	Preset
  Specification Type > Preset	Choose a specification and number of instances per replica. Total resource requirements of the inference unit = Unit instance specifications x Number of resource instances. For example, if the unit instance specification provides eight accelerators and you need 32 in total, enter 4.	/
  Specification Type > Custom	Use a custom specification if the preset ones do not fit your needs. GPU virtualization is enabled for this resource pool. Resources will be strictly allocated based on request volume. Configure specific vCPUs and memory. For details, see Viewing Details About a Real-Time Service. CPU cores: The value must be at least 0.01 with exactly two decimal places and should not exceed the total cores available in the resource pool. Memory: The memory value must be an integer of at least 4 and should not exceed the total memory available in the resource pool. NOTE: Actual consumption will be slightly higher than the selected specification due to system overhead. When deploying a real-time service, you may select a heterogeneous dedicated resource pool—one that contains multiple architectures or specifications. For example, a resource pool might include node pool 1 (x86 architecture, CPU-only, 8 vCPUs, 32 GB), node pool 2 (x86 architecture, CPU-only, 8 vCPUs, 64 GB), and node pool 3 (Arm architecture, NPU-accelerated, 192 vCPUs, 1536 GB). When deploying a real-time service using such a heterogeneous resource pool, you can first select the node pool specifications, for example, node pool 1 with CPU-only specifications, and then select the job specifications required for deploying the service, such as the number of CPUs, memory, or accelerators required for deploying the real-time service. If you have node pools with the same specification, like node pool 1 and node pool 2, and you choose node pool 1 during setup, the service may still be deployed in either node pool 1 or node pool 2. This depends on which pool has better resource usage at the time. When deploying multiple inference units, you cannot use heterogeneous node pools. All inference units must be in the same node pool.	/
  Image Type	Select the source of the inference image. Preset image: Use images from ModelArts asset management. You can tag them with details like supported specifications and frameworks for easy management. For details, see ModelArts Images. Custom image: Select your custom image. For details about how to create a custom image, see Creating a Custom Image for a Model. Use your own image via any of the following options: SoftWare Repository for Container (SWR): SWR is a secure, reliable, and easy-to-use container image management service that supports full lifecycle management of container images. Registered image: Select an image registered with ModelArts. Image URL: Enter the image path of a custom image. Pre-warmed image: Select an image pre-warmed in dedicated resource pool in advance. If the image warmup task is done and the status is normal, deployment speeds up automatically. The selected image path and the service to be deployed must be in the same region.	Custom Images
  Environment Variables (Optional)	Inject environment variables into the pod. To ensure data security, do not enter sensitive information, such as plaintext passwords, in environment variables. Environment variables are key-value pairs. For example, the key is A and the value is AAAA. Only letters, digits, underscores (_), hyphens (-), and periods (.) are supported. They cannot start with a digit and cannot exceed 64 characters. The value of an environment variable cannot be in HTML format, such as <p>, <^>, and <...>. You can upload an Excel file to batch import environment variables. Only .xlsx and .xls files are supported, and up to 100 parameters can be uploaded. To ensure correct parsing, fill in the file strictly according to the template format. To download the template, click Download Template. Click Local Upload. In the dialog box that is displayed, click Add File to import the local Excel file. Check the parsed environment variable key values. If the check result is To be modified, modify the values as required and upload the file again. After the check result is Passed, select the key values and click OK to complete the batch upload of environment variables.	/
  Mount File Storage	Mount persistent storage for data access or artifact dumping. Supported storage types for file storage: OBS buckets, OBS parallel file systems, SFS Turbo, and pre-warmed models. Supported storage type for artifact dumping: OBS parallel file system. You can add up to 15 storage paths for file storage and 10 storage paths for artifact dump. Available storage types are subject to the GUI. Artifact dump cannot share the same OBS bucket or PFS with file or model mounting. For details about storage parameters and requirements, see Storage Mounting.	Object Storage Service – Bucket
  Health Check	Configure probes to monitor the model health. It can only be configured when the health check API is configured in the custom image. Otherwise, the model deployment fails. The following probes are supported: Startup Probe: This probe checks if the instance has started. If a startup probe is configured, the liveness or readiness probe is not executed until the startup probe is successful, allowing sufficient time for the application to complete initialization. If the startup probe fails, the instance is restarted. If startup probe is not configured, the service status changes to success immediately after the service is scheduled. The service may be in the Running state and the prediction cannot be performed because the model is being loaded. Readiness Probe: This probe verifies whether the instance is ready to handle traffic. If the readiness probe fails (meaning the instance is not ready), the instance is taken out of the service load balancing pool. Traffic will not be routed to the instance until the probe succeeds. Liveness Probe: This probe monitors the application health status. If the liveness probe fails (indicating the application is unhealthy), the instance is automatically restarted. For more information about probe configurations and descriptions, see Real-Time Service Health Check.	/
  Boot Command	Set the service boot command.	/
  More Settings > Automatic Rebuild	When enabled, if a pod restarts due to deployment configuration changes or failures, the platform will automatically rebuild it using the selected policy. If disabled, the platform will not intervene. For details, see Auto Rebuild upon a Real-Time Service Fault.	/
  More Settings > Auto Restart	When enabled, if an NPU, switch, or hardware fault is detected, services on the faulty node are automatically rescheduled. Some capabilities are only supported by Snt9b and Snt9b2 resources. For details, see Auto Restart upon a Real-Time Service Fault. To ensure service continuity, configure multiple instances.	/
  More Settings > Graceful Shutdown	Enabling this feature allows you to configure shutdown timeouts and commands. This prevents in-progress requests from being forcefully interrupted, thereby enhancing the availability and stability of the system. If you have configured health checks and set a large sleep value in this command, it will result in a longer restart or stop time for the container when the health check fails. Shutdown Timeout (s): This parameter indicates the maximum length of time that can pass between when a Pod receives a stop signal to when it is forcefully stopped. It is used for the Pod to perform cleanup operations (such as closing connections, releasing resources, and saving states). Shutdown Command: The shutdown command is triggered when the container receives a stop signal, but it must be completed within the grace period or the shutdown times out. If this happens, the container will be forcefully stopped. You can use this command to perform operations such as closing database connections, releasing file handles, and stopping child processes.	/
  More Settings > Affinity Scheduling	You can configure the node affinity type and strength to flexibly schedule workloads in a resource pool. If no nodes are specified, the pods will be randomly scheduled according to the default cluster scheduling policy. After this function is enabled, you can refine the pod deployment policy. Node affinity (strong): The pod must be scheduled onto the specified node; otherwise, scheduling will not proceed. Node affinity (weak): The system will try to place the pod on the specified node, but it is not guaranteed. Node anti-affinity (strong): The pod must not be scheduled onto the specified node; otherwise, scheduling will not proceed. Node anti-affinity (weak): The system will try to avoid deploying the pod on the specified node, but it is not guaranteed In the Add Node list, select the nodes that meet the preceding configuration rules. When you choose a pre-warmed model, only the pre-warmed nodes appear on the affinity scheduling page. Unwarmed nodes do not show. A message appears stating: The selected model is pre-warmed and will deploy automatically on the best node. Specifying a node might cause warmup to fail.	/
  More Settings > Container User ID	If this option is selected, enter the user ID and user group ID (optional).	/
  Authentication Credential	Displays when System Log Reporting is selected in Advanced Settings. A secret is used to verify identity and authorize access. In the information security and identity authentication fields, a secret is a key mechanism to ensure that only authorized users can access the system, resources, or services. If the version of CCE Container Storage (Everest) in your dedicated resource pool is v2.4.204 or later and the cluster version is v1.28 or later, temporary credentials are enabled by default (no AK/SK is required, ensuring higher security). If your CCE Container Storage (Everest) version is too low or the cluster does not support temporary credentials, you must mount the credentials using DEW. When using DEW secrets to mount storage, you must include 'accessKeyId' and 'secretAccessKey' (corresponding to your AK and SK, respectively). Ensure the information provided is correct, or the function may not work as expected. To create a secret, click Create Secret to go to the DEW console. For details, see Creating a Secret. Set the AK/SK in the Secret key/value tab. Enter accessKeyId and secretAccessKey in the Key column. To obtain the values, go to the DEW console, and choose My Credentials > Access Keys.	/

• Deployment management settings

  Table 5 Deployment management settings

  Parameter	Description
  Container Protocol	The network transmission protocol used by the container. Configure it according to the API definitions in your actual setup. If Service Protocol is set to HTTP or HTTPS in Table 2, Container Protocol can be set to HTTP or HTTPS. If Service Protocol is set to WSS or WS in Table 2, the container protocol is the same as the service protocol by default and is not displayed on the console.
  Container Port	The port number that the image listens on. Requests are sent to the instance through this port.
  More Settings > Deployment Timeout (Minutes)	The wait time before a single service times out. This value includes both the deployment and startup time. Set this parameter properly. The system will cancel the deployment if it exceeds the time limit.
  More Settings > Max Surge Replicas (%)	The maximum number of replicas that can be created above the target count during a rolling upgrade When percentages are used, the instance count is rounded up. Example: When Max Surge Replicas (%) is set to 1% during a rolling upgrade of four instances, you can add one new instance at a time. When Max Surge Replicas (%) is set to 100% during a rolling upgrade of two instances, you can add two new instances immediately.
  More Settings > Max Unavailable Replicas (%)	The maximum number of replicas that can be unavailable relative to the target count during a rolling update When percentages are used, the instance count is rounded down. Example: When Max Unavailable Replicas (%) is set to 1 during a rolling upgrade of four instances, four instances must remain available. An old instance can only be deleted after its replacement starts running. When Max Unavailable Replicas (%) is set to 50 during a rolling upgrade of two instances, one old instance can be deleted immediately to release resources, but at least one instance must remain available. If you deploy a service with Max Unavailable Replicas (%) at 20, requiring five instances, and only one fails due to lack of resources while four start successfully, the deployment succeeds, and the service status becomes Running. If Max Unavailable Replicas equals the target replica count, there is a risk of downtime (Min Available Replicas = Total Replicas – Max Unavailable Replicas). Example: Setting Max. Invalid Instances to 100 allows the service stay running when an instance fails because of quick recovery. Yet, the service will not be able to perform predictions, potentially leading to interruptions. To ensure a hitless upgrade, set Max. Invalid Instances to 1%. (This requires extra resources. Make sure you have enough, or the update will fail.)

• Advanced settings

  Table 6 Advanced settings

  Parameter	Description
  Key Settings	Inference services support secret mounting. This uses encryption to protect sensitive data and securely mount it to containers. To protect your data, do not enter sensitive information in plaintext. Custom key: Enter the key, key value, and mount path. DEW: Data Encryption Workshop (DEW) is a comprehensive cloud data encryption service, including Key Management Service (KMS), Key Pair Service (KPS), and Dedicated Hardware Security Module (Dedicated HSM). To use DEW to configure the key, go to the DEW console to create a key, select the DEW key on the ModelArts console, and enter the mount path. Choose whether to select Associate Image User Group ID. The user group ID of the user who starts the image can be associated. After the association, the secret mounting security is improved. For details, see Secret Mounting.
  Intelligent Routing Policy	If intelligent routing is enabled, set intelligent routing policies. The following policies are supported: Round robin: Tasks are distributed sequentially to different nodes in order. This ensures an even distribution across the cluster and achieves load balancing. Source IP hash: The system calculates hash values using client IP addresses to route requests from the same IP address to the same node. Least connections: Requests are routed to the node with the fewest real-time connections. This method ensures that the load is distributed more evenly, improving service stability and response time. Minimum time to first token: Requests are routed to the node with the lowest average time to first token. This method aims to minimize the wait time between receiving a request and starting its processing. This latency applies even when system resources are available immediately. Overall loads: Requests are routed to the node with the lowest overall pressure, considering factors such as the number of connections, time to first token, and custom metrics. It directs new requests to less busy nodes to avoid resource waste and overloading. SLO priority: Services are prioritized according to their assigned Service Level Objective (SLO) (0–3, where 0 is the highest). This ensures lower latency for higher-priority workloads. For details, see Intelligent Routing Policy.
  Custom Metric Collection	If this function is enabled, you need to input the metric collection port details. Custom metrics will send data to AOM's Prometheus instances for viewing on the AOM console. To query custom metrics, see Viewing Performance Metrics of a Real-Time Service on AOM and Viewing Performance Metrics of a Real-Time Service on ModelArts. Constraints ModelArts calls the HTTP API provided in the custom metric configuration every 10 seconds to obtain metric data. The metric data text returned by the HTTP API provided in the custom metric configuration cannot exceed 32 KB. Data Format of Custom Metrics The format of custom metrics data must comply with the open metrics specifications. That is, the format of each metric must be: <metric_name>{<tag_name>=<tag_value>,...} <sample_value> [timestamp_in_millisecond] The following shows an example (the comment starts with #, which is optional): # HELP http_requests_total The total number of HTTP requests. # TYPE http_requests_total gauge html_http_requests_total{method="post",code="200"} 1656 1686660980680 html_http_requests_total{method="post",code="400"} 2 1686660980681
  System Log Reporting	Displays only for NPU dedicated resource pools. When enabled, system logs are mounted to a fixed parallel file system for O&M engineers to analyze. System logs are stored for 30 days and will be automatically deleted after that. The mount path of system logs cannot be changed.

Confirming Settings

On the Deploy Real-Time Service > Confirmation page, confirm the configuration information and click Confirm Deployment.

Deploying a service generally requires a period of time, which may be several minutes or tens of minutes depending on the amount of your data and resources.

You can go to the real-time service list to check if the deployment is complete. Once the service status changes from Deploying to Running, the service is deployed.

After a real-time service is deployed, it is started immediately.

Follow-Up Operations

• For details about how to test a real-time service, compare the model effect in real time, and debug a real-time service using CloudShell, see Testing a Real-Time Service.
• For details about how to view real-time service details, see Viewing Details About a Real-Time Service.
• For details about how to modify, stop, and delete a real-time service, see Managing the Lifecycle of a Real-Time Service Deployment.