Help Center/ ModelArts/ Model Inference/ Inference Deployment (New Version)/ Advanced Functions/ Traffic Policies
Updated on 2026-07-07 GMT+08:00
View PDF
Share

Traffic Policies

Overview

ModelArts real-time inference services provide multi-deployment traffic scheduling capabilities. For multiple deployment instances created under the same real-time inference service, an independent traffic weight value can be configured for each deployment. The platform automatically distributes and routes inference service call requests proportionally based on the relative weight of each deployment. This enables fine-grained O&M control, including model gray releases, A/B testing, failover traffic switching, and on-demand traffic allocation.

For real-time services with existing deployment configurations, you can adjust the traffic weights of each deployment under the current service by modifying the traffic weight settings. The specific rules are as follows:

  • Single deployment traffic weight: The traffic weight value for a single deployment must be an integer not exceeding 100.
  • Multi-deployment traffic weight allocation: Traffic is routed to different deployments proportionally.
  • All deployment traffic weights set to 0: The system automatically distributes traffic evenly across all normally running deployments to ensure service continuity.
  • Enable traffic mirroring: The system replicates a fixed percentage (10%) of live traffic to the corresponding deployment for shadow verification, without affecting user requests.
Table 1 Use cases

Use Case

Service Objective

Configuration Method

Value Delivered

Model gray release

Launches a new model version without interrupting services, minimizing failure risks through gradual rollout and validation.
  1. Deploy a stable old version (V1) and a new version (V2) under the same real-time service.
  2. Traffic weights: V1 = 90, V2 = 10.
  3. Monitor QPS, latency, accuracy, and error rates, and gradually adjust to V1 = 50, V2 = 50.
  4. If anomalies occur, switch V1 back to 100 for rapid rollback.

Zero-downtime release; controllable risk; completely seamless to users.

Online A/B testing

Compares the real-world performance of multiple model versions or strategies to back iterative decisions with data.
  1. Deploy V1 (baseline) and V2 (experimental group) under the same real-time service.
  2. Traffic weights: V1 = 50, V2 = 50.
  3. Utilize a unified entry to split traffic by weight while monitoring QPS, latency, accuracy, and error rates.

Leverages live production traffic for objective evaluation, eliminating offline evaluation bias.

Traffic mirroring (shadow testing)

Validates the compatibility, performance, and stability of a new version using real traffic without impacting users.
  1. Enable traffic mirroring for the new version deployment.
  2. The system automatically replicates 10% of production traffic to the new deployment.
  3. The new deployment generates no real-world responses and is used purely for observation.

Real-world validation with zero service risk; ideal for pre-launch verification of LLMs or critical services.

Multi-deployment load balancing & auto scaling

Absorbs peak traffic spikes while saving resources during off-peak hours to maximize utilization and stability.
  • Peak hours: Distribute weights equally across multiple deployments (e.g., 30:30:30) to handle the load together.
  • Off-peak hours: Retain one primary deployment at 100% weight, while scaling down or stopping others.
  • Setting all weights to 0: The system automatically balances traffic evenly to prevent single-point overload.

Dynamically adapts to varying workloads, eliminates resource waste, and enhances service availability.

Multi-model & multi-service traffic scheduling

Shares infrastructure resources under a single inference endpoint, allocating traffic based on priority or importance.

Core service model deployment: Weight = 70.

Secondary/Experimental model deployment: Weight = 30.

During resource constraints, lower non-core weights to safeguard core SLAs.

Promotes resource reuse, optimizes infrastructure costs, and establishes clear service priorities.

Fast fault isolation & lossless rollback

Mitigates losses immediately when a specific model version experiences anomalies, keeping the overall service safe.

Instantly set the weight of the anomalous deployment to 0.

Route 100% of the traffic back to the stable deployment version (Weight = 100).

Isolate the problematic deployment for troubleshooting and patching.

Rapid loss mitigation; near-zero service impact; high O&M efficiency.

Rapid iteration & validation in test environments

Enables parallel development and testing validation without affecting production traffic.

Production deployment: Weight = 100.

Test deployment: Weight unconfigured + traffic mirroring enabled.

Validate functionality using shadow traffic before initiating a gradual gray release.

Strong isolation between production and testing; secure and highly efficient iteration cycles.

Constraints

Mirrored traffic occurs only in synchronous calls, not in asynchronous ones.

Viewing/Modifying Traffic Weights

Log in to the ModelArts console and choose Model Inference > Real-Time Inference.

Method 1:

In the service list, click a service name to enter the service details page. In the Deploy tab of the service details page, you can view the actual traffic percentage of the service. The actual traffic percentage is calculated based on the traffic weight configuration and the deployment status.

Figure 1 Viewing the traffic percentage

In the traffic weight area, click Modify Configuration to modify the traffic percentage of the deployment.

Figure 2 Modify Traffic Weight

Table 2 Traffic weight policy parameters

Parameter

Description

Mirrored Traffic

Also called shadow traffic. It is a secure deployment verification function that routes real-time copies of live traffic to a new version, without affecting actual user requests. It provides you with a shadow test environment that is identical to the production environment.

Traffic mirroring requires that the service protocol be HTTP/HTTPS.

When mirrored traffic is enabled for a deployment, it will no longer receive live traffic. The traffic weight is not used to estimate the traffic percentage. Instead, the system copies 10% of the actual traffic to the deployment using a standard policy. This helps you safely test the model. Ensure that resource quotas are sufficient to handle the mirrored traffic load.

Mirrored traffic occurs in synchronous calls, not in asynchronous ones.

Traffic Weight

Configure traffic weights for deployments. Requests are routed proportionally. Each deployment's traffic weight must be an integer. Max. value: 100 If all weights are 0, traffic will be evenly split to avoid disruptions.

Mirror traffic is not included in weight calculations for actual traffic. Its percentage is fixed by default and cannot be modified.

Keep at least one deployment that receives live requests for each service.

Estimated Traffic Percentage

The estimated traffic percentage is the ratio of the traffic expected to be received by a single deployment instance group to the total traffic. The actual traffic percentage is calculated based on the traffic weight configuration and the deployment status.

Method 2:

In the service list, click a service name to enter the service details page. In the Service tab of the service details page, you can view the traffic weight of the service. Click Modify Traffic Allocation to modify the traffic weight of the deployment under the current service.
Figure 3 Viewing traffic weights

Method 3:

In the Operation column of the service list page, choose More > Set Traffic Policy to modify the traffic weight.

Figure 4 Set Traffic Policy

Click OK.

Parent Topic: Advanced Functions