Help Center/ ModelArts/ Model Inference/ Inference Deployment (New Version)/ Advanced Functions/ Traffic Policies
Updated on 2026-11-11 GMT+08:00
View PDF
Share

Traffic Policies

Overview

ModelArts real-time inference services provide traffic scheduling capabilities for multiple deployments. For multiple deployments created under the same real-time inference service, an independent traffic weight can be manually or automatically assigned to each deployment. The platform then automatically distributes inference service requests based on the relative weight of each deployment, enabling refined operations management for model gray releases, A/B testing, failover, and on-demand traffic allocation.

ModelArts provides two traffic allocation modes:

  • Manual allocation: You manually configure a fixed traffic weight value for each deployment, and the platform performs traffic splitting based on the relative proportion of each deployment's weight. This is suitable for scenarios requiring precise control over traffic ratios, such as gray releases and A/B testing.
  • Automatic allocation: The platform automatically detects status changes of deployment replicas (such as online, offline, or abnormal) and dynamically calculates traffic allocation proportions based on the actual number of running deployment replicas, eliminating the need for manual weight modifications. This is suitable for scenarios where capabilities across deployments are relatively similar and traffic is expected to automatically adjust alongside the number of deployment replicas.

For real-time services with existing deployment configurations, you can adjust the traffic weights of each deployment under the current service by modifying the traffic weight settings. The specific rules are as follows:

  • Single deployment traffic weight: The traffic weight value for a single deployment must be an integer not exceeding 100.
  • Multi-deployment traffic weight allocation: Traffic is routed to different deployments proportionally.
  • All deployment traffic weights set to 0: The system automatically distributes traffic evenly across all normally running deployments to ensure service continuity.
  • Enable traffic mirroring: The system replicates a fixed percentage (10%) of live traffic to the corresponding deployment for shadow verification, without affecting user requests.
  • Enable automatic allocation: Traffic proportions are automatically calculated based on the number of running deployment replicas, without requiring manually set weights. Automatic allocation and manual allocation are mutually exclusive.
Table 1 Use cases

Use Case

Service Objective

Configuration Method

Value Delivered

Model gray release

Launches a new model version without interrupting services, minimizing failure risks through gradual rollout and validation.
  1. Deploy a stable old version (V1) and a new version (V2) under the same real-time service.
  2. Traffic weights: V1 = 90, V2 = 10.
  3. Monitor QPS, latency, accuracy, and error rates, and gradually adjust to V1 = 50, V2 = 50.
  4. If anomalies occur, switch V1 back to 100 for rapid rollback.

Zero-downtime release; controllable risk; completely seamless to users.

Online A/B testing

Compares the real-world performance of multiple model versions or strategies to back iterative decisions with data.
  1. Deploy V1 (baseline) and V2 (experimental group) under the same real-time service.
  2. Traffic weights: V1 = 50, V2 = 50.
  3. Utilize a unified entry to split traffic by weight while monitoring QPS, latency, accuracy, and error rates.

Leverages live production traffic for objective evaluation, eliminating offline evaluation bias.

Traffic mirroring (shadow testing)

Validates the compatibility, performance, and stability of a new version using real traffic without impacting users.
  1. Enable traffic mirroring for the new version deployment.
  2. The system automatically replicates 10% of production traffic to the new deployment.
  3. The new deployment generates no real-world responses and is used purely for observation.

Real-world validation with zero service risk; ideal for pre-launch verification of LLMs or critical services.

Multi-deployment load balancing & auto scaling

Absorbs peak traffic spikes while saving resources during off-peak hours to maximize utilization and stability.
  • Peak hours: Distribute weights equally across multiple deployments (e.g., 30:30:30) to handle the load together.
  • Off-peak hours: Retain one primary deployment at 100% weight, while scaling down or stopping others.
  • Setting all weights to 0: The system automatically balances traffic evenly to prevent single-point overload.

Dynamically adapts to varying workloads, eliminates resource waste, and enhances service availability.

Multi-model & multi-service traffic scheduling

Shares infrastructure resources under a single inference endpoint, allocating traffic based on priority or importance.

Core service model deployment: Weight = 70.

Secondary/Experimental model deployment: Weight = 30.

During resource constraints, lower non-core weights to safeguard core SLAs.

Promotes resource reuse, optimizes infrastructure costs, and establishes clear service priorities.

Fast fault isolation & lossless rollback

Mitigates losses immediately when a specific model version experiences anomalies, keeping the overall service safe.

Instantly set the weight of the anomalous deployment to 0.

Route 100% of the traffic back to the stable deployment version (Weight = 100).

Isolate the problematic deployment for troubleshooting and patching.

Rapid loss mitigation; near-zero service impact; high O&M efficiency.

Rapid iteration & validation in test environments

Enables parallel development and testing validation without affecting production traffic.

Production deployment: Weight = 100.

Test deployment: Weight unconfigured + traffic mirroring enabled.

Validate functionality using shadow traffic before initiating a gradual gray release.

Strong isolation between production and testing; secure and highly efficient iteration cycles.

Automatic allocation: automatic traffic adjustment upon deployment replica exceptions.

Traffic proportions automatically adjust based on the number of deployment replicas when a deployment replica becomes abnormal or during scaling, preventing overloaded abnormal deployments.
  1. When multiple deployments exist with consistent capabilities across each (for example: deployment A has two replicas and deployment B has two replicas):
  2. Automatic allocation is enabled, and the system automatically allocates traffic proportionally based on the number of running deployment replicas.
  3. If one deployment replica of deployment A goes offline due to an abnormality, the traffic ratio automatically adjusts from 50:50 to 33:67 (based on the remaining deployment replicas).
  4. Manual weight modification is unnecessary, and traffic automatically recovers once the deployment replica recovers.

Self-healing, unattended operation, and stable request latency.

Constraints

Mirrored traffic occurs only in synchronous calls, not in asynchronous ones.

Constraints on automatic allocation:

  • Prerequisites: There are little to no capability differences among the various deployments of the service, meaning each deployment provides the same inference capabilities and can be substituted for one another.
  • Mutual exclusivity: Automatic allocation and manual allocation are mutually exclusive. Once automatic allocation is enabled, the system ignores the static weight values set by you and dynamically allocates traffic based on the proportion of deployment replicas. Once automatic allocation is disabled, the system resumes using fixed weight values for distribution.
  • Mirror traffic groups excluded from automatic allocation calculations: Deployments with mirror traffic enabled are not included in the running deployment replica count statistics and are instead handled by an independent mirror traffic policy.
  • Fallback policy: When automatic allocation cannot obtain real-time deployment replica counts due to exceptions (such as Redis being unavailable), the system degrades to using static weight values for traffic splitting to ensure workload continuity.
  • New deployments: After automatic allocation is enabled, any newly added deployments have mirror traffic enabled by default and do not participate in traffic splitting.
  • Asynchronous inference: Asynchronous inference does not support automatic traffic allocation.

Principles of Automatic Traffic Allocation

In multi-deployment real-time inference services, when using manual traffic allocation, if some deployment replicas in a specific deployment become abnormal or go offline while traffic continues to be distributed according to the original weight proportions, it will cause request congestion (pending) in that deployment, increase request latency, and impact overall service availability.

For example, a real-time inference service contains two deployments, each with two deployment replicas, and a fixed weight of 50:50. If a replica in deployment A fails but the weight stays at 50:50, one replica handles 50% of the traffic while two replicas in deployment B handle the other 50%. This causes deployment A to become overloaded.

Automatic allocation solves the above problem by detecting changes in deployment replica status and dynamically adjusting the traffic allocation proportion among deployments. Its core mechanisms are as follows:

  • Traffic proportion calculation: After automatic allocation is enabled, the system queries every deployment capable of receiving traffic in real time, counts the number of deployment replicas with a status of RUNNING, and replaces static weights with the number of running deployment replicas as the basis for traffic splitting. Traffic proportion calculation formula:

    Traffic proportion = Number of running deployment replicas for the deployment/Sum of running deployment replicas across all available deployments × 100%

  • Configuration delivery and refresh: When the number of deployment replicas changes (such as during scaling or abnormal restarts of deployment replicas), the system automatically triggers a routing configuration refresh, re-fetches the latest number of running deployment replicas, and delivers it to the routing components. This ensures traffic proportions adjust automatically as deployment replicas go online or offline, without requiring manual weight modifications.
  • Compatibility: Automatic allocation is disabled by default. Services that are not enabled continue to use the original fixed-weight allocation method without being affected. If you switch back to manual traffic allocation after enabling automatic allocation, the traffic statistics will display the fixed splitting values set by you.

Viewing/Modifying Traffic Weights

Log in to the ModelArts console and choose Model Inference > Real-Time Inference.

Method 1:

In the service list, click a service name to enter the service details page. In the Deploy tab of the service details page, you can view the actual traffic percentage of the service. The actual traffic percentage is calculated based on the traffic weight configuration and the deployment status.

Figure 1 Viewing the traffic percentage

In the traffic weight area, click Modify Configuration to modify the traffic percentage of the deployment.

Figure 2 Modify Traffic Weight

Table 2 Traffic weight policy parameters (manual allocation)

Traffic Assignment Mode

Parameter

Description

Manual

Traffic Weight

Configure traffic weights for deployments. Requests are routed proportionally. Each deployment's traffic weight must be an integer. Max. value: 100 If all weights are 0, traffic will be evenly split to avoid disruptions.

Mirror traffic is not included in weight calculations for actual traffic. Its percentage is fixed by default and cannot be modified.

Keep at least one deployment that receives live requests for each service.

Manual

Mirrored Traffic

Also called shadow traffic. It is a secure deployment verification function that routes real-time copies of live traffic to a new version, without affecting actual user requests. It provides you with a shadow test environment that is identical to the production environment.

Traffic mirroring requires that the service protocol be HTTP/HTTPS.

When mirrored traffic is enabled for a deployment, it will no longer receive live traffic. The traffic weight is not used to estimate the traffic percentage. Instead, the system copies 10% of the actual traffic to the deployment using a standard policy. This helps you safely test the model. Ensure that resource quotas are sufficient to handle the mirrored traffic load.

Mirrored traffic occurs in synchronous calls, not in asynchronous ones.

Manual

Estimated Traffic Percentage

The estimated traffic percentage is the ratio of the traffic expected to be received by a single deployment to the total traffic. The actual traffic percentage is calculated based on the traffic weight configuration and the deployment status.
Table 3 Traffic weight policy parameters (automatic allocation)

Traffic Assignment Mode

Parameter

Description

Automatic

Assignment Method > By deployment replicas

With automatic allocation, traffic is dynamically routed to deployments based on how many replicas are available.

The estimated traffic percentage includes all deployments that have mirrored traffic disabled.

Automatic

Mirrored Traffic

Also called shadow traffic. It is a secure deployment verification function that routes real-time copies of live traffic to a new version, without affecting actual user requests. It provides you with a shadow test environment that is identical to the production environment.

Traffic mirroring requires that the service protocol be HTTP/HTTPS.

When mirrored traffic is enabled for a deployment, it will no longer receive live traffic. The traffic weight is not used to estimate the traffic percentage. Instead, the system copies 10% of the actual traffic to the deployment using a standard policy. This helps you safely test the model. Ensure that resource quotas are sufficient to handle the mirrored traffic load.

Mirrored traffic occurs in synchronous calls, not in asynchronous ones. Mirrored traffic is enabled by default for new deployments added in synchronous calls.

Automatic

Estimated Traffic Percentage

The estimated traffic percentage is the ratio of the traffic expected to be received by a single deployment to the total traffic. The system automatically allocates the actual traffic percentage based on how many deployment replicas are running.

Method 2:

In the service list, click a service name to enter the service details page. In the Service tab of the service details page, you can view the traffic weight of the service. Click Modify Traffic Allocation to modify the traffic weight allocation mode of the deployment under the current service.
Figure 3 Viewing traffic weights

Method 3:

In the Operation column on the right of the service list page, choose More > Set Traffic Policy to modify the traffic weight allocation mode.

Figure 4 Set Traffic Policy

Click OK.

Parent Topic: Advanced Functions