Updated on 2026-07-07 GMT+08:00

Auto Rebuild upon a Real-Time Service Fault

Overview

Automatic rebuild is a core capability that guarantees high availability, self-healing, and stable operation for real-time inference services. It is applicable to the following scenarios:

Accidental container or process crashes, requiring fast recovery
When an inference instance (pod) exits unexpectedly due to process anomalies, out of memory (OOM) errors, missing dependencies, or code bugs, the platform automatically reconstructs the pod and restarts the inference service without manual intervention, minimizing service downtime.
Smooth application of deployment configuration changes
After modifying configurations such as images, environment variables, resource specifications, or mount paths, automatic reconstruction can automatically restart and rebuild instances based on predefined policies, allowing new configurations to take effect quickly while ensuring service continuity.
Fault tolerance during rolling upgrades and version iterations
When upgrading services, switching model versions, or changing deployment configurations, if some instances fail to upgrade, automatic reconstruction can automatically restore the abnormal instances to prevent the overall service from being affected.
Fast migration and reconstruction after underlying node failures
In the event of dedicated resource pool node crashes, hardware anomalies, or NPU failures, automatic rebuild can schedule pods to healthy nodes and rebuild the instances, guaranteeing the continuous availability of inference services.
Long-term stable operation and unattended production environments
For 24/7 real-time services such as LLM real-time inference, intelligent customer service, and real-time prediction, the system relies on automatic rebuild capabilities to achieve fault self-healing, lower O&M costs, and reduce manual intervention.

When automatic rebuild is enabled, if a pod restarts due to deployment configuration changes or failures, the platform will automatically rebuild it using the selected policy. If disabled, the platform will not intervene.

Rebuild policies:

Deployment replica rebuild: When a pod restarts, the entire deployment replica is rebuilt.
Unit rebuild: When a pod restarts, the entire unit is rebuilt.
Unit replica rebuild: When a pod restarts, the entire unit replica is rebuilt.
Pod rebuild: When a pod restarts, the entire pod is rebuilt.

**Table 1** Suggestions on configuring automatic rebuild policies
Use Case	Recommended Policy	Description
Single-PU/Small-model inference, no communication dependencies between pods	Pod rebuild	Minimizes the scope of fault impact. Only the faulty pod is reconstructed, leaving the remaining pods unaffected.
Multi-PU inference (TP), strong communication dependencies within the same unit	Unit replica rebuild	A failure in one pod means the entire unit replica becomes unavailable. All pods within that replica must be reconstructed to ensure communication consistency.
Multi-unit inference (PP+TP), sequential dependencies between units	Unit rebuild	Multiple pods within a unit jointly handle the computing tasks for a specific stage. A failure in any single pod will prevent that stage from completing its computation. The entire unit must be reconstructed to restore computing capability and guarantee RankTable consistency across units.
Scenarios requiring strong global state consistency (e.g., distributed inference)	Deployment replica rebuild	A failure in any single pod impacts the entire global system. All pods across the entire deployment replica must be reconstructed to ensure the global RankTable and HCCL are renegotiated.

Constraints

Resource pool limitations: Both dedicated resource pools and public resource pools support automatic rebuild.
Dependence on health checks: If health checks (readiness / liveness probes) are not configured, the platform cannot precisely identify anomalies, which may lead to invalid reconstructions or reconstruction delays.
Difference between automatic rebuild and automatic restart: Automatic rebuild focuses on instance rebuild after configuration changes or software anomalies; automatic restart focuses on scheduling and restarting after hardware, NPU, or switch failures. The two can be used in combination.
Under unit rebuild, unit replica rebuild, or pod rebuild policies: The system must wait for the container to exit before rebuild. Due to graceful shutdown mechanisms or batch execution, when the number of resource instances for a unit replica is greater than 1, the total time required for containers to exit may reach approximately twice the graceful shutdown duration.

Prerequisites

The real-time inference service is in the Running state.

Enabling Auto Rebuild

Method 1:

When adding a deployment, choose Unit Settings > More Settings, select Automatic Rebuild, and select a rebuild policy.

Method 2:

If automatic rebuild is not enabled during deployment, you can update the deployment service and change the configurations. The procedure is as follows:

Log in to the ModelArts console and choose Model Inference > Real-Time Inference.
Click the name of the target real-time service to enter its details page. On the details page, switch to the Deploy tab, select the target deployment card, and click Upgrade.
Under Unit Settings > More Settings, select Automatic Rebuild, and select a rebuild policy (one out of four options).
- Pod rebuild: Only the abnormal pod is rebuilt.
- Unit rebuild: The entire inference unit is rebuilt.
- Unit replica rebuild: When a pod restarts, the entire unit replica is rebuilt.
- Deployment replica rebuild: The entire deployment replica is rebuilt.
Save the configuration to enable automatic rebuild.
After this function is enabled, if a pod exits due to an exception or restarts due to configuration changes, the platform will automatically rebuild the instance based on the selected policy.

Related Operations

You are advised to configure health checks at the same time to improve anomaly detection accuracy and reduce invalid rebuild. For details, see Real-Time Service Health Check.

Parent Topic: Reliability