Auto Rebuild upon a Real-Time Service Fault

Overview

Automatic rebuild is a core capability that guarantees high availability, self-healing, and stable operation for real-time inference services. It is applicable to the following scenarios:

• Accidental container or process crashes, requiring fast recovery

  When an inference instance (pod) exits unexpectedly due to process anomalies, out of memory (OOM) errors, missing dependencies, or code bugs, the platform automatically reconstructs the pod and restarts the inference service without manual intervention, minimizing service downtime.

• Smooth application of deployment configuration changes

  After modifying configurations such as images, environment variables, resource specifications, or mount paths, automatic reconstruction can automatically restart and rebuild instances based on predefined policies, allowing new configurations to take effect quickly while ensuring service continuity.

• Fault tolerance during rolling upgrades and version iterations

  When upgrading services, switching model versions, or changing deployment configurations, if some instances fail to upgrade, automatic reconstruction can automatically restore the abnormal instances to prevent the overall service from being affected.

• Fast migration and reconstruction after underlying node failures

  In the event of dedicated resource pool node system crashes, hardware anomalies, or NPU failures, automatic rebuild can schedule pods to healthy nodes and rebuild the instances, guaranteeing the continuous availability of inference services.

• Long-term stable operation and unattended production environments

  For 24/7 real-time services such as LLM real-time inference, intelligent customer service, and real-time prediction, the system relies on automatic rebuild capabilities to achieve fault self-healing, lower O&M costs, and reduce manual intervention.

When automatic rebuild is enabled, if a pod restarts due to deployment configuration changes or failures, the platform will automatically rebuild it using the selected policy. If disabled, the platform will not intervene.

Rebuild policies:

• Deployment replica rebuild: When a pod restarts, the entire deployment replica is rebuilt.
• Unit rebuild: When a pod restarts, the entire unit is rebuilt.
• Unit replica rebuild: When a pod restarts, the entire unit replica is rebuilt.
• Pod rebuild: When a pod restarts, the entire pod is rebuilt.

Constraints

• Resource pool limitations: Both dedicated resource pools and public resource pools support automatic rebuild.
• Dependence on health checks: If health checks (readiness / liveness probes) are not configured, the platform cannot precisely identify anomalies, which may lead to invalid reconstructions or reconstruction delays.
• Difference between automatic rebuild and automatic restart: Automatic rebuild focuses on instance rebuild after configuration changes or software anomalies; automatic restart focuses on scheduling and restarting after hardware, NPU, or switch failures. The two can be used in combination.
• Plugin dependency: The ClusterX plugin must be installed in the A5 resource pool before deploying inference. This allows the system to detect node subhealth faults and trigger automatic rebuilding. For details about the ClusterX plugin, see ClusterX under Managing Dedicated Resource Pool Plug-ins.
• Unit replica rebuilding: The resource pool KubeInfer plugin 7.5.2 or later can be used.
• Under unit rebuild, unit replica rebuild, or pod rebuild policies: The system must wait for the container to exit before rebuild. Due to graceful shutdown mechanisms or batch execution, when the number of resource instances for a unit replica is greater than 1, the total time required for containers to exit may reach approximately twice the graceful shutdown duration.
• When using version 7.5.2 or higher of the resource pool KubeInfer plugin and when the number of single-replica instances is greater than 1, if the rebuild policy is set to pod rebuild, an abnormal restart or deletion of the leader pod will trigger the rebuild of all pods under the entire unit replica. If automatic rebuild is not enabled, deleting the leader pod will also trigger the rebuild of all pods under the entire unit replica. As shown in Figure 1 (where the unit name is role-0, the number of unit replicas is 2, and the number of single-replica instances is 3), role-0-0, role-0-0-1, and role-0-0-2 form unit replica 1, while role-0-1, role-0-1-1, and role-0-1-2 form unit replica 2. role-0-0 and role-0-1 serve as the leader pods for the two unit replicas, respectively.
  Figure 1 Pod details of a deployment replica
  

Prerequisites

The real-time inference service is in the Running state.

Enabling Auto Rebuild

Method 1:

When adding a deployment, choose Unit Settings > More Settings, select Automatic Rebuild, and select a rebuild policy.

Method 2:

If automatic rebuild is not enabled during deployment, you can update the deployment service and change the configurations. The procedure is as follows:

1. Log in to the ModelArts console and choose Model Inference > Real-Time Inference.
2. Click the name of the target real-time service to enter its details page. On the details page, switch to the Deploy tab, select the target deployment card, and click Upgrade.
3. Under Unit Settings > More Settings, select Automatic Rebuild, and select a rebuild policy (one out of four options).
   • Pod rebuild: Only the abnormal pod is rebuilt.
   • Unit rebuild: The entire inference unit is rebuilt.
   • Unit replica rebuild: When a pod restarts, the entire unit replica is rebuilt.
   • Deployment replica rebuild: The entire deployment replica is rebuilt.
4. Save the configuration to enable automatic rebuild.

   After this function is enabled, if a pod exits due to an exception or restarts due to configuration changes, the platform will automatically rebuild the instance based on the selected policy.

Related Operations

You are advised to configure health checks at the same time to improve anomaly detection accuracy and reduce invalid rebuild. For details, see Real-Time Service Health Check.

FAQs

Residual processes caused by NPU faults lead to failure of the automatic rebuild mechanism.

Symptom

When running foundation models or inference services on NPUs, if the heartbeat of the underlying daemon process becomes abnormal and triggers an NPU hardware fault (such as the 0x8C204E00 fault on Ascend A5), communication anomalies between the host side and the device side often trigger the following chain reaction:

• After the application container exits normally, residual NPU-related processes on the host side fail to completely terminate.
• This leaves NPU memory unable to be released in a timely manner, which in turn causes the upper-layer automatic rebuild mechanism to fail.

Typically, these residual processes recover after a driver or kernel communication timeout of approximately 5 minutes.

Solution: Configure a graceful shutdown script for the service. Within the script, proactively terminate processes and wait for the NPU memory release to complete:

Proactively terminate related processes in the shutdown script:

# Terminate related processes
pkill -9 route-server;pkill -9 vllm;pkill -9 VLLM;pkill -9 python;pkill -9 python3

Add polling detection logic to the shutdown script: Ensure that vllm-related residual processes are completely reclaimed and NPU memory release is finished before allowing the container to fully exit or triggering the rebuild workflow. Reference example for the detection command:

# Check whether NPU memory is released
 
npu-smi info | grep NA | grep -oP '\d+(?=\s*/\s*\d+\s*\|)' | awk '$1>5000{print "Card " NR-1 " unreleased HBM: " $1 "MB"}'

For more information on graceful shutdown, see Graceful Shutdown of Real-Time Services.