Graceful Shutdown of Real-Time Services

Overview

Graceful shutdown for real-time inference services is used in scenarios such as service discontinuation, upgrades, scale-ins, and fault migrations. Its purpose is to smoothly complete in-flight requests, release resources, save states, and avoid forced interruptions, thereby guaranteeing service continuity and data integrity.

Typical scenario:

• Normal service discontinuation: During service iterations, version replacements, or resource reclamation, the system waits for current inference requests to complete before terminating instances, preventing client-side errors.
• Rolling upgrades: When updating models or container images, old instances are gracefully shut down step-by-step while new instances are pulled up, achieving a smooth upgrade with zero disruption.
• Instance scale-in: When reducing the number of replicas or releasing idle resources, priority is given to gracefully terminating idle instances without affecting requests currently being processed.
• Node and hardware fault migration: When NPU, CPU, or switch anomalies trigger a migration, graceful shutdown saves the temporary inference context, reducing task failures.
• LLM and long-task inference: For streaming or long-text inference tasks that take a long time to execute, graceful shutdown prevents forced terminations that result in incomplete outputs or broken sessions.

Constraints

• Health check linkage: For services already configured with health checks, enabling graceful shutdown introduces a forced 3-minute delay before the service stops after receiving a stop command.
• Resource pool limitations: Both dedicated resource pools and public resource pools are supported.
• Deployment modes: Both the basic mode and the multi-role separation mode are supported. For the multi-role separation mode, the configuration must be applied uniformly across all inference units.
• Command dependencies: Shutdown commands rely on basic system utilities inside the container (such as sh, kill, and trap). If these dependencies are missing, the execution will fail.

Graceful Shutdown Configuration

Log in to the ModelArts console and choose Model Inference > Real-Time Inference. When deploying a real-time service, select Graceful Shutdown in Unit Settings > More Settings, and set the shutdown time and command. For details, see Deploying a Real-Time Inference Service Using a Single Node.

Shutdown Timeout (s): 30s for common services; 60s to 300s for LLMs or long tasks.

Shutdown Command (executed inside the container): Choose and enter only one of the three examples below:

• Example 1 (terminating a process)

  kill -SIGTERM $(pgrep -f app.py)

  Application: Directly and gracefully terminates the primary service process.

  Principle: Sends a standard graceful termination signal, allowing the process to clean up and exit on its own.

  Scenarios: Python scripts, standalone Flask/FastAPI process services.

• Example 2 (graceful exit)

  trap 'exit 0' SIGTERM

  Application: Directly and gracefully terminates the primary service process.

  Principle: Sends a standard graceful termination signal, allowing the process to clean up and exit on its own.

  Scenarios: Shell resident services, background daemon processes.

• Example 3 (releasing connections/saving state)

  /bin/sh -c stop_service.sh

  Application: Complex shutdown logic (multi-step cleanup).

  Principle: Calls a custom shutdown script inside the container to execute a complete suite of actions, such as saving logs, disconnecting, flushing cache to disk, and closing child processes.

  Scenarios: Services with complex service workflows that require custom cleanup operations.

After the configuration is complete, stop or upgrade the service, and observe that in-flight requests finish normally without errors.