Real-Time Service Health Check
Overview
ModelArts real-time service health check is a core capability that ensures the stable running and high availability of AI inference services. It periodically checks the startup, readiness, and running status of service instances, automatically identifies abnormal instances, and triggers repair or isolation operations. This prevents issues such as invalid traffic forwarding, service freezing, and process breakdown. It is applicable to the following core scenarios:
- Slow model loading: When deploying foundation models (such as 100-B LLMs or multimodal models), the initial loading process takes a long time. Health checks prevent the "false active" dilemma, where the service status shows as "Running" but cannot actually respond to incoming requests.
- Precise traffic distribution: This ensures that inference requests are forwarded exclusively to instances that are fully ready and capable of processing tasks normally, successfully eliminating request failures and timeouts.
- Automated fault self-healing: When an anomaly occurs, such as an instance process crashing, resource exhaustion, or missing dependencies, the platform automatically restarts the instance without manual intervention, thereby boosting service availability.
- Rolling upgrade assurance: During a service upgrade, health checks are used to verify the availability of new instances, enabling zero-downtime, disruption-free upgrades.
- Custom image deployment: For inference services developed based on custom container images, health checks verify whether the internal service processes within the image are running as expected.
Constraints
- API/Script constraints: Before configuring a health check, you must deploy the health check API (HTTP/HTTPS) or probe script inside the image in advance; otherwise, it will directly cause the model deployment to fail.
- Stop delay: For services with health checks configured, the stop command is delayed by 3 minutes to prevent request interruption.
- Resource pool constraints: Health checks can be configured for both public and dedicated resource pools.
Prerequisites
The health check API (default path: /health) or detection script (for example, /home/ma-user/health.sh) has been deployed in the image.
Health Check Configuration
Log in to the ModelArts console and choose Model Inference > Real-Time Inference. When deploying a real-time service, select Health Check in the Unit Settings. For details, see Deploying a Real-Time Inference Service Using a Single Node. The following describes health check parameters.
Health checks support three types of probes. Choose one or more based on your needs. Probes can be set up alone or together. If you configure a startup probe, other probes will not work.
- Startup probe: This probe checks if the instance has started. If a startup probe is configured, the liveness or readiness probe is not executed until the startup probe is successful, allowing sufficient time for the application to complete initialization. If the startup probe fails, the instance is restarted. If startup probe is not configured, the service status changes to success immediately after the service is scheduled. The service may be in the Running state and the prediction cannot be performed because the model is being loaded. It is recommended for foundation models and services that take a long time to start.
- Readiness probe: This probe verifies whether the instance is ready to handle traffic. If the readiness probe fails (meaning the instance is not ready), the instance is taken out of the service load balancing pool. Traffic will not be routed to the instance until the probe succeeds. It is recommended for scenarios sensitive to traffic distribution.
- Liveness probe: This probe monitors the application health status. If the liveness probe fails (indicating the application is unhealthy), the instance is automatically restarted. It is recommended for high availability and fault self-healing scenarios.
| Parameter | Description | Example |
|---|---|---|
| Check Method | Select the health check method. Select either of the following options, which match the health check capability of the custom image:
| HTTP |
| URL | This parameter appears when you select HTTP. It sets the path for the health check to send an HTTP GET request. Make sure your application listens on this path and responds with a 200-399 status code to indicate success. The path must match the pattern ^[/][a-zA-Z0-9-_/:]{0,1024}. | /health |
| Command | This parameter appears when you choose Command. Enter the command to check the container's health status. Before using this method, you must package the required programs and tools in the container image. The cluster executes commands directly in the container. Host file systems and other containers' file systems are inaccessible. If the dependent programs or tools (such as curl, nc, or custom scripts) are not included in the image, error message "Command not found" will be displayed. If a shell script is executed, you must specify a script interpreter. The cluster does not provide an interactive terminal, so you cannot execute scripts directly. You must use the interpreter to invoke the script. For example, if the script is located in /home/ma-user/health.sh, specify sh /home/ma-user/health.sh when running the command. | / |
| Interval (s) | Interval for performing a health check, in seconds. This value sets how often the probe does health checks. Shorter intervals find problems faster but use more resources. Longer intervals save resources but might delay problem detection. The value ranges from 1 to 2147483647. | 10 |
| Delay (s) | Number of seconds to wait after the container is started before starting the probe. The value ranges from 1 to 2147483647. If a startup probe is set, the liveness and readiness probes start their delay only after the startup probe succeeds. If the health check interval is longer than the delay, the delay is skipped. | 60 |
| Timeout (s) | Number of seconds to wait after the check request times out. If no response is received within this time, the check fails. For applications that process slowly or have poor network conditions, increase this value to prevent incorrect failure judgments. The value ranges from 1 to 2147483647. | 30 |
| Failure Threshold | Number of consecutive health check failures that mark a container as unhealthy. This mechanism stops brief network issues or temporary application overload from causing false alarms. For example, a default setting of 12 means a restart or traffic removal happens only after 12 straight health check failures. The value ranges from 1 to 2147483647. If the service has too many failed health checks in a row during startup, it goes into an abnormal state. If the service has too many failed health checks in a row during running, it goes into an alarm state. | 1800 |
| Protocol | Network protocol used for health checks. The value must be the same as that of the real-time inference service.
| HTTP |
| Parameter | Description | Example |
|---|---|---|
| Check Method | Select the health check method. Select either of the following options, which match the health check capability of the custom image:
| HTTP |
| URL | This parameter appears when you select HTTP. It sets the path for the health check to send an HTTP GET request. Make sure your application listens on this path and responds with a 200-399 status code to indicate success. The path must match the pattern ^[/][a-zA-Z0-9-_/:]{0,1024}. | /health |
| Command | This parameter appears when you choose Command. Enter the command to check the container's health status. Before using this method, you must package the required programs and tools in the container image. The cluster executes commands directly in the container. Host file systems and other containers' file systems are inaccessible. If the dependent programs or tools (such as curl, nc, or custom scripts) are not included in the image, error message "Command not found" will be displayed. If a shell script is executed, you must specify a script interpreter. The cluster does not provide an interactive terminal, so you cannot execute scripts directly. You must use the interpreter to invoke the script. For example, if the script is located in /home/ma-user/health.sh, specify sh /home/ma-user/health.sh when running the command. | / |
| Interval (s) | Interval for performing a health check, in seconds. This value sets how often the probe does health checks. Shorter intervals find problems faster but use more resources. Longer intervals save resources but might delay problem detection. The value ranges from 1 to 2147483647. | 10 |
| Delay (s) | Number of seconds to wait after the container is started before starting the probe. The value ranges from 1 to 2147483647. If a startup probe is set, the readiness probe starts its delay only after the startup probe succeeds. If the health check interval is longer than the delay, the delay is skipped. | 60 |
| Timeout (s) | Number of seconds to wait after the check request times out. If no response is received within this time, the check fails. For applications that process slowly or have poor network conditions, increase this value to prevent incorrect failure judgments. The value ranges from 1 to 2147483647. | 30 |
| Failure Threshold | Number of consecutive health check failures that mark a container as unhealthy. This mechanism stops brief network issues or temporary application overload from causing false alarms. For example, a default setting of 12 means a restart or traffic removal happens only after 12 straight health check failures. The value ranges from 1 to 2147483647. If the service has too many failed health checks in a row during startup, it goes into an abnormal state. If the service has too many failed health checks in a row during running, it goes into an alarm state. | 3 |
| Protocol | Network protocol used for health checks. The value must be the same as that of the real-time inference service.
| HTTP |
| Parameter | Description | Example |
|---|---|---|
| Check Method | Select the health check method. Select either of the following options, which match the health check capability of the custom image:
| HTTP |
| URL | This parameter appears when you select HTTP. It sets the path for the health check to send an HTTP GET request. Make sure your application listens on this path and responds with a 200-399 status code to indicate success. The path must match the pattern ^[/][a-zA-Z0-9-_/:]{0,1024}. | /health |
| Command | This parameter appears when you choose Command. Enter the command to check the container's health status. Before using this method, you must package the required programs and tools in the container image. The cluster executes commands directly in the container. Host file systems and other containers' file systems are inaccessible. If the dependent programs or tools (such as curl, nc, or custom scripts) are not included in the image, error message "Command not found" will be displayed. If a shell script is executed, you must specify a script interpreter. The cluster does not provide an interactive terminal, so you cannot execute scripts directly. You must use the interpreter to invoke the script. For example, if the script is located in /home/ma-user/health.sh, specify sh /home/ma-user/health.sh when running the command. | / |
| Interval (s) | Interval for performing a health check, in seconds. This value sets how often the probe does health checks. Shorter intervals find problems faster but use more resources. Longer intervals save resources but might delay problem detection. Set the liveness probe check interval long enough to avoid premature container restarts. The value ranges from 1 to 2147483647. | 10 |
| Delay (s) | Number of seconds to wait after the container is started before starting the probe. The value ranges from 1 to 2147483647. If a startup probe is set, the liveness probe starts its delay only after the startup probe succeeds. If the health check interval is longer than the delay, the delay is skipped. | 60 |
| Timeout (s) | Number of seconds to wait after the check request times out. If no response is received within this time, the check fails. For applications that process slowly or have poor network conditions, increase this value to prevent incorrect failure judgments. The value ranges from 1 to 2147483647. | 30 |
| Failure Threshold | Number of consecutive health check failures that mark a container as unhealthy. This mechanism stops brief network issues or temporary application overload from causing false alarms. For example, a default setting of 12 means a restart or traffic removal happens only after 12 straight health check failures. The value ranges from 1 to 2147483647. If the service has too many failed health checks in a row during startup, it goes into an abnormal state. If the service has too many failed health checks in a row during running, it goes into an alarm state. | 6 |
| Protocol | Network protocol used for health checks. The value must be the same as that of the real-time inference service.
| HTTP |
After deploying the real-time inference service, check the health logs and exception records in the monitoring tab of the service details page.
What is your overall rating for this page?
Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot
