Help Center/ ModelArts/ Model Inference/ Inference Deployment (New Version)/ Logging and Monitoring/ Viewing Events of a Real-Time Service
Updated on 2026-09-08 GMT+08:00
View PDF
Share

Viewing Events of a Real-Time Service

During the whole lifecycle of a service, every key event is automatically recorded. You can view the events on the details page of the service at any time.

This helps you better understand the service deployment and running process and accurately locate faults when a task exception occurs.

The supported event types include:

  • Service events: Record the operational status and operations at the service level.
  • Pod events: Record the lifecycle events and exception conditions of the underlying containers.

Procedure

  1. Log in to the ModelArts console. In the navigation pane, choose Model Inference > Real-Time Inference. In the service list, click the target service name to go to the service details page.
  2. View events in the Events tab.

    Events are classified into service events and Pod Events, which can be viewed by switching tabs.

    Service events record the status and operations at the service layer. Pod events record the lifecycle and exceptions of underlying containers.

    Figure 1 Viewing events

    During service deployment and running, key events can both be manually and automatically refreshed.

Service Events

Service events record key activities like starting, stopping, updating, or recovering services. These events help you monitor performance, understand past actions, solve issues, and improve settings. For details about common exception and warning events, see Table 1 and Table 2. Normal events are not described here.

Table 1 Exceptional events

Event Type

Event Message (xxx and %s represent placeholders)

Solution

Exception

Service deployment failed, %s.

Locate and rectify the fault based on the error information.

Exception

Service deployment timed out. Error: The deployment duration of the service exceeds %s minutes. The deployment failed.

Try again later or contact technical support.

Exception

Service deletion failed. Current status: %s, %s.

Locate and rectify the fault based on the error information.

Exception

Batch service deletion failed, %s.

Locate and rectify the fault based on the error information.

Exception

Service deletion timed out. Error message: Service deletion time exceeded %s minute. The deletion failed.

Try again later or contact technical support.

Exception

Service asynchronous task cancellation failed, %s.

Locate and rectify the fault based on the error information.

Exception

Creation of deployment [%s] failed, %s.

Try again later or contact technical support.

Exception

Update of deployment [%s] failed, %s.

Try again later or contact technical support.

Exception

The Volcano plugin for resource pool [%s] is unavailable.

Try again later or contact technical support.

Exception

Deletion of deployment [%s] failed, %s.

Locate and rectify the fault based on the error information.

Exception

Stopping of deployment [%s] failed, %s.

Locate and rectify the fault based on the error information.

Exception

Deletion of secret [%s] failed, %s.

Locate and rectify the fault based on the error information.

Exception

Verification of secret [%s] failed, %s.

Locate and rectify the fault based on the error information.

Exception

Service configuration delivery failed, %s.

Locate and rectify the fault based on the error information.

Exception

Binding API key: %s failed, %s.

Locate and rectify the fault based on the error information.

Exception

Unbinding API key: %s failed, %s.

Locate and rectify the fault based on the error information.

Exception

Service startup failed, %s; status transitioned from %s to %s.

Service startup failures can happen for various reasons. For details about how to locate and rectify the fault, see Failed to Start a Service.

Exception

Service stop failed, %s.

Locate and rectify the fault based on the error information.

Exception

Service interruption failed. Current service status: %s, %s.

Locate and rectify the fault based on the error information.

Exception

Unable to release quota.

Try again later or contact technical support.

Exception

Interruption of deployment [%s] failed, %s; status transitioned from %s to %s.

Locate and rectify the fault based on the error information.

Exception

Interruption of deployment [%s] failed. Current deployment status: [%s], %s.

Locate and rectify the fault based on the error information.

Exception

Stopping of deployment [%s] failed, %s; status transitioned from %s to %s.

Locate and rectify the fault based on the error information.

Exception

Deletion of deployment [%s] failed, %s; status transitioned from %s to %s.

Locate and rectify the fault based on the error information.

Exception

Service update failed; the service will roll back, %s.

Locate and rectify the fault based on the error information.

Exception

Service update timed out. Error message: Service deployment time exceeded %s minutes; the service is rolled back.

Locate and rectify the fault based on the error information.

Exception

Service upgrade configuration failed, %s.

Locate and rectify the fault based on the error information.

Exception

Update of deployment [%s] failed; the deployment will roll back, %s.

Locate and rectify the fault based on the error information.

Exception

The service fails to be rolled back. Current service status: %s, %s.

Locate and rectify the fault based on the error information.

Exception

Service [%s] rollback timed out. Error message: %s.

Locate and rectify the fault based on the error information.

Exception

Upgrade rollback of deployment [%s] failed. Current status: [%s].

Locate and rectify the fault based on the error information.

Exception

Upgrade rollback of deployment [%s] failed; status transitioned from %s to %s.

Locate and rectify the fault based on the error information.

Exception

Rollback of deployment [%s] timed out; version: [%s].

Locate and rectify the fault based on the error information.

Exception

Service status transitioned from %s to %s; anomalous deployment [%s]; error message: %s.

Locate and resolve the issue based on the error message. For more solutions, see Abnormal Real-Time Service Deployment Instances.

Exception

Status of deployment [%s] transitioned from %s to %s; error message: %s.

Locate and resolve the issue based on the error message. For more solutions, see Abnormal Real-Time Service Deployment Instances.

Exception

Service status transitioned from %s to %s.

Locate and resolve the issue based on the error message. For more solutions, see Abnormal Real-Time Service Deployment Instances.

Exception

Service detected a chip fault in resource pool [%s]: %s.

Locate and resolve the issue based on the error message. If it cannot be resolved, contact technical support.

Exception

Service detected a switch fault in resource pool [%s]: %s.

Locate and resolve the issue based on the error message. If it cannot be resolved, contact technical support.

Exception

Service detected a node fault in resource pool [%s]: %s.

Locate and resolve the issue based on the error message. If it cannot be resolved, contact technical support.

Exception

Service detected a network service anomaly event: %s.

Locate and resolve the issue based on the error message. If it cannot be resolved, contact technical support.

Exception

Failed to submit the task for deleting instance [%s]. Reason: %s.

Locate and resolve the issue based on the error message. If it cannot be resolved, contact technical support.

Exception

Failed to delete instance [%s]. Try again later or contact service O&M personnel for a resolution.

Locate and resolve the issue based on the error message. If it cannot be resolved, contact technical support.

Exception

Failed to delete instance [%s]. Reason: Retries reached the maximum number of attempts (%s).

Locate and resolve the issue based on the error message. If it cannot be resolved, contact technical support.

Exception

Failed to delete instance [%s]. Only 1 instance remaining currently.

Locate and resolve the issue based on the error message. If it cannot be resolved, contact technical support.

Exception

Creation of deployment [%s] failed, %s; status transitioned from %s to %s.

Try again later or contact technical support.

Exception

Startup of deployment [%s] failed, %s; status transitioned from %s to %s.

Locate and resolve the issue based on the error message. For more solutions, see Failed to Start a Service.

Exception

Manual scaling of deployment [%s] failed. Error message: %s.

Check the error information. If resources are insufficient, release other resources or reduce the scaling quantity.

Exception

An anomaly exists in the service components that deployment [%s] depends on. Rescheduling is in progress; the current service status may be inaccurate.

Try again later or contact technical support.
Table 2 Warning events

Event Type

Event Message (xxx and %s represent placeholders)

Solution

Warning

Deployment [%s] has no [Health Check > Startup Probe] configured, which will prevent the model loading process from being monitored. The service may display a Running status but fail to handle predictions because the model is still loading.

Configure a health check startup probe for the corresponding deployment. For details, see Real-Time Service Health Check.

Warning

Checking the health status of some pods failed during the deployment [%s]. (Check the pod events for the cause.)
  • During service deployment or modification, it is common that the probe check fails. If the fault persists, the image service cannot be started. Locate the fault based on the logs.
  • During service runtime, if a chip or network fault occurs, the readiness probe and liveness probe check may fail. If the number of retries exceeds the threshold, the pod will be restarted.

Warning

Failed to pull the pod image in deployment [%s]. (Check the pod events for the cause.)
  • The specified image may not exist or has been deleted. Check whether the selected image exists.
  • If you create a service using an API, check whether the entered image path is correct and whether the current account has the permission to access the API.

Warning

Failed to start the pod container in deployment [%s]. (Check the pod events for the cause.)
  • Check whether the image architecture matches the resource pool architecture. For example, the resource pool of the Arm architecture requires the Arm image.
  • Check whether the configured resource specifications are proper and whether the resources are sufficient for starting the image. If model mounting is set, for example, in the mounting weight scenario, the memory must be greater than the total size of the mounted file.
  • Check whether the startup command configuration is correct.
  • Check whether the image can be started and work properly.

Warning

Pod scheduling failed in deployment [%s]. (Check the pod events for the cause.)

A scheduling failure event indicates that the scheduler fails to schedule the pod this time, but does not indicate that the pod cannot be scheduled. The scheduler continuously attempts to schedule the pod until the service deployment time expires.

  • Check whether the node resources are abundant. If "insufficient cpu/memory" is displayed in the pod event, the existing node resources cannot meet the requirements.
  • Check whether the node is tainted. If "taints" is displayed in the pod event, the node is tainted.
  • Check whether there are too many pods on the node. If "too many pods" is displayed in the pod event, the node is overloaded with pods. In this case, you need to stop some services.
  • Check the status of the model warmup task. When you deploy a real-time service and set the model source to Pre-warmed model, strong affinity rules are automatically configured to schedule pods only on pre-warmed nodes. The possible causes are as follows:
    • Insufficient nodes prevent scheduling the configured instances for model warmup. Add more nodes to fulfill the service requirements.
    • Pre-warmed nodes might lack enough resources for scheduling the current service. You can disable unnecessary services or scale out the resource pool. For details about how to stop a service deployment, see Stopping a Service Deployment. For details about how to scale out a resource pool, see Resizing a Dedicated Resource Pool.
  • Check the affinity scheduling configuration. If affinity scheduling is configured, the possible causes are as follows:
    • Affinity is configured for some nodes, and strong affinity is required. The available resources of the selected nodes cannot meet the pod requirements or the nodes are tainted. As a result, the pod cannot be scheduled.
    • Anti-affinity is configured for some nodes, and strong affinity is required. The available resources of the nodes other than the selected nodes cannot meet the pod requirements or the nodes are tainted. As a result, the pod cannot be scheduled.

Warning

Pod mounting failed during the deployment [%s]. Retrying... (Check the pod events for the cause.)

It takes some time to mount PVCs. If this error occurs in a short time, ignore it.

If SFS Turbo shows a mounting failure, verify its association status on the console's Network page.

Warning

Model configuration loading failed for pods in service deployment unit [%s]. Retrying... Details: %s

Rectify the fault by scenario based on the detailed error information. For details, see Table 3.
Table 3 Mode configuration loading failure details and solutions

Error Details

Solution

Unknown error. Try again later.

An unknown error has occurred. Please try again later

Try again later or contact technical support.

A system error (such as insufficient disk space or disk damage) occurs during the download of OBS files.

A system-related error (such as disk full error, disk corruption, etc.) occurred during the OBS download process

Check whether disk alarms (such as disk pressure alarms) are generated on the nodes where the service resource pool is deployed. If yes, handle the alarms in a timely manner or expand the capacity.

An exception (such as network or permission problems) occurs when downloading OBS files.

A download error (such as network issues, permission issues, etc.) occurred during the OBS download process

This problem may be caused by network fluctuation. Try again later.

The OBS permission and policy configuration may be incorrect. As a result, the OBS file fails to be downloaded when local mounting acceleration is enabled.

The disk space is insufficient before the OBS file is downloaded.

Insufficient disk space detected before downloading the OBS file

When local storage acceleration is enabled, the system checks the disk space before downloading OBS files. If the disk space is insufficient for OBS files, this event is generated. You are advised to expand the disk capacity or clear the disk.

Model warmup file not found.

The model warmup file does not exist

If the model warmup file is not found when the service is started during model warmup, the event information is displayed. Check the status of the model warmup task or create a warmup task again.

The model warmup task is not successful.

The model warmup task status is not successful

Try again later or create a model warmup task again.

An exception occurred when mounting SFS Turbo. Check whether the configuration is correct.

An error occurred while mounting the SFS Turbo. Please check whether the related configurations are correct

When local cache acceleration is enabled, an exception occurs during SFS Turbo mounting. On the Network page of the console, check whether the SFS Turbo file system is disassociated and whether its status is normal.

Mounting SFS Turbo failed because access to the mount directory timed out. Try again later.

A timeout was detected while accessing the mount directory when mounting the SFS Turbo. Please try again later

An access error occurred in the directory during the SFS Turbo mounting process when local cache acceleration was enabled. On the Network page of the console, check whether the SFS Turbo file system is disassociated and whether the status is normal. If the SFS Turbo file system is not disassociated and the status is normal, restore the SFS Turbo file system and deploy the service again.

Pod Events

pod events monitor the lifecycles and errors of pods within a Kubernetes cluster. In Kubernetes, a pod serves as the smallest deployable unit. Every instance of a real-time service matches one pod. Choose a specific instance to see its related pod events. Pod events help you understand the status and exceptions of service instances. For details about common pod events, see Table 4.

If you turn on Ingest Logs to LTS and choose Kubernetes events while setting up your service, these events (pod events) will be sent to LTS. Logs are kept for the past seven days by default. You can see pod events from the last seven days.

If you do not choose this option, you can only see pod events from the last hour.

Table 4 Pod events

Type

Event

Description

Solution

Normal

SuccessfulCreate

The container is created.

N/A

Normal

Started

The container is started.

N/A

Normal

Scheduled

The pod is scheduled to the node.

N/A

Normal

SuccessfulMountVolume

The storage volume is mounted.

N/A

Normal

Pulling

The image is being pulled.

N/A

Normal

Pulled

The image is pulled.

N/A

Normal

Healthy

The container is in the healthy state.

N/A

Normal

Killing

The container is being terminated.

N/A

Alarm

BackOffStart

The container fails to start.
  • Check whether the image architecture matches the resource pool architecture. For example, the resource pool of the Arm architecture requires the Arm image.
  • Check whether the configured resource specifications are proper and whether the resources are sufficient for starting the image. If model mounting is set, for example, in the mounting weight scenario, the memory must be greater than the total size of the mounted file.
  • Check whether the startup command configuration is correct.
  • Check whether the image can be started and work properly.

Alarm

CrashLoopBackOff

The container is repeatedly restarted and then breaks down.

Alarm

FailedPullImage

Pulling the image failed.
  • The specified image may not exist or has been deleted. Check whether the selected image exists.
  • If you create a service using an API, check whether the entered image path is correct and whether the current account has the permission to access the API.

Alarm

BackOffPullImage

The image fails to be pulled again.

Alarm

Unhealthy

The health check fails.
  • During service deployment or modification, it is common that the probe check fails. If the fault persists, the image service cannot be started. Locate the fault based on the logs.
  • During service runtime, if a chip or network fault occurs, the readiness probe and liveness probe check may fail. If the number of retries exceeds the threshold, the pod will be restarted.

Alarm

FailedScheduling

The pod cannot be scheduled to the node temporarily.

A scheduling failure event indicates that the scheduler fails to schedule the pod this time, but does not indicate that the pod cannot be scheduled. The scheduler continuously attempts to schedule the pod until the service deployment time expires.

  • Check whether the node resources are abundant. If "insufficient cpu/memory" is displayed in the pod event, the existing node resources cannot meet the requirements.
  • Check whether the node is tainted. If "taints" is displayed in the pod event, the node is tainted.
  • Check the affinity scheduling configuration. If affinity scheduling is configured, the possible causes are as follows:
    • Affinity is configured for some nodes, and strong affinity is required. The available resources of the selected nodes cannot meet the pod requirements or the nodes are tainted. As a result, the pod cannot be scheduled.
    • Anti-affinity is configured for some nodes, and strong affinity is required. The available resources of the nodes other than the selected nodes cannot meet the pod requirements or the nodes are tainted. As a result, the pod cannot be scheduled.

Alarm

FailedMount

The storage volume fails to be mounted.

It takes some time to mount PVCs. If this error occurs in a short time, ignore it.

If SFS Turbo shows a mounting failure, verify its association status on the console's Network page.

Alarm

InfersInitContainerFailed (incident)

The event details are as follows: Infers init container checked failed, errcode: %s, errmsg: %s. Exception information reported during the pre-check or operation performed by the init container where inference is started.

Rectify the fault based on the handling suggestions in Table 3 for the error message.
Parent Topic: Logging and Monitoring