Updated on 2026-09-08 GMT+08:00

Viewing Real-Time Service Logs

Overview

ModelArts inference services allow you to see real-time logs and connect with LTS. Use these logs to debug model startups, check request inputs and outputs, find runtime errors, confirm service logic, and analyze performance issues.

**Table 1** Log description
Dimension	Real-Time Service Logs	LTS Integration
Log coverage	Displays only container standard output and error logs. It does not include platform scheduling logs.	Simultaneously collects container standard output, error logs, and Kubernetes events (pod events). It does not include platform scheduling logs. Collects container file logs.
Storage and historical capabilities	No persistence. Retains only the latest snippets of the currently running instance. Logs are lost and cannot be retroactively retrieved once an instance restarts or is destroyed.	Persistent storage with custom retention periods. Caches operational logs within the last 7 days by default and supports historical queries.
Core functions	Basic viewing and simple keyword search.	Multi-dimensional retrieval, data analysis, log alerting, and log archiving.
Use cases	Model debugging and instant troubleshooting for individual inference issues.	Production O&M, fault tracing, compliance auditing, and performance statistics.
Enabling method	Enabled automatically by default.	Manually configured during the service information configuration phase when deploying the real-time service.
Billing	Real-time logging itself is free of charge.	LTS is billed on a pay-per-use basis. For details, see Log Tank Service Pricing Details.

Viewing Real-Time Logs

Create a real-time service by referring to Deploying a Real-Time Inference Service Using a Single Node. Do not select Ingest Logs to LTS in Advanced Settings.
Figure 1 Ingest Logs to LTS not selected
After the service is in the Running state, click the service name to access its details page. Switch to the Logs tab and select the instance and Pod to view real-time service logs.
Figure 2 Viewing service logs

Configuring Log Ingestion to LTS

Prerequisites

The cloud native log collection plugin must be installed in the dedicated resource pool for deploying services. For details, see Creating a Dedicated Resource Pool.

Configuration method

Create a real-time service by referring to Configuring Deployment Settings. Select Ingest Logs to LTS in Advanced Settings.
Figure 3 Ingest Logs to LTS
Alternatively, enable LTS when modifying a real-time service. To do so, locate the target service on the Real-Time Services page and choose More > Modify in the Operation column. Then, select Ingest Logs to LTS in Advanced Settings.

**Table 2** Parameters for ingesting logs to LTS
Parameter	Description
Container stdout	After you select Ingest Logs to LTS, Container stdout is selected by default and cannot be modified. Collect all container standard output and report it to Log Tank Service (LTS). The log retention period follows the setting of the corresponding log group, which is 30 days by default: For dedicated resource pools, the LTS log groups and log streams are those created and selected during the installation of the resource pool log collection plugin. For public resource pools, the system automatically creates LTS log groups and log streams. The naming conventions are: Log group: Modelarts-Infer-Log-Group-{NUM}; log stream: Inf-Stdout-{serviceId}.
Kubernetes events	After you select Ingest Logs to LTS, you can manually select Kubernetes events. Once enabled, Kubernetes events (pod events) will be collected and reported to LTS. By default, logs are retained for 7 days. You can view pod events in the event list on the inference service details page. For details, see Viewing Events of a Real-Time Service. If this option is not selected, you can only view pod events of the last hour in the event list on the inference service details page.
Collect container file logs	After you select Ingest Logs to LTS, you can manually select Collect container file logs. Collect container log files of the current service and report them to Log Tank Service (LTS). The log retention period follows the setting of the corresponding log group, which is 30 days by default. You need to specify the directory and file name where log files are stored inside the container. Configuration instructions: A log directory of /var/logs/* and a file name of .log represents all files with a .log* extension across all directories under /var/logs. A log directory of /var/logs/app_* and a file name of .log represents all files with a .log* extension in any directory matching the app_* pattern under /var/logs. Duplicate path configurations will only take effect once. Common paths: Device log: */home/ma-user/ascend/log/run/device-/ Run plog: /home/ma-user/ascend/log/run/plog/ Debug plog: /home/ma-user/ascend/log/debug/plog/ System log: /var/logs/ Exception log: /home/ma-user/ascend/log/exceptionDump/**

Constraints

Container standard output and container log files are exported to the same log stream; routing them to different log streams separately is not supported.
A maximum of 10 container log file configurations are supported.
If each service is configured with 10 container log file collection paths, a single resource pool can support a maximum of 100 deployed services. The log collection policy size supports up to 1 MB; exceeding 1 MB will prevent adding new log collection policies.
The LTS output log group and log stream can only be selected when adding the log plugin to a resource pool. All services deployed within the same resource pool share a single log stream.

Viewing Logs Reported to LTS

After the service is interconnected with LTS, access the service details page, and switch to the Logs tab to view logs.
Select the deployment, instance, and Pod to be viewed. Deleted ones can also be selected for fault locating.
Figure 4 Viewing instance logs
Select the time range of logs.
Figure 5 Selecting the time range of logs to be viewed
View log information.
- Log search: Search logs using specific keywords or phrases. Narrow your results by selecting a specific time range to find events and issues during that period.
- Statistical charts: After sending logs to LTS, use SQL analysis syntax to find important log data and view the results as statistical charts.
- Log analysis: Before searching for analyzing logs, set up structured data and indexing for them.
- Real-time logs: Once you connect your real-time service logs to LTS, they will be sent every minute. You can view these updates from the Real-Time Logs tab, where you can also easily search and analyze the data.
For details, see Log Tank Service.
To detect abnormal logs in a timely manner and further resolve issues, follow the steps in Configuring LTS Log Alarms.

Configuring LTS Log Alarms

After the service is interconnected with LTS, access the service details page, and switch to the Logs tab to view logs.
Click Log Settings. In the displayed dialog box, click the Alarm Rules tab.
Click Create. In the displayed dialog box, configure an alarm rule.
For details about the parameters, see Configuring Log Alarm Rules.

To add a notification when an alarm is generated, enable and create alarm notification rules on the LTS console. For details, see Creating a Message Template on the LTS Console.
Click OK.
You can change the alarm settings in the log settings dialog box. For details, see Follow-up Operations on Alarm Rules.

FAQs

How Do I Promptly Detect Service Container Exceptions During Real-Time Service Deployment or Instance Restart?

During model service deployment, the system must download these files from the storage services and mount them to service containers to complete the deployment process. However, in actual operation, if any of the dependency storage services encounter issues, such as network interruptions, service unavailability, or insufficient disk space, the system may fail to detect these exceptions effectively. This can lead to deployment failures or prolonged service unavailability, without any corresponding event reporting, thereby impacting service reliability and user experience.

To address this issue, the inference platform integrates with the LTS system. Through LTS log-based alerts, developers can be promptly notified of anomalies during the model weight file reading process, helping ensure smooth service deployment.

Rectify the fault by following FAQs > "How Do I Promptly Detect Service Container Exceptions During Real-Time Service Deployment or Instance Restart?"

Parent Topic: Logging and Monitoring