Help Center/ ModelArts/ Model Inference/ Inference Deployment (New Version)/ Logging and Monitoring/ Viewing Performance Metrics of a Real-Time Service on ModelArts
Updated on 2026-09-08 GMT+08:00
View PDF
Share

Viewing Performance Metrics of a Real-Time Service on ModelArts

Overview

ModelArts inference monitoring provides a full-stack, end-to-end observability capability for real-time inference services. Covering four major dimensions (the resource layer, network layer, request layer, and model inference layer), it collects real-time operational data and presents it visually. It supports performance diagnosis, exception alerting, and capacity planning to ensure the stable and highly efficient operation of AI services.

Table 1 lists the supported monitoring dimensions and metrics.

Table 1 Supported monitoring dimensions

Dimension

Metric

Resource utilization monitoring

CPU/Memory: Real-time collection of CPU utilization, core count, memory utilization, and memory usage (MB) for inference instances to reflect basic resource load.

GPU/NPU: Collects GPU/NPU utilization, VRAM utilization, and VRAM occupancy to precisely monitor AI acceleration hardware loads. This is optimized for LLMs and deep learning inference scenarios.

Network traffic monitoring

Inbound/Outbound traffic rate: Real-time statistics of data received/sent rates (Byte/s) to identify network bandwidth bottlenecks and anomalous traffic anomalies.

Connections: Real-time active connection counts and connection trends to evaluate concurrent handling capacity and detect long-connection leaks.

Request performance monitoring

Service request count: Aggregated total of 2xx (success), 4xx, and 5xx (exception) requests within a statistical period, providing a direct reflection of service availability.

Service request QPS: Queries per second, measuring the service's concurrent processing capabilities.

Service request latency: Average latency along with TP50/TP90/TP99 latency (ms) to pinpoint slow requests and performance bottlenecks.

Principle: Intercepts incoming traffic at the inference service ingress, records request/response timestamps and status codes, and aggregates them per statistical period. Supports millisecond-level latency precision.

Specialized model inference monitoring (LLM adapted)

TTFT: The duration from when a request is initiated until the first token is returned. This is a core metric for LLM streaming inference that reflects model initialization and first-frame generation efficiency.

TPOT: The generation latency for each subsequent token, used to measure the stability of continuous model inference.

Token count statistics: Input, output, total token counts, and increments. This adapts to billing systems and helps analyze the scale of model inputs and outputs.

Viewing Different Types of Monitoring Metrics

Table 2 Methods for viewing different types of monitoring metrics

Viewing Method

Metrics

Use Case

Reference

ModelArts console

LLM inference service metrics (QPS, latency, TTFT, token counts, service status) and basic resource monitoring metrics.

Daily operations and service observation.
  1. Inference Metrics Viewable on the ModelArts Console
  2. Viewing Metrics on the ModelArts Console
  3. Viewing Metrics on a Custom Dashboard on the ModelArts Console

AOM console

Underlying infrastructure resources (CPU/GPU/NPU/nodes), full raw metrics, alarms, and custom dashboards.

O&M troubleshooting, capacity planning, and SLA assurance.

Viewing Performance Metrics of a Real-Time Service on AOM

Custom metrics

Internal model states, service-specific dimensions, performance breakdowns, cache/queues, and billing dimensions.

Deep LLM O&M scenarios that are not covered by default metrics.

Custom Metric Collection

Prerequisites

  • Permission configuration: You must be configured with AOM read-only permissions. If you are using role/policy-based permissions, assign the AOM ReadOnlyAccess system policy. If you are using identity policy-based permissions, assign the AOMReadOnlyPolicy system identity policy.
  • Service deployment: The real-time service must be deployed, with its status showing as Running, Alarm, or Upgrading (services in Deploying or Stopped status will not have complete monitoring data).

Constraints

The monitoring time span can be up to 15 days, and the statistical period can be 1 minute, 5 minutes, 15 minutes, or 1 hour.

Inference Metrics Viewable on the ModelArts Console

Table 3 Inference metrics viewable on the ModelArts console

Parameter

Description

Used CPU

Used CPU cores of a real-time service.

CPU Usage

CPU usage of a real-time service.

Used Memory

Used memory (unit: MB) of a real-time service.

Memory Usage

Memory usage of a real-time service.

GPU Usage

Available when GPU resources are used. GPU usage of a real-time service.

GPU Memory

Available when GPU resources are used. GPU memory utilization and usage of a real-time service.

NPU Usage

Available when NPU resources are used. NPU usage of a real-time service.

NPU Memory

Available when NPU resources are used. NPU memory utilization and usage of a real-time service.

Outbound Network Throughput

Inbound traffic rate of a real-time service, in Byte/s.

Inbound Network Throughput

Outbound traffic rate of a real-time service, in Byte/s.

Service Requests

Number of calls with various return codes (2xx, 4xx, 5xx) during a specific period for a real-time service. This represents the total call volume in that period. The 2xx code shows the number of successful calls.

Service Request QPS

QPS of a real-time service. The value is the total number of calls in a statistical period divided by the number of seconds in the selected period.

Connections

Real-time connection sampling data of a real-time service, which is used to provide the number of requests that are in the connection setup state.

Service Request Latency

Average request latency trend, TP50, TP90, and TP99 of a real-time service, in milliseconds (ms).

TPxx represents the threshold value below which xx% of user requests fall for a given metric within the statistical interval. For example, TP90 = 300 ms indicates that 90% of service requests are completed within 300 milliseconds. The value is an estimated value, which may be a decimal.

Time to First Token

Time to first token in a period, including the average latency, TP50, TP90, and TP99.

TPxx represents the threshold value below which xx% of user requests fall for a given metric within the statistical interval. For example, a TP90 value of 300 ms for time to first token indicates that 90% of user requests experience a time to first token within 300 milliseconds. The value is an estimated value, which may be a decimal.

Post-First-Token Latency

Time to each token in a period, including the average latency, TP50, TP90, and TP99.

TPxx represents the threshold value below which xx% of user requests fall for a given metric within the statistical interval. For example, a TP90 value of 300 ms for non-first-token latency indicates that, within a single generation cycle, 90% of users experience per-token output latency within 300 milliseconds. The value is an estimated value, which may be a decimal.

Service Token Increment

Increment of tokens in a period, including the total number of input tokens, output tokens, and service tokens. Total number of service tokens = Total number of input tokens + Total number of output tokens

Service Input Tokens

TP50, TP90, and TP99 of the input tokens of real-time services in a period.

TPxx represents the threshold value below which xx% of user requests fall for a given metric within the statistical interval. For example, if TP90 of the input tokens is 300, the input tokens of 90% of user requests are within 300. The value is an estimated value, which may be a decimal.

Output Tokens

TP50, TP90, and TP99 of the output tokens of real-time services in a period.

TPxx represents the threshold value below which xx% of user requests fall for a given metric within the statistical interval. For example, if TP90 of the output tokens is 300, the output tokens of 90% of user requests are within 300. The value is an estimated value, which may be a decimal.

The CPU usage, memory usage, GPU usage and memory, NPU usage and memory are real-time data.

After creating the service, the system collects data on service requests, QPS, connections, request latency, token latency, and token count.

Monitoring metrics involving increments and TPxx are calculated incrementally. There may be too few sample points in a period and the calculated value may be 0.

Viewing Metrics on the ModelArts Console

  1. Log in to the ModelArts console and choose Model Inference > Real-Time Inference.
  2. Click the name of the target real-time service to go to its details page.
  3. Click the Monitoring tab. The system shows the last 15 minutes of service-level monitoring data by default, with statistics updated every minute.
    Figure 1 Service-level monitoring information on the service details page

  4. View monitoring data of the service and its versions in the Monitoring tab.
    • Choose a time range (up to 15 days) and a statistical period to view the needed monitoring data.
      Figure 2 Choosing a time range and a statistical period

    • Choose a deployment to view service deployment-level monitoring (only resource usage metrics are supported).
      Figure 3 Monitoring information of a service instance

    • Choose a deployment, instance, and pod to view service pod-level monitoring (only resource usage metrics are supported).
      Figure 4 Monitoring information of service pods

Viewing Metrics on a Custom Dashboard on the ModelArts Console

To display metrics on the console in a personalized way, customize a dashboard based on your service needs.

  1. Log in to the ModelArts console. In the navigation pane, choose Model Inference > Real-Time Inference.
  2. Click the name of the target real-time service to go to its details page.
  3. Click the Monitoring tab. Switch to the Custom Dashboard tab.

    The custom dashboard allows you to monitor metric values and trends in real time, and set alarm rules for important ones. For details about monitoring metrics and related operations, see Observability Metric Browsing.

Parent Topic: Logging and Monitoring