Viewing Inference Metrics and Configuring Alarms on the AOM Platform

Overview

ModelArts sends all monitoring data for its inference services to AOM. The ModelArts console shows only a subset of this data. For refined O&M, alarms, and custom dashboards, you need to use AOM.

AOM is a one-stop, multi-dimensional O&M management platform for cloud applications. It monitors applications and cloud resources in real time, and can perform daily monitoring of the status of ModelArts real-time services and model workloads. You can obtain the metrics of each ModelArts real-time service and model on the AOM console. It takes a period of time to transmit and display monitored data. The statuses displayed on AOM are obtained 5 to 10 minutes before. You can view the monitoring data of a newly created real-time service 5 to 10 minutes later.

Impact of Inference Service Response Formats on Token Metrics

Inference service responses must comply with the standard OpenAI format, specifically requiring:

The response must be streaming or non-streaming, and the response body must contain the usage.prompt_tokens, usage.completion_tokens, and usage.total_tokens fields (for streaming responses, these must be included in the final data chunk).

If an inference service response fails to meet these requirements, token-related metrics will be abnormal:

Streaming scenario: Both input token count and output token count will display as 0. TTFT can still be displayed, but TPOT will be biased as it is calculated based on chunk count.
Non-streaming scenario: All token-related metrics will fail to display.

Service Access Metrics

The cloud service platform provides Application Operations Management (AOM) to help you better understand the statuses of ModelArts real-time services and model workloads. You can use Cloud Eye to automatically monitor your ModelArts real-time services and model loads in real time and manage alarms and notifications so that you can obtain the performance metrics of ModelArts and models.

Constraints on ModelArts response body parsing statistics:

1. When calculating the average values for metrics such as tokens per request, TTFT, and TPOT, the statistical upper limit is 110K. When calculating TPXX trend statistics, the upper limit is 100K due to histogram bucket boundary limitations.

2. The gateway parses data in OpenAI format to count the number of tokens. Currently, data in non-OpenAI format cannot be counted.

**Table 1** Service access metrics (histogram type)
Category	Name	Atomic Metric	Description	Unit	Upper Limit (K)	Value Range	Collection Period	Metric Label
Service latency	Average request latency of a real-time service	infer_service_request_cost	This metric is used to calculate the average request latency. It is calculated as follows: infer_service_request_cost_sum/infer_service_request_cost_count	ms	No upper limit	≥ 0	30s	Service metric label
TTFT	The response time recorded for the first token across requests	infer_service_first_token_cost	This metric is used to measure the TTFT in a measurement period. It is used to calculate the average latency using the following formula: sum(increase(infer_service_first_token_cost_sum{service_id="{service_id}"}[59999ms]))by(service_id)/sum(increase(infer_service_first_token_cost_count{service_id="{service_id}"}[59999ms]))by(service_id)	ms	110K	≥ 0	30s	Service metric dimension
TPOT	Average latency of each output token	infer_service_per_token_cost	This metric is used to measure the TPOT in a measurement period. It is used to calculate the average latency using the following formula: sum(increase(infer_service_per_token_cost_sum{service_id="{service_id}"}[59999ms]))by(service_id)/sum(increase(infer_service_per_token_cost_count{service_id="{service_id}"}[59999ms]))by(service_id)	ms	110K	≥ 0	30s	Service metric dimension
Input tokens	Average input tokens of real-time service requests	infer_service_input_token_quantity	This metric is used to calculate the average number of input tokens in requests. It is calculated as follows: infer_service_input_token_quantity_sum/infer_service_input_token_quantity_count	Number	110K	≥ 0	30s	Service metric label
Output tokens	Average output tokens of real-time service requests	infer_service_output_token_quantity	This metric is used to calculate the average number of output tokens in requests. It is calculated as follows: infer_service_output_token_quantity_sum/infer_service_output_token_quantity_count	Number	110K	≥ 0	30s	Service metric label
TTFT trend	TTFT trend of a service	infer_service_first_token_cost_bucket	TTFT trend of a service. For details about how to calculate TPxx, see the Prometheus statement: histogram_quantile(XX%, avg(rate(infer_service_first_token_cost_bucket{service_id="{service_id}"}[1m]))by(le))	ms	100K	≥ 0	30s	Service metric label
Service input tokens	Service input tokens	infer_service_input_token_quantity_bucket	Input tokens used by a service. For the calculation method of TPxx input token count, see the Prometheus statement. For example, TP90 is calculated as: histogram_quantile(0.90, avg(rate(infer_service_input_token_quantity_bucket{service_id="6d26f238-52b0-4184-8b50-d621a0c80eb4"}[1m]))by(le))	Number	100K	≥ 0	30s	Service metric label
Service output tokens	Service output tokens	infer_service_output_token_quantity_bucket	Output tokens used by a service. For the calculation method of TPxx output token count, see the Prometheus statement. For example, TP90 is calculated as: histogram_quantile(0.90, avg(rate(infer_service_output_token_quantity_bucket{service_id="6d26f238-52b0-4184-8b50-d621a0c80eb4"}[1m]))by(le))	Number	100K	≥ 0	30s	Service metric label
TPOT trend	TPOT trend	infer_service_per_token_cost_bucket	TPOT trend of a service. For details about how to calculate TPxx, see the Prometheus statement: histogram_quantile(XX%, avg(rate(infer_service_per_token_cost_bucket{service_id="{service_id}"}[1m]))by(le))	ms	100K	≥ 0	30s	Service metric label
Service request latency trend	Service request latency trend	infer_service_request_cost_bucket	Request latency trend of a service. For the calculation method of TPxx latency , see the Prometheus statement. For example, TP90 is calculated as: histogram_quantile(0.90, avg(rate(infer_service_request_cost_bucket{service_id="6d26f238-52b0-4184-8b50-d621a0c80eb4"}[1m]))by(le))	ms	3600K	≥ 0	30s	Service metric label

**Table 2** Service access metrics (dashboard type)
Category	Name	Atomic Metric	Description	Unit	Upper Limit (K)	Value Range	Collection Period	Metric Label
Requests	Number of service requests	infer_service_request_total	Number of API calls of a real-time service in a collection period	Count	No upper limit (Gauge)	≥ 0	30s	Service API metric label
Real-time connections	Number of connections	infer_service_request_connect_count	Number of real-time connections of a real-time service at the time when the metric is collected	Number	100 (default upper limit: 10; theoretical upper limit: 100)	≥ 0	30s	Service metric label
QPS	Service request QPS	infer_service_request_total	Request rate of service calls in a unit time, specifically the number of requests per second. Request QPS = Number of requests/60. NOTE: The number of requests and request QPS use the same atomic metric infer_service_request_total.	Queries/s	Management plane service delivery settings	≥ 0	30s	Service metric label
Used cores	CPU cores used by a real-time service	ma_container_cpu_used_core	Real-time CPU core usage of a real-time service at the time of collection, representing the actual number of CPU cores consumed by the container.	Core	Subject to the upper limit of the server specifications requested by the service	≥ 0	30s	Service metric label
CPU utilization	CPU utilization of a real-time service	ma_container_cpu_util	Real-time CPU utilization of a real-time service at the time when the metric is collected	%	100	0%–100%	30s	Service metric label
Used memory	Memory usage of a real-time service	ma_container_memory_used_megabytes	Real-time memory usage of a real-time service at the time when the metric is collected	MB	Subject to the upper limit of the server specifications requested by the service	≥ 0	30s	Service metric label
Memory utilization	Memory utilization of a real-time service	ma_container_memory_util	Real-time memory utilization of a real-time service at the time when the metric is collected	%	100	0%–100%	30s	Service metric label
NPU utilization	NPU utilization of a real-time service	ma_container_npu_util	Real-time NPU utilization of a real-time service at the time when the metric is collected	%	100	0%–100%	30s	Service metric label
Used NPU memory	NPU memory usage of a real-time service	ma_container_npu_memory_used_megabytes	Real-time NPU memory usage of a real-time service at the time when the metric is collected	MB	Subject to the upper limit of the server specifications requested by the service	≥ 0	30s	Service metric label
NPU memory utilization	NPU memory utilization of a real-time service	ma_container_npu_memory_util	Real-time NPU memory utilization of a real-time service at the time when the metric is collected	%	100	0%–100%	30s	Service metric label
GPU utilization	GPU utilization of a real-time service	ma_container_gpu_util	Real-time GPU utilization of a real-time service at the time when the metric is collected	%	100	0%–100%	30s	Service metric label
Used GPU memory	GPU memory usage of a real-time service	ma_container_gpu_mem_used_megabytes	Real-time GPU memory usage of a real-time service at the time when the metric is collected	MB	Subject to the upper limit of the server specifications requested by the service	≥ 0	30s	Service metric label
GPU memory utilization	GPU memory utilization of a real-time service	ma_container_gpu_mem_util	Real-time GPU memory utilization of a real-time service at the time when the metric is collected	%	100	0%–100%	30s	Service metric label
Inbound network throughput	Inbound network throughput of a real-time service	ma_container_network_receive_bytes	Real-time inbound network throughput of a real-time service at the time when the metric is collected	Byte/s	Subject to the upper limit of the service network traffic rate	≥ 0	30s	Service metric label
Outbound network throughput	Outbound network throughput of a real-time service	ma_container_network_transmit_bytes	Real-time outbound network throughput of a real-time service at the time when the metric is collected	Byte/s	Subject to the upper limit of the service network traffic rate	≥ 0	30s	Service metric label
Token metrics	Number of tokens generated by a real-time service	infer_service_token_count	Number of tokens generated by a real-time service in a statistical period	Number	No upper limit (Gauge)	≥ 0	30s	Service metric label
	Average TTFT	infer_service_first_token_avg_cost	Average TTFT in a period	ms	No upper limit (Gauge)	≥ 0	30s	Service metric label
	Minimum TTFT	infer_service_first_token_min_cost	Minimum TTFT in a period	ms	No upper limit (Gauge)	≥ 0	30s	Service metric label
	Maximum TTFT	infer_service_first_token_max_cost	Maximum TTFT in a period	ms	No upper limit (Gauge)	≥ 0	30s	Service metric label
	Average output latency per token	infer_service_per_token_avg_cost	Average output latency of each token in a period (excluding the first token)	ms	No upper limit (Gauge)	≥ 0	30s	Service metric label
	Maximum output latency per token	infer_service_per_token_max_cost	Maximum output latency of each token in a period (excluding the first token)	ms	No upper limit (Gauge)	≥ 0	30s	Service metric label
	Minimum output latency per token	infer_service_per_token_min_cost	Minimum output latency of each token in a period (excluding the first token)	ms	No upper limit (Gauge)	≥ 0	30s	Service metric label
	Average time consumed by each token	infer_service_token_latency_avg_cost	Average time consumed by each token in a period	ms	No upper limit (Gauge)	≥ 0	30s	Service metric label
	Minimum time consumed by each token	infer_service_token_latency_min_cost	Minimum time consumed by each token in a period	ms	No upper limit (Gauge)	≥ 0	30s	Service metric label
	Maximum time consumed by each token	infer_service_token_latency_max_cost	Maximum time consumed by each token in a period	ms	No upper limit (Gauge)	≥ 0	30s	Service metric label
Total TTFT	Total TTFT of a service in a period	infer_service_first_token_cost_sum	Total TTFT of a real-time service in a period	ms	No upper limit (Gauge)	≥ 0	30s	Service metric label
Total input tokens	Total service input tokens in a period	infer_service_input_token_quantity_sum	Total number of input tokens used by a real-time service in a period	Number	No upper limit (Gauge)	≥ 0	30s	Service metric label
Total output tokens	Total service output tokens in a period	infer_service_output_token_quantity_sum	Total number of output tokens used by a real-time service in a period	Number	No upper limit (Gauge)	≥ 0	30s	Service metric label
Total TPOT	Total TPOT of a service in a period	infer_service_per_token_cost_sum	Total TPOT of a real-time service in a period	ms	No upper limit (Gauge)	≥ 0	30s	Service metric label
Total service request latency	Total request latency of a service in a period	infer_service_request_cost_sum	Total latency of real-time service requests in a period	ms	No upper limit (Gauge)	≥ 0	30s	Service metric label

Label Metrics

**Table 3** Service API metric labels
Label	Description
service_id	Inference service ID, for example, 9f322d5a-b1d2-4370-94df-5a87de27d36e.
group_id	Inference service deployment ID.
project_id	Project ID of the account to which the user belongs.
code	Request return code, including 2xx, 4xx, and 5xx.
method	Inference request method.
path	Inference request path.
source	Pod ID for the request to pass through the inference component.
namespace	Metric name, which corresponds to the Metric column in Table 1.
instance_name	Name of the inference service deployment replica.
service_name	Name of the inference service.

**Table 4** Service metric labels
Label	Description
service_id	Inference service ID, for example, 9f322d5a-b1d2-4370-94df-5a87de27d36e.
group_id	Inference service deployment ID.
project_id	Project ID of the account to which the user belongs.
source	Pod ID for the request to pass through the inference component.
namespace	Metric name, which corresponds to the Metric column in Table 1.

Viewing Metrics on AOM

Prerequisites

The ModelArts real-time service is running properly.

The real-time service has been properly running for at least 10 minutes.
The monitored data and graphics are available for a new real-time service after the service runs for at least 10 minutes.

Cloud Eye does not display the metrics of a faulty or deleted real-time service. The monitoring metrics can be viewed after the real-time service starts or recovers.

Procedure

In the service list, choose Management & Governance > Application Operations Management.
In the navigation pane on the left, choose Metric Browsing. Set Metric Sources to Prometheus_AOM_Default and set the statistical method and statistical period as needed.
Figure 1 Selecting metric sources
At the bottom of the metric browsing page, select one or more target metrics by configuring All metrics or Prometheus statement.
The following uses All metrics to obtain real-time service job metrics.
- Metric: Enter a metric, for example, ma_container_cpu_util.
- Conditions: Enter the condition (instance_name) and dimension value (real-time service job ID, which can be obtained from the real-time service details page on the ModelArts console). The system will show the metric monitoring curve for that job instantly.
  Figure 2 Setting real-time service job metrics
For details about how to add metrics by configuring Prometheus statement, see "Observability Metric Browsing" in Application Operations Management > User Guide (2.0).

Custom Metric Collection

Overview

If the default ModelArts metrics do not meet your needs, you can set up custom metrics (in Prometheus format) when deploying a real-time inference service. ModelArts will collect and send these metrics to AOM.

Constraints

ModelArts calls the HTTP API provided in the custom metric configuration every 30 seconds to obtain metric data.
The metric data text returned by the HTTP API provided in the custom metric configuration cannot exceed 240 KB.

Configuration entry

Log in to the ModelArts console and choose Model Inference > Real-Time Inference. When deploying a real-time service, check Custom Metric Collection under Advanced Settings during the deployment configuration phase. For details, see Deploying a Real-Time Inference Service Using a Single Node.

Select HTTPS (recommended) or HTTP as the protocol. Set the metric collection port to 8000 or 9090. Use metrics as the fixed API path.

Figure 3 Custom Metric Collection
Click to enlarge

Custom metrics are sent to the Prometheus instance in AOM. You can query these metrics in AOM. For details, see Viewing Metrics on AOM.

Data Format of Custom Metrics

The format of custom metrics data must comply with the open metrics specifications. That is, the format of each metric must be:

<metric_name>{<tag_name>=<tag_value>,...} <sample_value> [timestamp_in_millisecond]

The following shows an example (the comment starts with #, which is optional):

# HELP http_requests_total The total number of HTTP requests.
# TYPE http_requests_total gauge
html_http_requests_total{method="post",code="200"} 1656 1686660980680
html_http_requests_total{method="post",code="400"} 2 1686660980681

Configuring Alarms on AOM

Configuring alarms automatically detects issues like service resource problems and faulty requests, and sends timely alerts. This removes the need for constant human checks, accelerates issue resolution, ensures smooth service, and helps improve maintenance policies.

Alarm configuration path 1:

Log in to the ModelArts console and choose Model Inference > Real-Time Inference.
Click the name of the target real-time service. On the service details page, go to the Monitoring tab and click Alarm Rules. In the AOM console, configure alarm rules and notifications. For details, see AOM Alarm Rule Overview.
Figure 4 Alarm Rules

Alarm configuration path 2:

Log in to the AOM console. In the navigation pane, choose Alarm Center > Alarm Rules. Then click Create Alarm Rule.

Set alarm rules (using connection count as an example):

Rule Type: Select Metric alarm rule.
Configuration Mode: Select PromQL.

Default Rule: Select Custom and enter the following information in the command input box:

sum(avg_over_time(infer_service_request_connect_count{service_id="XXX"}[1m]))by(service_id) > 8000

Alarm Rule Details > Duration: Select 1 minute, which triggers a major alarm when the rule's conditions persist continuously for one minute.
(Optional) Alarm Notification: To get notified of alarms by email or SMS message, configure action rules for the alarm rule. If no action rule is available, you can create one.