Real-Time Service Intelligent O&M (HRA Plugin)

Overview

The core purpose of intelligent O&M (HRA plugin) is to resolve inappropriate P/D allocation ratios in LLM deployments. It adapts to highly complex scenarios such as large-scale inference, cross-node deployment, and load balancing. Typical scenarios are as follows:

• High-concurrency LLM inference scenarios

  LLM inference is divided into prefill units and decode units. During service peaks, the P/D ratio can easily become imbalanced, leading to low resource utilization, high response latency, and insufficient throughput. Intelligent O&M uses simulation algorithms to provide optimal ratio recommendations. Adjusting these manually improves resource utilization, reduces latency, and increases service throughput.

• Performance optimization with fixed resource capacity

  When resource pool hardware specifications (GPU/CPU/memory) are fixed and cannot be scaled out, Intelligent O&M recommends the optimal P/D ratio to maximize inference performance within existing resource limits, preventing resource waste or performance bottlenecks.

• Iterative tuning for multi-version models

  When multiple deployments and model versions run concurrently under the same service, different models have varying P/D ratio requirements. Intelligent O&M can provide exclusive ratio recommendations based on the asset tags of each model, adapting to the compute demands of different models and ensuring the stable operation of multi-version services.

• Fine-grained resource management

  This applies when inference instance resource allocation must be precisely controlled to avoid cost increases from over-provisioning or service anomalies from under-provisioning. Intelligent O&M provides real-time ratio recommendations to replace traditional manual empirical configurations, achieving fine-grained and scientific resource scheduling.

Constraints

• Resource pool limitations: The intelligent O&M feature is only supported in physical dedicated resource pools that have the HRA plugin installed. Public resource pools and dedicated resource pools without the HRA plugin do not support this capability.
• Model asset limitations: Model assets deployed for real-time services must carry the "dynamic ratio recommendation" tag. Models without this tag cannot trigger intelligent O&M monitoring and recommendations. Currently, only model assets pre-built on ModelArts support this feature; custom models are not supported.
• Functional dependency limitations: Intelligent O&M is a manual, assisted tuning capability and does not depend on auto-scaling features. You must manually adjust the P/D ratio based on the recommended values; automatic adjustment of instance ratios is not supported.
• Scope of application limitations: This feature only adapts to inference services using LLM cross-node deployments and disaggregated P/D architectures. Simple small models and single-node deployment services do not have the concept of a P/D ratio and cannot use this feature.
• Unit name limitations: The current algorithm only supports monitoring the ratio of prefill and decode inference units, and these names serve as the unique basis for metric collection. Do not modify these unit names; otherwise, data collection anomalies will occur.

Prerequisites

• The real-time service has been deployed in a physical dedicated resource pool with the HRA plugin installed, and the service status is Running. To check whether a dedicated pool has the HRA plugin installed, see Viewing Resource Pool Plug-ins.
• The deployed model asset carries the "dynamic ratio recommendation" tag.

Configuration Operations

1. Log in to the ModelArts console and choose Model Inference > Real-Time Inference.
2. Click the name of the target real-time service to enter its details page. Switch to the Intelligent O&M tab to enter the configuration interface.
3. Turn on the monitoring switch and configure the monitoring algorithm parameters. The system will then automatically calculate and display the optimal P/D ratio recommendation based on real-time metrics and simulation algorithms.
4. Follow the prompt and click the optimize button to view the optimization suggestions provided by the system. Click the apply button to complete the intelligent O&M ratio tuning.