Help Center/ ModelArts/ Model Inference/ Inference Deployment (New Version)/ Deploying and Using Real-Time Inference
Updated on 2026-07-07 GMT+08:00
View PDF
Share

Deploying and Using Real-Time Inference

The one-click deployment feature for real-time services is only available in the new console in the CN Southwest-Guiyang1 region.

ModelArts helps you quickly deploy AI models as ready-to-use inference services and offers APIs to easily incorporate these services into your custom applications.

ModelArts allows you to deploy a model as a real-time service that provides a real-time test UI and monitoring capabilities. This service provides a callable API. Real-time inference is used in situations that need fast responses, like online intelligent customer service and autonomous driving decisions.

Figure 1 Real-time inference deployment process

Preparations

Deploying a Real-Time Service

ModelArts enables cloud deployment, which involves running inference services using cloud infrastructure like servers, storage, and networks. This approach works best for tasks needing significant compute and handling large datasets.

Cloud-based deployment uses real-time inference to process individual requests instantly and deliver results immediately. ModelArts allows you deploy models into web services with internet-accessible APIs. It includes a UI for testing and monitoring these services. Once deployed, the service offers RESTful APIs for sending requests and receiving responses. Real-time inference is used in situations that need fast responses, like online intelligent customer service and autonomous driving decisions.

ModelArts allows you to deploy models as real-time inference services. For details, see Deploying a Real-Time Inference Service Using a Single Node.

Accessing a Real-Time Service

If a real-time service is in the Running state, it has been deployed. This service provides a standard RESTful API for calling. When accessing a real-time service, you can choose the authentication method, access channel, and transmission protocol that best suit your needs. These three elements make up your access requests and can be mixed and matched without any interference. For example, you can use different authentication methods for different access channels and transmission protocols.

Figure 2 Authentication method, access channel, and transmission protocol

ModelArts supports the following authentication methods for accessing real-time services (HTTPS requests are used as examples):

  • No authentication: No authentication is required.
  • Token-based authentication: Use Huawei Cloud Identity and Access Management (IAM) for authentication. The validity period of a token is 24 hours. When using a token for authentication, cache it to prevent frequent calls.
  • API key authentication: API key authentication provides a straightforward method for securing APIs with basic access control needs. Create an API key in the Huawei Cloud console and include it in the request header for API calls.

ModelArts allows you to call APIs to access real-time services in the following ways (HTTPS requests are used as examples):

Real-time service APIs are accessed using HTTPS by default. Additionally, the following transmission protocols are also supported:

  • Accessing a Real-Time Service Using WebSocket: WebSocket simplifies data exchange between the client and server and allows the server to proactively push data to the client. In the WebSocket API, if the initial handshake between the client and server is successful, a persistent connection can be established between them and bidirectional data transmission can be performed.
  • Accessing a Real-Time Service Using Server-Sent Events: Server-Sent Events (SSE) primarily facilitates unidirectional real-time communication from the server to the client, such as streaming ChatGPT responses. In contrast to WebSockets, which provide bidirectional real-time communication, SSE is designed to be more lightweight and simpler to implement.
Parent Topic: Inference Deployment (New Version)