Testing a Real-Time Service

Accessing a Synchronous Real-Time Service

On the ModelArts console, choose Model Inference > Real-Time Inference in the left navigation pane. On the displayed real-time inference list, click Call Service in the Operation column of the target service to view the call information.

Alternatively, click the target service name to go to the service details page and obtain the call information in Call Service.

Figure 1 Calling information

URL example: https://10.XX.XXX.XXX/v2/infer/testxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

Custom Access Address

If you have already set up other network components that handle Layer-4 traffic before creating the real-time service, click Custom URL in the Call Information and add the IP address or domain name of the network component. This allows you to access the real-time service using the IP address or domain name. For example, if you have also connected APIG before configuring the ELB and wants to access the real-time service via the APIG domain name, you need to add the APIG domain name to the custom addresses.

Figure 2 Modifying the access address

• Access protocol: HTTPS, HTTP, WSS, and WS are supported. Using HTTP or WS plaintext can leave you vulnerable to threats like data leakage, tampering, traffic hijacking, and phishing attacks. Exercise caution when using this protocol.
• Domain name: Enter a domain name or IP[:Port].
• Path: The default path is /v2/{infer_type}/{service_id}. infer_type indicates the inference call type, which can be synchronous (infer) or asynchronous (async-infer). service_id indicates the service ID. You can also customize the path, which must start with a slash (/).

Using the Prediction Feature

After a service is deployed, verify its performance using the Prediction tab on the service details page. The service prediction tab offers the rest client. You can choose your service's request method and input the prediction path.

In the Body section, choose the appropriate data format (raw, binary, stream, or fromData).

• raw: Used to send raw text data, such as JSON, XML, or plaintext.
• binary: Used to upload binary files, such as images, audios, videos, or model files.
• stream: Block-based stream transmission is supported, which is applicable to real-time or continuous data input scenarios.
• fromData: Submit data in multipart/form-data format. Files can be uploaded and other text fields can be entered at the same time.

You can enter header information in Headers, for example, API key authentication information. Replace {API Key} with your own API key. Click Predict to send a prediction request.

Deleting the authorization key pair makes the system switch to IAM token authentication automatically.

Example:

This guide demonstrates deploying the Qwen3-32B model using the Ascend-vLLM framework with one click. Here are the required parameters:

• Request method: POST
• Request path: https://***/v2/infer/***/v1/chat/completions
• Request body:

{
    "model": "qwen3_32b",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Hello"}
    ]
}

Figure 3 Real-time prediction example

Figure 4 Example headers

Using Cloud Shell to Debug a Real-Time Service Instance Container

You can use Cloud Shell provided by the ModelArts console to log in to the instance container of a running real-time service.

Cloud Shell can only access a container when the associated real-time service is deployed within a dedicated resource pool.

1. Log in to the ModelArts console. In the navigation pane on the left, choose Model Inference > Real-Time Inference.
2. Click the real-time service name or ID to access its details page.
3. In the Cloud Shell tab, select the deployment, instance, and Pod. If the connection status changes to Connection Succeeded, you have logged in to the instance container.
   If the server disconnects due to an error or remains idle for 10 minutes, you can select Reconnect to regain access to the Pod.

   Figure 5 Cloud Shell
   

   If you encounter a path display issue when logging in to Cloud Shell, press Enter to resolve the problem.

   Figure 6 Path display issue
   
4. After logging in to the container, execute the necessary debugging commands in its terminal. Example:
   

   The following is for reference only. The actual log paths and service health check methods depend on your service configuration. Refer to your image settings and container startup commands for details.

   View logs:

   tail -f /var/log/app.log

   Check the service status:

   systemctl status app

   Run a custom script:

   ./debug_script.sh

5. After the debugging, exit the container:

   exit

   After returning to the Cloud Shell terminal, you can view the debugging result or log file.

Live Comparison

Once a service is deployed, you can perform real-time inference comparisons with other services. This feature is currently only available for real-time text generation services. Live comparison can only be initiated when the real-time service status is Running, Alarm, or Upgrading.

1. On the ModelArts console, choose Model Inference > Real-Time Inference. On the displayed page, click Live Comparison in the Operation column on the right.
   Figure 7 Live comparison
   

2. Evaluate the Q&A service performance.

   To compare the performance of multiple models, click Service Comparison in the upper right corner. Select the target services and click OK. Currently, only text generation services are supported. Only deployed services can be selected for comparison.

   Enter a prompt to compare the responses generated by the models.

For more information about live comparison, see Live Comparison.