Testing a Real-Time Service

Accessing a Synchronous Real-Time Service

On the ModelArts console, choose Model Inference > Real-Time Inference in the left navigation pane. On the displayed real-time inference list, click Call Service in the Operation column of the target service to view the call information.

Alternatively, click the target service name to go to the service details page and obtain the call information in Network Settings.

Using the Prediction Feature

After a service is deployed, verify its performance using the Prediction tab on the service details page. The service prediction tab offers the rest client. You can choose your service's request method and input the prediction path.

In the Body section, choose the appropriate data format (raw, binary, stream, or fromData).

• raw: Used to send raw text data, such as JSON, XML, or plaintext.
• binary: Used to upload binary files, such as images, audios, videos, or model files.
• stream: Block-based stream transmission is supported, which is applicable to real-time or continuous data input scenarios.
• fromData: Submit data in multipart/form-data format. Files can be uploaded and other text fields can be entered at the same time.

You can enter header information in Headers, for example, API key authentication information. Replace {API Key} with your own API key. Click Predict to send a prediction request.

Deleting the authorization key pair makes the system switch to IAM token authentication automatically.

For details about how to set the body and request header on the inference page, see How Do I Fill in the Request Header and Request Body When a ModelArts Real-Time Service Is Running?

Example:

This guide demonstrates deploying the Qwen3-32B model using the Ascend-vLLM framework with one click. Here are the required parameters:

• Request method: POST
• Request path: https://***/v2/infer/***/v1/chat/completions
• Request body:

{
    "model": "qwen3_32b",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Hello"}
    ]
}

Figure 1 Example headers

Using Cloud Shell to Debug a Real-Time Service Instance Container

You can use Cloud Shell provided by the ModelArts console to log in to the instance container of a running real-time service.

Cloud Shell can only access a container when the associated real-time service is deployed within a dedicated resource pool.

1. Log in to the ModelArts console. In the navigation pane on the left, choose Model Inference > Real-Time Inference.
2. Click the real-time service name or ID to access its details page.
3. In the Cloud Shell tab, select the deployment, instance, and Pod. If the connection status changes to Connection Succeeded, you have logged in to the instance container.
   If the server disconnects due to an error or remains idle for 10 minutes, you can select Reconnect to regain access to the Pod.

   Figure 2 Cloud Shell
   

   If you encounter a path display issue when logging in to Cloud Shell, press Enter to resolve the problem.

   Figure 3 Path display issue
   
4. After logging in to the container, execute the necessary debugging commands in its terminal. Example:
   

   The following is for reference only. The actual log paths and service health check methods depend on your service configuration. Refer to your image settings and container startup commands for details.

   View logs:

   tail -f /var/log/app.log

   Check the service status:

   systemctl status app

   Run a custom script:

   ./debug_script.sh

5. After the debugging, exit the container:

   exit

   After returning to the Cloud Shell terminal, you can view the debugging result or log file.

Live Comparison

Once a service is deployed, you can perform real-time inference comparisons with other services. This feature is currently only available for real-time text generation services. Live comparison can only be initiated when the real-time service status is Running, Alarm, or Upgrading.

1. On the ModelArts console, choose Model Inference > Real-Time Inference. On the displayed page, click Live Comparison in the Operation column on the right.
   Figure 4 Live comparison
   

2. Evaluate the Q&A service performance.
   To compare the performance of multiple models, click Service Comparison in the upper right corner. Select the target services and click OK. Currently, only text generation services are supported. Only deployed services can be selected for comparison.

   Figure 5 Live comparison
   

   Enter a prompt to compare the responses generated by the models.

For more information about live comparison, see Live Comparison.