Asynchronous Inference

Prerequisites

• You have prepared data as instructed in Preparations.
• Your account is not in arrears to ensure available resources for running services.
• You have configured the inference service information as instructed in Configuring Service Information.

Notes

Resource pools allocate quota for real-time services even when they are Abnormal or Stopped. If the quota is insufficient and no more services can be deployed, delete some abnormal services to release resources.

• Quota calculation:

  The quota stays the same when you deploy real-time services with a dedicated resource pool. It only changes if you create, modify, or delete a resource pool.

• Usage metering:

  Deploying real-time services in a dedicated resource pool is not metered. Only the usage of the dedicated resource pool itself is metered.

• Mounting SFS Turbo:

  Before mounting an SFS Turbo file system to a real-time service, associate the dedicated resource pool network with the file system.

Procedure

1. Log in to the ModelArts console. In the navigation pane, choose Model Inference > Real-Time Inference.
2. In the real-time inference list, click Deploy.
3. On the service deployment page, set the following key parameters. For details about other parameters, see Procedure.

   Table 1 Asynchronous service parameters

   Parameter	Sub-Parameter	Description
   Basic Information	Service Call Mode	Select Asynchronous. The call mode cannot be modified after the service is created.
   	Max Tasks per Service	The maximum number of tasks that can be created for a single service. The value ranges from 0 to 10,000. Tasks in deleted, succeeded, or timeout state do not consume task quota. You can obtain the task status by calling the API for viewing task details.
   Network Settings	Service Protocol	Asynchronous inference services support HTTPS and HTTP. Currently, asynchronous services do not support traffic mirroring.
   	More Settings	Public Network Access: Specifies whether to allow external network access to the real-time service. If this function is enabled, external networks can access the service. View the API URL on the service details page. If this function is disabled, real-time services cannot be accessed over the Internet. Auto-approved Private Network Connection: Specifies whether private network connection to real-time services requires approval. When this function is enabled, private network connection requests from third-party users will be approved automatically. If this function is disabled, approval is required. For details about how to access a service through a private network, see Accessing a Real-Time Service Through a Private Network. Access Control: Once selected, you can specify a whitelist or blacklist for access control. Whitelist: Only user IP addresses from the CIDR blocks configured here are allowed access. You can add up to ten regular expressions. Blacklist: Only user IP addresses from the CIDR blocks configured here not allowed access. You can add up to ten regular expressions.
   High Availability Settings	More settings	Request Size Limit: maximum size of a request for a single service. Value range: 1 to 50. Unit: MB. Request Timeout (s): timeout interval for a service prediction request. The value ranges from 1 to 1200. Unit: second.

4. Click Next to configure service deployment. The following table describes the key parameters. For details about other parameters, see Configuring Deployment Settings.

   Table 2 Asynchronous service deployment parameters

   Parameter	Sub-Parameter	Description
   Resource Settings	Max Concurrent Tasks per Replica	This parameter is mandatory when the Service Call Mode is set to Asynchronous. Maximum number of tasks that can be concurrently processed by a single replica. The value ranges from 1 to 100. As long as a task is not deleted from the replica container, the quota is occupied.

5. Click Next. On the Deploy Real-Time Service > Confirmation page, confirm the configuration information and click Confirm Deployment.

   Deploying a service generally requires a period of time, which may be several minutes or tens of minutes depending on the amount of your data and resources.

   You can go to the real-time service list to check whether the deployment of the real-time service is complete. Once the service status changes from Deploying to Running, the service is deployed.

   

   After a real-time service is deployed, it is started immediately.

Accessing an Asynchronous Real-Time Service

On the ModelArts console, choose Model Inference > Real-Time Inference in the left navigation pane. On the displayed real-time inference list, click Call Service in the Operation column of the target service to view the call information.

Calling an Asynchronous Real-Time Service

After an asynchronous service is deployed, verify its performance using the Call tab on the service details page. The service call tab offers the rest client. You can choose your service's request method and input the prediction path.

In the Params column, you can set path parameters such as task_id, or set the limit, offset, and status parameters for querying the task list.

In the Body tab, set data format to raw to send original text data. Ensure that the data complies with the API specifications and the JSON format is correct.

You can enter header information in Headers, for example, API key authentication information. Replace {API Key} with your own API key. Click Call to send a request.

Deleting the authorization key pair makes the system switch to IAM token authentication automatically.

For details about how to set the body and request header on the inference page, see How Do I Fill in the Request Header and Request Body When a ModelArts Real-Time Service Is Running?

The status of an asynchronous service affects the calling of task APIs.

Using Cloud Shell to Debug a Real-Time Service Instance Container

You can use Cloud Shell provided by the ModelArts console to log in to the instance container of a running real-time service.

Cloud Shell can only access a container when the associated real-time service is deployed within a dedicated resource pool.

1. Log in to the ModelArts console. In the navigation pane on the left, choose Model Inference > Real-Time Inference.
2. Click the real-time service name or ID to access its details page.
3. In the Cloud Shell tab, select the deployment, instance, and Pod. If the connection status changes to Connection Succeeded, you have logged in to the instance container.
   If the server disconnects due to an error or remains idle for 10 minutes, you can select Reconnect to regain access to the Pod.

   Figure 1 Cloud Shell
   

   If you encounter a path display issue when logging in to Cloud Shell, press Enter to resolve the problem.

   Figure 2 Path display issue
   
4. After logging in to the container, execute the necessary debugging commands in its terminal. Example:
   

   The following is for reference only. The actual log paths and service health check methods depend on your service configuration. Refer to your image settings and container startup commands for details.

   View logs:

   tail -f /var/log/app.log

   Check the service status:

   systemctl status app

   Run a custom script:

   ./debug_script.sh

5. After the debugging, exit the container:

   exit

   After returning to the Cloud Shell terminal, you can view the debugging result or log file.

Follow-Up Operations

• For details about how to view real-time service details, see Viewing Details About a Real-Time Service.
• For details about how to modify, stop, and delete a real-time service, see Managing the Lifecycle of a Real-Time Service Deployment.