Accessing a Real-Time Service Through a Public Network

Context

ModelArts inference supports access to real-time services through the public network, supporting both HTTPS and WebSocket protocols. A standard, callable RESTful API is provided after deployment of a real-time service.

Test the real-time service's RESTful API before deploying it to production. This section describes how to send a prediction request to a real-time service using API key authentication.

Method 1: Use GUI-based Software for Prediction (Postman). For Windows, Postman is recommended.
Method 2: Send a Prediction Request via cURL. For Linux, cURL commands are recommended.
Method 3: Use Python to Send a Prediction Request.
Method 4: Use Java to Send a Prediction Request.

Prerequisites

To access a service from a public network, select Public Network Access during Procedure.
This section uses API key authentication as an example. During Procedure, select API key authentication. Before calling a service, create an API key and bind the API key to the real-time service to be accessed.
To prevent failed API calls, ensure that the number of concurrent requests, request body size, and request timeout interval do not exceed the limits set during deployment.
Figure 1 Configuring API key authentication for a real-time service

Obtaining Service Call Information

Before calling a service, create an API key and bind the API key to the target real-time service. Obtain the local path of the prediction file, the API URL of the real-time service, and the input parameters of the real-time service.

Obtain the API key content: Open the CSV file automatically downloaded when you create the API key. Find the api_key field to get the API key content.
The local path to the prediction file can be an absolute path (for example, D:/test.png for Windows and /opt/data/test.png for Linux) or a relative path (for example, ./test.png).
The API URL and input parameters of the real-time service: To obtain them, choose Model Inference > Real-Time Inference on the console, click the target real-time service, and obtain the information from Call Info in the Basic Information tab.
API URL is the URL of the real-time service. If a path is defined for apis in the model configuration file, the URL must be followed by the user-defined path, for example, {URL of the real-time service}/v1/chat/completions.

Method 1: Use GUI-based Software for Prediction (Postman)

Download Postman and install it, or install the Postman Chrome extension. Alternatively, use other software that can send POST requests. Postman 8.11.1 is recommended.
Open Postman. Figure 2 shows the Postman interface.
Figure 2 Postman interface
Set parameters on Postman. The following uses image classification as an example.
- Select a POST task and copy the API URL to the POST text box. In the Headers tab, set KEY to Authorization and VALUE to Bearer <API-key-content>.
  Figure 3 Parameter settings
- In the Body tab, file input and text input are available.
  - File input
    Select form-data. Set KEY to the input parameter of the model, which must be the same as the input parameter of the real-time service. In this example, the KEY is images. Set VALUE to an image to be inferred (only one image can be inferred). See Figure 4.
    
    Figure 4 Setting parameters in the Body tab
  - Text input
    Select raw and JSON (application/json), and enter the request body in the text box below. The format and contents of the request body depend on the model being used. The platform does not process the input data in any way.
    
    Figure 5 Text prediction request for a real-time service using IAM authentication
    
    Example request body:
```
{
    "model": "test",
    "messages": [
        {
            "role": "user",
            "content": " Who are you?"
        }
    ],
    "max_tokens": 100,
    "top_k": -1,
    "top_p": 1,
    "temperature": 0,
    "ignore_eos": false,
    "stream": false
}
```
After setting the parameters, click send to send the request. The result will be displayed in Response.
- Prediction result using file input: Figure 6 shows an example. The field values in the return result vary with the model.
- Figure 7 shows an example of prediction result for text input. The returned contents and format depend on the model being used.
  Figure 6 File prediction result
  
  Figure 7 Text prediction result

Method 2: Send a Prediction Request via cURL

The curl command for sending prediction requests can be input as a file or text.

File input
```
curl -kv -F 'images=@ Image path' -H 'Authorization:Bearer API key content' -X POST Real-time service address
```
- -k indicates that SSL websites can be accessed without using a security certificate.
- -F indicates file input. In this example, the parameter name is images, which can be changed as required. The image storage path follows @.
- -H specifies the post command's headers. It includes the Authorization header with a fixed key value. API key content contains the API key details.
- POST is followed by the API URL of the real-time service.
The following is an example of the curl command for prediction with file input:
```
curl -kv -F 'images=@/home/data/test.png' -H 'Authorization:Bearer 4**************w' -X POST https://{{infer-endpoint}}/v2/infers/eb3e0c54-3dfa-4750-af0c-95c45e5d3e83
```

Text input

curl -kv -d '{"data":{"req_data":[{"sepal_length":3,"sepal_width":1,"petal_length":2.2,"petal_width":4}]}}' -H 'Authorization:Bearer 4**************w' -H 'Content-type: application/json' -X POST https://{{infer-endpoint}}/v2/infers/eb3e0c54-3dfa-4750-af0c-95c45e5d3e83

-d indicates the text input of the request body.

Method 3: Use Python to Send a Prediction Request

Download the Python SDK and configure it in the development tool. For details, see Integrating the Python SDK for API request signing.

Create a request body for prediction.

File input

# coding=utf-8

import requests

if __name__ == '__main__':
    # Config url, token and file path.
    url = "Real-time service URL"
    api_key= "API key content"
    file_path = "Local path to the inference file"

    # Send request.
    headers = {
        'Authorization': 'Bearer ' + api_key
    }
    files = {
        'images': open(file_path, 'rb')
    }
    resp = requests.post(url, headers=headers, files=files)

    # Print result.
    print(resp.status_code)
    print(resp.text)

The files name is determined by the input parameter of the real-time service. The parameter name must be the same as that of the input parameter of the file type. The input parameter images obtained in Obtaining Service Call Information is an example.

Text input (JSON)

The following is an example of the request body for reading the local prediction file and performing Base64 encoding:

# coding=utf-8

import base64
import requests

if __name__ == '__main__':
    # Config url, token and file path
    url = "Real-time service URL"
    api_key= "API key content"
    file_path = "Local path to the inference file"
    with open(file_path, "rb") as file:
        base64_data = base64.b64encode(file.read()).decode("utf-8")

    # Set body,then send request
    headers = {
        'Content-Type': 'application/json',
        'Authorization': 'Bearer ' + api_key
    }
    body = {
        'image': base64_data
    }
    resp = requests.post(url, headers=headers, json=body)

    # Print result
    print(resp.status_code)
    print(resp.text)

The body name is determined by the input parameter of the real-time service. The parameter name must be the same as that of the input parameter of the string type. The input parameter images obtained in Obtaining Service Call Information is an example. The value of base64_data in body is of the string type.

Method 4: Use Java to Send a Prediction Request

Download the Java SDK and configure it in the development tool. For details, see Integrating the Java SDK for API request signing.

(Optional) If the inference request input is in a file format, follow these steps to ensure the Java project includes the httpmime module as a dependency.

Add httpmime-x.x.x.jar to the libs folder. Figure 8 shows a complete Java dependency library.
You are advised to use httpmime-x.x.x.jar 4.5 or later. Download httpmime-x.x.x.jar from https://mvnrepository.com/artifact/org.apache.httpcomponents/httpmime.

Figure 8 Java dependency library

After httpmime-x.x.x.jar is added, add httpmime information to the .classpath file of the Java project as follows:

<?xml version="1.0" encoding="UTF-8"?>
<classpath>
<classpathentry kind="con" path="org.eclipse.jdt.launching.JRE_CONTAINER"/>
<classpathentry kind="src" path="src"/>
<classpathentry kind="lib" path="libs/commons-codec-1.11.jar"/>
<classpathentry kind="lib" path="libs/commons-logging-1.2.jar"/>
<classpathentry kind="lib" path="libs/httpclient-4.5.13.jar"/>
<classpathentry kind="lib" path="libs/httpcore-4.4.13.jar"/>
<classpathentry kind="lib" path="libs/httpmime-x.x.x.jar"/>
<classpathentry kind="lib" path="libs/java-sdk-core-3.1.2.jar"/>
<classpathentry kind="lib" path="libs/okhttp-3.14.9.jar"/>
<classpathentry kind="lib" path="libs/okio-1.17.2.jar"/>
<classpathentry kind="output" path="bin"/>
</classpath>

Create a Java request body for prediction.

File input

A sample Java request body is as follows:

// Package name of the demo.
package com.apig.sdk.demo;

import org.apache.http.Consts;
import org.apache.http.HttpEntity;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.entity.ContentType;
import org.apache.http.entity.mime.MultipartEntityBuilder;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.util.EntityUtils;

import java.io.File;

public class MyTokenFile {

    public static void main(String[] args) {
        // Config url, token and filePath
        String url = "Real-time service URL";
        String apiKey = "API key content";
        String filePath = "Local path to the prediction file";

        try {
            // Create post
            HttpPost httpPost = new HttpPost(url);

            // Add header parameters
            httpPost.setHeader("Authorization", "Bearer " + api_key);

            // Add a body if you have specified the PUT or POST method. Special characters, such as the double quotation mark ("), contained in the body must be escaped.
            File file = new File(filePath);
            HttpEntity entity = MultipartEntityBuilder.create().addBinaryBody("images", file).setContentType(ContentType.MULTIPART_FORM_DATA).setCharset(Consts.UTF_8).build();
            httpPost.setEntity(entity);

            // Send post
            CloseableHttpResponse response = HttpClients.createDefault().execute(httpPost);

            // Print result
            System.out.println(response.getStatusLine().getStatusCode());
            System.out.println(EntityUtils.toString(response.getEntity()));
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

The addBinaryBody name is determined by the input parameter of the real-time service. The parameter name must be the same as that of the input parameter of the file type. The file images obtained in Obtaining Service Call Information is used as an example.

Text input (JSON)

The following is an example of the request body for reading the local prediction file and performing Base64 encoding:

// Package name of the demo.
package com.apig.sdk.demo;

import org.apache.http.HttpHeaders;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.entity.StringEntity;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.util.EntityUtils;

public class MyTokenTest {

    public static void main(String[] args) {
        // Config url, token and body
        String url = "Real-time service URL";
        String apiKey = "API key content";
        String body = "{}";

        try {
            // Create post
            HttpPost httpPost = new HttpPost(url);

            // Add header parameters
            httpPost.setHeader(HttpHeaders.CONTENT_TYPE, "application/json");
            httpPost.setHeader("Authorization", "Bearer " + api_key);

            // Special characters, such as the double quotation mark ("), contained in the body must be escaped.
            httpPost.setEntity(new StringEntity(body));

            // Send post.
            CloseableHttpResponse response = HttpClients.createDefault().execute(httpPost);

            // Print result
            System.out.println(response.getStatusLine().getStatusCode());
            System.out.println(EntityUtils.toString(response.getEntity()));
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

body is determined by the text format. JSON is used as an example.