Accessing a Real-Time Service Using WebSocket

Context

Traditional HTTP protocols often create and drop connections repeatedly during real-time chats or online gaming, causing delays and using excessive bandwidth. WebSocket is a communication protocol that enables easy two-way data transfer between clients and servers, allowing servers to send data directly to clients. In the WebSocket API, if the initial handshake between the client and the server is successful, a persistent connection will be established between them and data can be transferred bidirectionally. WebSocket is ideal for applications requiring real-time, bidirectional communication, such as online games and real-time chat.

ModelArts uses WebSocket for real-time service deployment. WebSocket only supports real-time services and, when used on ModelArts, the protocol is converted to WebSocket Secure (WSS), which supports one-way authentication.

Key features of WebSocket include:

• Persistent connection: Unlike HTTP, WebSocket maintains a persistent connection between the client and server, reducing the overhead of frequent connection establishment.
• Bidirectional communication: Data can flow in both directions, allowing the server to proactively send data to the client, not just respond to client requests.
• Reduced latency and bandwidth: Real-time data transmission is possible because the connection remains open, minimizing latency and reducing unnecessary HTTP requests and bandwidth usage.

Prerequisites

• The service protocol in the network configuration is set to WSS or WS during service information configuration.
• The image for importing the model is WebSocket-compliant.

Constraints

• WebSocket only supports the deployment of real-time services.
• When a server response exceeds 64 KB, the system splits it into smaller frames as per the protocol rules. Each frame must stay under 64 KB.
• When you call an API to access a real-time service, the size of the prediction request body and the prediction time are subject to the following limitations:
  • The request body must be within the allowed size set by the service's configuration; otherwise, the request will be blocked.
  • Each request must be completed within its configured timeout period, which is recalculated during data transmission.

Obtaining the Authentication Information, Local Path to the Prediction File, and URL of the Real-Time Service

• Authentication Information

  WebSocket itself does not require additional authentication. WSS supports only one-way authentication, from the client to the server.

  Obtain authentication information based on the authentication mode selected during service deployment. Authentication information is not required if no authentication is used.

  • Token authentication
  • API key authentication
  • No authentication

  This section uses API key authentication as an example. Before calling a service, create an API key and bind the API key to the real-time service to be accessed. Open the CSV file automatically downloaded after the API key is created. The api_key field in the file is the API key.

• Local Path to the Prediction File

  The local path to the prediction file can be an absolute path (for example, D:/test.png for Windows and /opt/data/test.png for Linux) or a relative path (for example, ./test.png).

• URL of the Real-Time Service

  The API URL and input parameters of the real-time service: To obtain them, choose Model Inference > Real-Time Inference on the console, click the target real-time service, and obtain the information from Call Info in the Basic Information tab.

  API URL is the URL of the real-time service. If a path is defined for apis in the model configuration file, the URL must be followed by the user-defined path, for example, {URL of the real-time service}/v1/chat/completions.

Accessing a Real-Time Service Through Authentication

WebSocket itself does not require additional authentication. WSS only supports one-way authentication, from the client to the server.

Call the real-time service based on the authentication mode selected during service deployment.

• Accessing a Real-Time Service Through IAM Token-based Authentication
• Accessing a Real-Time Service with No Authentication
• Accessing a Real-Time Service Through API Key Authentication

Creating a WebSocket Connection

1. Open Postman of a version later than 8.5, for example, 10.12.0. Click in the upper left corner and choose File > New. In the displayed dialog box, select WebSocket Request (beta version currently).
   Figure 1 WebSocket Request
   
2. Configure parameters for the WebSocket connection.

   Select Raw in the upper left corner. Do not select Socket.IO (a type of WebSocket implementation, which requires that both the client and the server run on Socket.IO). In the address box, enter the API URL obtained from Call Info in the Service tab on the service details page.

   If there is a finer-grained URL in the custom image, add the URL to the end of the address. If queryString is available, add this parameter in the params column.

   Add authentication information into the header. The header varies depending on the authentication mode, which is the same as that in the HTTPS-compliant inference service. Take API key authentication as an example: In the Headers tab, set KEY to Authorization and VALUE to Bearer API key content.

3. Click Connect in the upper right corner to establish a WebSocket connection.
   • If the information is correct, CONNECTED will be displayed in the lower right corner.
   • If establishing the connection failed and the status code is 401, check the authentication.
   • If a keyword such as WRONG_VERSION_NUMBER is displayed, check whether the port configured in the custom image is the same as that configured in WebSocket or WSS.

   The following shows an established WebSocket connection.

   Figure 2 Connection established
   
   

   Prioritize verifying the WebSocket service provided by the custom image. Different tools implement varying WebSocket services, which might result in an inability to maintain the connection once established, or the connection could be interrupted after a single request requiring a reconnection. ModelArts solely guarantees that the WebSocket configuration of the custom image remains consistent both before and after deployment on ModelArts (with the exception of differing addresses and authentication methods).

Exchange data between the WebSocket client and the server.

After the connection is established, WebSocket uses TCP for full-duplex communication. The WebSocket client sends data to the server. The implementation types vary depending on the client, and the lib package may also be different for the same language. Different implementation types are not considered here.

Postman accepts various formats like Text, JSON, XML, HTML, or Binary for sending data. For example, type your text into the text box and press Send on the right side to submit the request. The response will appear in the Response area below.

Figure 3 Sending data