Accessing a Real-Time Service Using Server-Sent Events

Context

Server-Sent Events (SSE) is a server push technology enabling a server to push events to a client via an HTTP connection. This technology is usually used to enable a server to unidirectionally push real-time data to a client, for example, a real-time news update or stock prices.

SSE primarily facilitates unidirectional real-time communication from the server to the client, such as streaming LLM responses. In contrast to WebSockets, which provide bidirectional real-time communication, SSE is designed to be more lightweight and simpler to implement.

Key features of SSE include:

• Easy to use: SSE is based on the HTTP protocol and is straightforward to implement. No complex configurations or additional libraries are needed, and data can be pushed in real-time over standard HTTP connections.
• Automatic reconnection: SSE supports automatic reconnection. If the connection is interrupted, the client automatically attempts to reconnect, ensuring continuous data delivery.
• Unidirectional communication: SSE is unidirectional, meaning the server can send events to the client, but the client cannot send data back to the server through the same connection.
• Low resource usage: SSE uses HTTP connections, making it less resource-intensive compared to other real-time communication protocols like WebSocket. This makes SSE ideal for lightweight real-time data push scenarios.

Prerequisites

• The image for importing the model in Configuring Deployment Settings is SSE-compliant.
• The service protocol in the network configuration is set to HTTP or HTTPS during service information configuration.

Constraints

• SSE supports only the deployment of real-time services.
• It supports only real-time services deployed using models imported from custom images.
• When you call an API to access a real-time service, the size of the prediction request body and the prediction time are subject to the following limitations:
  • The request body must be within the allowed size set by the service's configuration; otherwise, the request will be blocked.
  • Each request must be completed within its configured timeout period.

Obtaining the Authentication Information, Local Path to the Prediction File, and URL of the Real-Time Service

• Authentication Information

  The SSE protocol itself does not introduce new authentication mechanisms; it relies on the same methods as HTTP requests.

  Obtain authentication information based on the authentication mode selected during service deployment. Authentication information is not required if no authentication is used.

  • Token authentication
  • API key authentication
  • No authentication

  This section uses token authentication as an example. For details about how to obtain a user token, see Token-based Authentication. The real-time service APIs generated by ModelArts do not support tokens whose scope is domain. Therefore, you need to obtain the token whose scope is project.

• URL of the Real-Time Service

  The API URL and input parameters of the real-time service: To obtain them, choose Model Inference > Real-Time Inference on the console, click the target real-time service, and obtain the information from Call Info in the Service tab.

  API URL is the URL of the real-time service. If a path is defined for apis in the model configuration file, the URL must be followed by the user-defined path, for example, {URL of the real-time service}/v1/chat/completions.

Calling a Real-Time Service via SSE

1. Download Postman and install it, or install the Postman Chrome extension. Alternatively, use other software that can send POST requests. Postman 8.11.1 is recommended.
2. Open Postman and set parameters on the Postman interface.
   • Select a POST task and copy the obtained API URL to the POST text box. Add /v1/sse to the end of the address to access the service via SSE. Add authentication information into the header. The header varies depending on the authentication mode, which is the same as that in the HTTPS-compliant inference service. Take token authentication as an example: On the Headers tab page, set Key to X-Auth-Token and Value to the user token.

     In normal cases, the value of Content-Type in the response header is text/event-stream;charset=UTF-8.

     Figure 1 Parameter settings

     

     Figure 2 Response header Content-Type
     
   • Select raw and JSON (application/json), and enter the request body in the text box below. The format and contents of the request body depend on the model being used. The platform does not process the input data in any way.

     Example request body:

     {
         "model": "test",
         "messages": [
             {
                 "role": "user",
                 "content": " Who are you?"
             }
         ],
         "max_tokens": 100,
         "top_k": -1,
         "top_p": 1,
         "temperature": 0,
         "ignore_eos": false,
         "stream": false
     }

3. After setting the parameters, click send to send the request. The result will be displayed in Response.

   Figure 7 shows an example of prediction result. The returned contents and format depend on the model being used.