Streaming Output

Streaming output refers to the process in which a large model immediately pushes each generated segment to the caller as it is produced, rather than waiting until the entire response is complete. It reduces the time to first token (TTFT), allowing users to see responses sooner and experience smoother interactions.

The process works as follows:

1. The caller initiates a request with the stream field set to True, enabling the streaming mode.
2. The model generates the response token by token.
3. Each segment is immediately pushed to the caller via Server-Sent Events (SSE) or other methods.
4. Once all content has been generated, an end-flag signal is sent.

Getting Started

The following uses the GLM-5.2 model as an example to demonstrate the streaming output in non-thinking mode.

In thinking mode, the thinking process is also output in streaming mode. In the thinking phase, the reasoning_content field contains the content that the model is thinking about, and the content field is empty. After the thinking is complete, the content field is populated with a value.

import requests
import json
if __name__ == '__main__':
    url = "https://api-ap-southeast-1.modelarts-maas.com/v2/chat/completions"  # API URL
    api_key = "MAAS_API_KEY"  # Replace MAAS_API_KEY with the obtained API key.
    # Send request.
    headers = {
        'Content-Type': 'application/json',
        'Authorization': f'Bearer {api_key}'
    }
    data = {
        "model": "glm-5.2", # Model
         "messages": [
             {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": "Hello"}
         ],
        "thinking": {
            "type": "disabled"
         },
         "stream": True
     }
    response = requests.post(url, headers=headers, data=json.dumps(data), verify=False)
    # Print result.
    print(response.status_code)
    print(response.text)

curl -X POST "https://api-ap-southeast-1.modelarts-maas.com/v2/chat/completions" \   
-H "Content-Type: application/json" \   
-H "Authorization: Bearer $MAAS_API_KEY" \   
-d '{     
    "model": "glm-5.2",     
    "messages": [       
        {"role": "system", "content": "You are a helpful assistant."},     
        {"role": "user", "content": "Hello"}
    ],
    "thinking": {
        "type": "disabled"
     },
    "stream": true
}'

from openai import OpenAI
import httpx

base_url = "https://api-ap-southeast-1.modelarts-maas.com/openai/v1/chat/completions"  # API URL
api_key = "MAAS_API_KEY"  # Replace MAAS_API_KEY with the obtained API key.

client = OpenAI(api_key=api_key, base_url=base_url, http_client=httpx.Client(verify=False))
response = client.chat.completions.create(
    model="glm-5.2",
    messages=[
        {"role": "system", "content": "You are a helpful assistant"},
        {"role": "user", "content": "Hello"},
    ],
    chat_template_kwargs = {
        "thinking": False
    },
    stream=True
)
for chunk in response:
    if not chunk.choices:
        continue

    print(chunk.choices[0].delta.content, end="")

Streaming output:

Data: {"id":"4dc9cd5407fa4fcf92e5d547984819fd","object":"chat.completion.chunk","created":1784770930,"model":"glm-5.2","choices":[{"index":0,"delta":{"role":"assistant","content":"Hello!"}}],"service_tier":"default","first_token_return_time":1784770931.0727186}

data: {"id":"4dc9cd5407fa4fcf92e5d547984819fd","object":"chat.completion.chunk","created":1784770930,"model":"glm-5.2","choices":[{"index":0,"delta":{"role":"assistant","content":"Is there anything"}}],"service_tier":"default","first_token_return_time":1784770931.0729105}

data: {"id":"4dc9cd5407fa4fcf92e5d547984819fd","object":"chat.completion.chunk","created":1784770930,"model":"glm-5.2","choices":[{"index":0,"delta":{"role":"assistant","content":"I can help you with"}}],"service_tier":"default","first_token_return_time":1784770931.1312485}

data: {"id":"4dc9cd5407fa4fcf92e5d547984819fd","object":"chat.completion.chunk","created":1784770930,"model":"glm-5.2","choices":[{"index":0,"delta":{"role":"assistant","content":"?"}}],"service_tier":"default","first_token_return_time":1784770931.1887114}

data: {"id":"4dc9cd5407fa4fcf92e5d547984819fd","object":"chat.completion.chunk","created":1784770930,"model":"glm-5.2","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":"stop"}],"service_tier":"default","first_token_return_time":1784770931.1887114}

data: {"id":"4dc9cd5407fa4fcf92e5d547984819fd","object":"chat.completion.chunk","created":1784770930,"model":"glm-5.2","choices":[],"usage":{"prompt_tokens":14,"total_tokens":23,"completion_tokens":9,"prompt_tokens_details":{"cached_tokens":0},"completion_tokens_details":{"reasoning_tokens":0}},"service_tier":"default","first_token_return_time":1784770931.188748}

data: [DONE]