Text Generation

Application Scenarios

You can use the model's text generation capabilities in the following scenarios.

Billing

Inference service deployment is billed based on duration. Costs are incurred when the status is Running or Alarm. Stop the service when not in use. For details, see Inference Deployment Billing Items.

Prerequisites

• The model service to be called has been deployed on ModelArts.
• You have obtained the service URL, API key, and model parameter. For details, see Calling a Preset Model.

API Reference

For the complete parameters for model calls, see Model API Call Guide.

Writing Prompts

Prompt is the text input by users when interacting with LLMs to guide the model in generating the desired response. Properly designed and written prompts can enhance the quality and accuracy of the model's output. The core functions of prompts:

• Define the task: Instruct the model on what to do (for example: translation, summarization, creation, inference, etc.).
• Provide context: Add background information to make the response more relevant and tailored to the request.
• Constraint output format: Specify the structure of the response (such as list, JSON, code, etc.).
• Control style and tone: Adjust the professionalism, conciseness, or creativity of the response.

In the Chat API, the messages object is used to pass prompt information to the model. The role field defines the role of the message sender, while the content field carries the message content. The model interprets the content based on the provided role and generates an appropriate response.

• User Message

  The end user sends messages to the model, at which point the role field should be set to user. This type of message typically contains specific tasks or information that the user wants the model to process.

  The following is a simple user message requesting the model to introduce itself.

  "messages": [
      { "role": "user", "content": "Introduce yourself. "}
  ]

• System Message

  Used to set the model's long-term personality, behavioral guidelines, and conversation context. In this case, the role field should be set to system. If you configure system messages, place them in the first position of the messages list.

  The following is an example of a system message that indicates the model is an assistant and restricts the content of its responses.

  "messages": [
      {"role": "system", "content": " You are a helpful and concise assistant. Keep your responses to no more than two sentences." }
  ]

• Model Message

  Used to provide conversation history, in which case the role field should be set to assistant. In multi-turn conversations, you need to pass in the conversation history, so that the model can remember what it said before and maintain coherence.

  "messages": [
      {"role": "system", "content": "You are a helpful assistant."},             
      {"role": "user", "content": "Hello"},
      {"role": "assistant", "content": "Hello! Nice to meet you! I am your AI assistant and I'm here to help you with anything you need. Whether it's answering questions, assisting with analysis, or just having a chat, I'm ready to assist. Is there anything specific you need help with?"},
      {"role": "user","content": "Introduce yourself" }
  ]

Single-Turn Dialogue

Interact with the model in a single turn of dialogue, where the model returns content based on system and user messages.

Since it is non-streaming output, the model needs to process all content before returning it to you in one go, which may introduce some latency.

The following is an example code for a single-turn conversation. You can replace the model by modifying the model parameter. For details about the model parameter, see Creating a Chat Request.

Multi-Turn Dialogue

Combining system messages, model messages, and user messages enables multi-turn conversations, allowing for multiple dialogues on a single topic.

Note: chat.completions API is stateless. In each request, all historical information is included in the messages field and the role field is set to inform the model of previous conversations from different roles, ensuring contextually relevant and continuous dialogue.

The following is an example code for a multi-turn conversation. You can replace the model by modifying the model parameter. For details about the model parameter, see Creating a Chat Request.

Streaming Output

With the LLM output, dynamic content is displayed as it is generated. This allows users to see intermediate outputs without waiting for the entire inference process to complete, improving the user experience by reducing perceived wait times (viewing content as it is produced).

The following is an example of stream output code. You can replace the model by modifying the model parameter. For details about the model parameter, see Creating a Chat Request.

Function Call

Function calling enables models to dynamically call external APIs by defining tool interface specifications, facilitating seamless function expansion.

The following is an example code using function calling. You can replace the model by modifying the model parameter. For details about the model parameter, see Creating a Chat Request.

Python SDK example:

import requests
import json
def get_weather(location: str, unit: str):
    return f"Getting the weather for {location} in {unit}..."

if __name__ == '__main__':
    url = "https://***/v2/infer/***/v1/chat/completions"  # API address
    api_key = "API_KEY"  # Replace API_KEY with the obtained API key.

    # Send request.
    headers = {
        'Content-Type': 'application/json',
        'Authorization': f'Bearer {api_key}'
    }
    data = {
        "model": "qwen3_32b",  # Model
        "messages": [
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": "Beijing weather"}
        ],
        "tools": [
            {
                "type": "function",
                "function": {
                    "name": "get_weather",
                    "description": "Get weather information for a specified location",
                    "parameters": {
                        "type": "object",
                        "properties": {
                            "location": {
                                "type": "string",
                                "description": "City and state, e.g., 'San Francisco, CA'"
                            }
                        },
                        "required": [
                            "location"
                        ]
                    }
                }
            }
        ],
        "thinking": {
            "type": "enabled"  # Specifies whether to enable deep thinking. It is disabled by default.
        }
    }
    response = requests.post(url, headers=headers, data=json.dumps(data), verify=False)
    # Print result.
    print(response.status_code)
    print(response.text)

Setting the Model Response Length Limit

To control costs or adjust response length, for example, limiting a response to 500 characters, you can configure the max_tokens field in your request to set a maximum output length.

The following is an example code for setting the model response length limit. You can replace the model parameter with your desired model. For details about the model parameter, see Creating a Chat Request.