Image Understanding
In the field of multimedia content processing, users often need to analyze and understand visual information in images or videos. However, traditional methods typically require complex image processing techniques and algorithms, which not only increases development costs but also raises technical barriers. Some large models possess visual understanding capabilities; for example, when you input an image or video, these models can interpret the visual information and use it to perform tasks such as describing objects within them. How can these large models simplify the workflow for processing multimedia content?
Through this tutorial, you will learn how to call the large model API to recognize information in input images and videos, thereby reducing development costs and technical barriers. The image understanding model supports single or multiple image inputs and is suitable for tasks such as image description, visual Q&A, and object localization. It can be used for automated video content moderation, intelligent monitoring and analysis, and more, significantly reducing manual labor costs. This model is applicable to fields like smart security, sports event analysis, and media content management.
Prerequisites
- You have subscribed to the built-in service on the Model Inference > Real-Time Inference > Built-in Services tab. For details, see Subscribing to a Built-in Service.
- (Optional) To control the service call traffic, you can create a custom endpoint in advance. For details, see Creating an Endpoint on MaaS.
- The API key has been obtained. For details, see Managing API Keys in MaaS.
Supported Models
| Model Parameter | Constraints | Supported API |
|---|---|---|
| deepseek-v4.1-flash | Context length: 1M Max input length: 1M Max output length: 384K Max CoT length: 96K Image restrictions: A maximum of 600 images are allowed. The size of a single image cannot exceed 32 MB. The image resolution is not limited (recommended resolution: less than 4096 x 2160). The JPEG, PNG, GIF, and WebP formats are supported. Upload method: Base64-encoded format or a publicly accessible image URL | V2 Chat API. For details, see MaaS Standard API V2. OpenAI-compatible API. For details, see OpenAI-compatible APIs. Anthropic-compatible API. For details, see Anthropic-compatible APIs. |
Image Format Description
| Image Format | Common Extension | MIME Type |
|---|---|---|
| JPEG | .jpe, .jpeg, .jpg | image/jpeg |
| PNG | .png | image/png |
| WEBP | .webp | image/webp |
| GIF | .gif | image/gif |
Effect Example
| Input | Output |
|---|---|
| Describe this image.
| This is an outdoor photograph featuring a cat being walked on a leash. The cat is a Dragon Li with black and brown stripes, white fur on its face, and bright amber eyes looking directly at the camera. It is wearing a black and green harness attached to a black leash. The other end of the leash is held by a person's leg on the right side of the frame, wearing denim shorts and black-and-white sneakers. The background is a path covered in withered yellow fallen leaves, with a blurred effect that highlights the cat in the foreground, creating a relaxing autumn outdoor walk atmosphere overall. |
Getting Started
LLMs supporting image understanding now allow you to pass Base64-encoded image content or a public URL in the request. The image information must be provided to the LLM as user-role input data, i.e., "role": "user". The following is an example demonstrating the code using the DeepSeek V4.1-Flash model with the V2 API.
A prompt supports the mixed arrangement of images and text, but the order of images and text may affect the output quality. When the prompt consists of multiple images + one text segment, it is recommended to place the text at the end of the prompt.
import requests
import json
import base64
# Base64-encode the image.
def encode_image(image_path):
with open(image_path, "rb") as image_file:
return base64.b64encode(image_file.read()).decode("utf-8")
base64_image = encode_image("test.png") # Replace it with the actual image path.
if __name__ == '__main__':
url = "https://api-ap-southeast-1.modelarts-maas.com/v2/chat/completions" # API URL
api_key = "MAAS_API_KEY" # Replace MAAS_API_KEY with the obtained API key.
# Send request.
headers = {
'Content-Type': 'application/json',
'Authorization': f'Bearer {api_key}'
}
data = {
"model": "deepseek-v4.1-flash", # Model
"messages": [
{
"role": "user",
"content": [
{
"type": "text",
"text": "Describe the content in the image"
},
{
"type": "image_url",
# Ensure the Base64 encoding matches the image/{format} specified in the Content-Type header listed for supported images. The "f" represents the string formatting method.
# PNG image: f"data:image/png;base64,{base64_image}"
# JPEG image: f"data:image/jpeg;base64,{base64_image}"
# WEBP image: f"data:image/webp;base64,{base64_image}"
"image_url": {
"url": f"data:image/png;base64,{base64_image}"
}
}
]
}
]
}
response = requests.post(url, headers=headers, data=json.dumps(data))
# Print result.
print(response.status_code)
print(response.text) Model response preview:
This image shows a cat walking outdoors. Here is a detailed description: A tabby cat with black stripes, white chest and paws. It has a pair of large, bright yellow eyes, looking intently ahead. The cat is wearing a black harness with green decorative trimming along the edges. At the top of the harness is a metal buckle connected to a black leash.
curl -X POST "https://api-ap-southeast-1.modelarts-maas.com/v2/chat/completions" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $MAAS_API_KEY" \
-d '{
"model": "deepseek-v4.1-flash",
"messages": [
{
"role": "user",
"content": [
{"type": "text", "text": "Describe the content in the image."},
{"type": "image_url", "image_url": {"url": "data:image/png;base64,$BASE64_IMAGE"}}
]
}
]
}' Model response preview:
This image shows a cat walking outdoors. Here is a detailed description: A tabby cat with black stripes, white chest and paws. It has a pair of large, bright yellow eyes, looking intently ahead. The cat is wearing a black harness with green decorative trimming along the edges. At the top of the harness is a metal buckle connected to a black leash.
JDK later than 15 is recommended.
import java.io.IOException;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.file.Files;
import java.nio.file.Path;
import java.time.Duration;
import java.util.Base64;
public class ImageDescriptionExample {
public static void main(String[] args) throws Exception {
// API URI
String apiUrl = "https://api-ap-southeast-1.modelarts-maas.com/v2/chat/completions";
String apiKey = "MAAS_API_KEY"; // Replace MAAS_API_KEY with the obtained API key.
String image_path = "test.png"; // Replace test.png with the actual image path.
// Read the image and convert it to Base64.
String base64Image = encodeImageToBase64(image_path);
String imageDataUrl = "data:image/png;base64," + base64Image;
// Replace it with the name of the model you want to call.
String modelName = "deepseek-v4.1-flash";
// Construct a request body.
String requestBody = String.format(
"""
{
"model": "%s",
"messages": [
{"role": "user",
"content": [
{"type": "text", "text": "Describe the content in the image."},
{
"type": "image_url",
"image_url": {
"url": "%s"
}
}
]}
]
}""", modelName, imageDataUrl);
// Create an HttpClient.
HttpClient client = HttpClient.newBuilder()
.connectTimeout(Duration.ofSeconds(10))
.build();
// Create a request.
HttpRequest request = HttpRequest.newBuilder()
.uri(URI.create(apiUrl))
.header("Content-Type", "application/json")
.header("Authorization", "Bearer " + apiKey)
.POST(HttpRequest.BodyPublishers.ofString(requestBody))
.build();
try {
// Send the request and print the result.
HttpResponse<String> response = client.send(
request, HttpResponse.BodyHandlers.ofString());
System.out.println("HTTP Status: " + response.statusCode());
System.out.println("Response Body:\n" + response.body());
} catch (Exception e) {
e.printStackTrace();
}
}
/**
* Convert the image file into a Base64-encoded string.
*/
private static String encodeImageToBase64(String imagePath) throws IOException {
byte[] imageBytes = Files.readAllBytes(Path.of(imagePath));
return Base64.getEncoder().encodeToString(imageBytes);
}
} Model response preview:
This image shows a cat walking outdoors. Here is a detailed description: A tabby cat with black stripes, white chest and paws. It has a pair of large, bright yellow eyes, looking intently ahead. The cat is wearing a black harness with green decorative trimming along the edges. At the top of the harness is a metal buckle connected to a black leash.
import base64
from openai import OpenAI
base_url = "https://api-ap-southeast-1.modelarts-maas.com/openai/v1" # API URL
api_key = "MAAS_API_KEY" # Replace MAAS_API_KEY with the obtained API key.
# Base64-encode the image.
def encode_image(image_path):
with open(image_path, "rb") as image_file:
return base64.b64encode(image_file.read()).decode("utf-8")
base64_image = encode_image("test.png")
client = OpenAI(api_key=api_key, base_url=base_url)
response = client.chat.completions.create(
model = "deepseek-v4.1-flash", # Model
messages = [
{
"role": "user",
"content": [
{"type": "text", "text": "Describe the content in the image."},
{
"type": "image_url",
# Ensure the Base64-encoded image matches the image/{format} specified in the Content-Type header listed for supported images. The f represents the string formatting method.
# Base64-encoded PNG image: f"data:image/png;base64,{base64_image}"
# Base64-encoded JPEG image: f"data:image/jpeg;base64,{base64_image}"
# Base64-encoded WEBP image: f"data:image/webp;base64,{base64_image}"
# Publicly accessible image URL: "https://example/xxx.png"
"image_url": {
"url": f"data:image/png;base64,{base64_image}"
}
}
]
}
]
)
print(response.choices[0].message.content) Model response preview:
This image shows a cat walking outdoors. Here is a detailed description: A tabby cat with black stripes, white chest and paws. It has a pair of large, bright yellow eyes, looking intently ahead. The cat is wearing a black harness with green decorative trimming along the edges. At the top of the harness is a metal buckle connected to a black leash.
Example of Multi-Image Input Scenario
The API can receive and process multiple image inputs simultaneously. You can either convert these images into Base64-encoded content for input or directly provide multiple public URLs for the images. The model will combine all incoming images and prompts to answer the question.
When passing a Base64-encoded image, the format should follow: data:image/<image_format>;base64,<Base64 encoding>, where:
- Image formats: JPEG, PNG, GIF, etc. For details about the supported image formats, see Image Format Description.
- Base64 encoding: The Base64 encoding of images.
The following is an example code for image understanding with multiple images input. You can replace the model by modifying the model parameter. For details about the model parameter, see Image Understanding.
import requests
import json
import base64
# Base64-encode the image.
def encode_image(image_path):
with open(image_path, "rb") as image_file:
return base64.b64encode(image_file.read()).decode("utf-8")
base64_image = encode_image("test.png")
if __name__ == '__main__':
url = "https://api-ap-southeast-1.modelarts-maas.com/v2/chat/completions" # API URL
api_key = "MAAS_API_KEY" # Replace MAAS_API_KEY with the obtained API key.
# Send request.
headers = {
'Content-Type': 'application/json',
'Authorization': f'Bearer {api_key}'
}
data = {
"model": "deepseek-v4.1-flash", # Model
"messages": [
{
"role": "user",
"content": [
{
"type": "text",
"text": "Describe the content in the image"
},
{
"type": "image_url",
"image_url": {
"url": f"data:image/png;base64,{base64_image}"
}
},
{
"type": "image_url",
"image_url": {
"url":"https://example.com/example.png"
}
}
]
}
]
}
response = requests.post(url, headers=headers, data=json.dumps(data))
# Print result.
print(response.status_code)
print(response.text) import base64
from openai import OpenAI
base_url = "https://api-ap-southeast-1.modelarts-maas.com/openai/v1" # API URL
api_key = "MAAS_API_KEY" # Replace MAAS_API_KEY with the obtained API key.
client = OpenAI(api_key=api_key, base_url=base_url)
response = client.chat.completions.create(
model = "deepseek-v4.1-flash", # Model
messages = [
{
"role": "user",
"content": [
{
"type": "text",
"text": "Describe the content in the image."
},
{
"type": "image_url",
"image_url": {
"url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEU...."}
},
{
"type": "image_url",
"image_url": {
"url": "data:image/png;base64,WmMnpZKr54yMyLu+pZKrIz...."}
}
]
}
]
)
print(response.choices[0].message.content) curl -X POST "https://api-ap-southeast-1.modelarts-maas.com/v2/chat/completions" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $MAAS_API_KEY" \
-d '{
"model": "deepseek-v4.1-flash",
"messages": [
{
"role": "user",
"content": [
{"type": "text", "text": "Describe the content in the image."},
{"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,/9j/4AAQSkZJRgABAgAAA...."}},
{"type": "image_url", "image_url": {"url": "data:image/png;base64,WmMnpZKr54yMyLu+pZKrIz...."}}
]
}
]
}' JDK later than 15 is recommended.
import java.io.IOException;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.file.Files;
import java.nio.file.Path;
import java.time.Duration;
import java.util.Base64;
/**
* Multi-image understanding
*/
public class ImageDescriptionExample {
public static void main(String[] args) throws Exception {
// API URI
String apiUrl = "https://api-ap-southeast-1.modelarts-maas.com/v2/chat/completions";
String apiKey = "MAAS_API_KEY"; // Replace MAAS_API_KEY with the obtained API key.
String image_path1 = "test1.png"; // Replace test1.png with the actual image path.
String image_path2 = "test2.png"; // Replace test2.png with the actual image path.
// Read the image and convert it to Base64.
String base64Image1 = encodeImageToBase64(image_path1);
String base64Image2 = encodeImageToBase64(image_path2);
String imageDataUrl1 = "data:image/png;base64," + base64Image1;
String imageDataUrl2 = "data:image/png;base64," + base64Image2;
// Replace it with the name of the model you want to call.
String modelName = "deepseek-v4.1-flash";
// Construct a request body.
String requestBody = String.format(
"""
{
"model": "%s",
"messages": [
{"role": "user",
"content": [
{"type": "text", "text": "Describe the content in the image."},
{
"type": "image_url",
"image_url": {
"url": "%s"
}
},
{
"type": "image_url",
"image_url": {
"url": "%s"
}
}
]}
]
}""", modelName, imageDataUrl1, imageDataUrl2);
// Create an HttpClient.
HttpClient client = HttpClient.newBuilder()
.connectTimeout(Duration.ofSeconds(10))
.build();
// Create a request.
HttpRequest request = HttpRequest.newBuilder()
.uri(URI.create(apiUrl))
.header("Content-Type", "application/json")
.header("Authorization", "Bearer " + apiKey)
.POST(HttpRequest.BodyPublishers.ofString(requestBody))
.build();
try {
// Send the request and print the result.
HttpResponse<String> response = client.send(
request, HttpResponse.BodyHandlers.ofString());
System.out.println("HTTP Status: " + response.statusCode());
System.out.println("Response Body:\n" + response.body());
} catch (Exception e) {
e.printStackTrace();
}
}
/**
* Convert the image file into a Base64-encoded string.
*/
private static String encodeImageToBase64(String imagePath) throws IOException {
byte[] imageBytes = Files.readAllBytes(Path.of(imagePath));
return Base64.getEncoder().encodeToString(imageBytes);
}
} Example of Mixed Text and Images
The API allows for flexible and interleaved sequence arrangement of prompt and image information. You can adjust the order of images and text as needed, and include them in the system message or user message. The model will return the processed results based on the provided sequence. You can replace the model by modifying the model parameter. For details about the model parameter, see Image Understanding.
In scenarios with mixed text and images, the order of images and text can affect the model's output. If the results do not meet your expectations, try changing the order of the images and text.
Prompt supports the mixed arrangement of images and text, but the order of images and text may affect the output quality, especially when there are multiple images and only one block of text. It is recommended to place the text after the images when concatenating.
import requests
import json
import base64
# Base64-encode the image.
def encode_image(image_path):
with open(image_path, "rb") as image_file:
return base64.b64encode(image_file.read()).decode("utf-8")
base64_image = encode_image("test.png")
if __name__ == '__main__':
url = "https://api-ap-southeast-1.modelarts-maas.com/v2/chat/completions" # API URL
api_key = "MAAS_API_KEY" # Replace MAAS_API_KEY with the obtained API key.
# Send request.
headers = {
'Content-Type': 'application/json',
'Authorization': f'Bearer {api_key}'
}
data = {
"model": "deepseek-v4.1-flash", # Model
"messages": [
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": f"data:image/png;base64,{base64_image}"
}
},
{
"type": "text",
"text": "Text extraction"
},
{
"type": "image_url",
"image_url": {
"url":"https://example.com/example.png"
}
},
{
"type": "text",
"text": "Describe the content in the image"
}
]
}
]
}
response = requests.post(url, headers=headers, data=json.dumps(data))
# Print result.
print(response.status_code)
print(response.text) import base64
from openai import OpenAI
base_url = "https://api-ap-southeast-1.modelarts-maas.com/openai/v1" # API URL
api_key = "MAAS_API_KEY" # Replace MAAS_API_KEY with the obtained API key.
client = OpenAI(api_key=api_key, base_url=base_url)
response = client.chat.completions.create(
model = "deepseek-v4.1-flash", # Model
messages = [
{
"role": "user",
"content": [
{"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,/9j/4AAQSkZJRgABAgAA...."}},
{"type": "text", "text": "Text extraction"}
{"type": "image_url", "image_url": {"url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEU...."}},
{"type": "text", "text": "Describe the content in the image."},
]
}
]
)
print(response.choices[0].message.content) curl -X POST "https://api-ap-southeast-1.modelarts-maas.com/v2/chat/completions" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $MAAS_API_KEY" \
-d '{
"model": "deepseek-v4.1-flash",
"messages": [
{
"role": "user",
"content": [
{"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,/9j/4AAQSkZJRgABAgAA...."}},
{"type": "text", "text": "Text extraction"}
{"type": "image_url", "image_url": {"url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEU...."}},
{"type": "text", "text": "Describe the content in the image."}
]
}
]
}' JDK later than 15 is recommended.
import java.io.IOException;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.file.Files;
import java.nio.file.Path;
import java.time.Duration;
import java.util.Base64;
public class ImageDescriptionExample {
public static void main(String[] args) throws Exception {
// API URI
String apiUrl = "https://api-ap-southeast-1.modelarts-maas.com/v2/chat/completions";
String apiKey = "MAAS_API_KEY"; // Replace MAAS_API_KEY with the obtained API key.
String image_path1 = "test1.png"; // Replace test1.png with the actual image path.
String image_path2 = "test2.png"; // Replace test2.png with the actual image path.
// Read the image and convert it to Base64.
String base64Image1 = encodeImageToBase64(image_path1);
String base64Image2 = encodeImageToBase64(image_path2);
String imageDataUrl1 = "data:image/png;base64," + base64Image1;
String imageDataUrl2 = "data:image/png;base64," + base64Image2;
// Replace it with the name of the model you want to call.
String modelName = "deepseek-v4.1-flash";
// Construct a request body.
String requestBody = String.format(
"""
{
"model": "%s",
"messages": [
{"role": "user",
"content": [
{"type": "text", "text": "Describe the content in the image."},
{
"type": "image_url",
"image_url": {
"url": "%s"
}
},
{"type": "text", "text": "Text extraction"},
{
"type": "image_url",
"image_url": {
"url": "%s"
}
}
]}
]
}""", modelName, imageDataUrl1, imageDataUrl2);
// Create an HttpClient.
HttpClient client = HttpClient.newBuilder()
.connectTimeout(Duration.ofSeconds(10))
.build();
// Create a request.
HttpRequest request = HttpRequest.newBuilder()
.uri(URI.create(apiUrl))
.header("Content-Type", "application/json")
.header("Authorization", "Bearer " + apiKey)
.POST(HttpRequest.BodyPublishers.ofString(requestBody))
.build();
try {
// Send the request and print the result.
HttpResponse<String> response = client.send(
request, HttpResponse.BodyHandlers.ofString());
System.out.println("HTTP Status: " + response.statusCode());
System.out.println("Response Body:\n" + response.body());
} catch (Exception e) {
e.printStackTrace();
}
}
/**
* Convert the image file into a Base64-encoded string.
*/
private static String encodeImageToBase64(String imagePath) throws IOException {
byte[] imageBytes = Files.readAllBytes(Path.of(imagePath));
return Base64.getEncoder().encodeToString(imageBytes);
}
} What is your overall rating for this page?
Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot
