Supported Models

**Table 1** Supported LLMs and their weight download addresses
No.	Model	FP16/BF16 Inference Supported	W4A16 Quantization Supported	W8A8 Quantization Supported	kv-cache-int8 Quantization Supported	Ascend_turbo graph Supported	Acl_graph Supported	v0/v1 Backend	Address for Obtaining the Open-Source Weight
1	DeepSeek-R1-Distill-Llama-8B	√	x	x	x	x	x	v1	https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Llama-8B
2	DeepSeek-R1-Distill-Llama-70B	√	x	x	x	x	x	v1	https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Llama-70B
3	DeepSeek-R1-Distill-Qwen-1.5B	√	x	x	x	√	√	v1	https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B
4	DeepSeek-R1-Distill-Qwen-7B	√	x	x	x	√	√	v1	https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-7B
5	DeepSeek-R1-Distill-Qwen-14B	√	x	x	x	√	√	v1	https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-14B
6	DeepSeek-R1-0528-Qwen3-8B	√	x	x	x	√	√	v1	https://huggingface.co/deepseek-ai/DeepSeek-R1-0528-Qwen3-8B
7	glm-4-9b	√	x	x	x	x	x	v1	https://huggingface.co/THUDM/glm-4-9b-chat
8	llama3-8b	√	x	x	x	x	x	v1	https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct
9	llama3-70b	√	x	x	x	x	x	v1	https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct
10	llama3.1-8b	√	x	x	x	x	x	v1	https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct
11	llama3.1-70b	√	x	x	x	x	x	v1	https://huggingface.co/meta-llama/Meta-Llama-3.1-70B-Instruct
12	llama-3.2-1B	√	x	x	x	x	x	v1	https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct
13	llama-3.2-3B	√	x	x	x	x	x	v1	https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct
14	qwen2-0.5b	√	√	√	x	√	√	v1	https://huggingface.co/Qwen/Qwen2-0.5B-Instruct
15	qwen2-1.5b	√	√	√	x	√	√	v1	https://huggingface.co/Qwen/Qwen2-1.5B-Instruct
16	qwen2-7b	√	√	√	x	√	√	v1	https://huggingface.co/Qwen/Qwen2-7B-Instruct
17	qwen2-72b	√	√	√	x	√	√	v1	https://huggingface.co/Qwen/Qwen2-72B-Instruct
18	qwen2.5-0.5b	√	√	√	x	√	√	v1	https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct
19	qwen2.5-1.5b	√	√	√	x	√	√	v1	https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct
20	qwen2.5-3b	√	√	√	x	√	√	v1	https://huggingface.co/Qwen/Qwen2.5-3B-Instruct
21	qwen2.5-7b	√	√	√	x	√	√	v1	https://huggingface.co/Qwen/Qwen2.5-7B-Instruct
22	qwen2.5-14b	√	√	√	x	√	√	v1	https://huggingface.co/Qwen/Qwen2.5-14B-Instruct
23	qwen2.5-32b	√	√	√	x	√	√	v1	https://huggingface.co/Qwen/Qwen2.5-32B-Instruct
24	qwen2.5-72b	√	√	√	x	√	√	v1	https://huggingface.co/Qwen/Qwen2.5-72B-Instruct
25	qwen3-0.6b	√	√	√	x	√	√	v1	https://huggingface.co/Qwen/Qwen3-0.6B
26	qwen3-1.7b	√	√	√	x	√	√	v1	https://huggingface.co/Qwen/Qwen3-1.7B
27	qwen3-4b	√	√	√	x	√	√	v1	https://huggingface.co/Qwen/Qwen3-4B
28	qwen3-8b	√	√	√	x	√	√	v1	https://huggingface.co/Qwen/Qwen3-8B
29	qwen3-14b	√	√	√	x	√	√	v1	https://huggingface.co/Qwen/Qwen3-14B
30	qwen3-30b-a3b	√	x	x	x	√	x	v1	https://huggingface.co/Qwen/Qwen3-30B-A3B
31	qwen3-32b	√	√	√	x	√	√	v1	https://huggingface.co/Qwen/Qwen3-32B
32	qwen3-235b-a22b	√	x	x	x	√	x	v1	https://huggingface.co/Qwen/Qwen3-235B-A22B
33	QwQ-32B	√	x	x	x	√	√	v1	https://huggingface.co/Qwen/QwQ-32B
34	Qwen3-Coder-480B-A35B	√	x	x	x	√	√	v1	https://huggingface.co/Qwen/Qwen3-Coder-480B-A35B-Instruct
35	Qwen3-Embedding-0.6B	√	x	x	x	x	√	v0	https://huggingface.co/Qwen/Qwen3-Embedding-0.6B
36	Qwen3-Embedding-4B	√	x	x	x	x	√	v0	https://huggingface.co/Qwen/Qwen3-Embedding-4B
37	Qwen3-Embedding-8B	√	x	x	x	x	√	v0	https://huggingface.co/Qwen/Qwen3-Embedding-8B
38	Qwen3-Reranker-0.6B	√	x	x	x	x	√	v0	https://huggingface.co/Qwen/Qwen3-Reranker-0.6B
39	Qwen3-Reranker-4B	√	x	x	x	x	√	v0	https://huggingface.co/Qwen/Qwen3-Reranker-4B
40	Qwen3-Reranker-8B	√	x	x	x	x	√	v0	https://huggingface.co/Qwen/Qwen3-Reranker-4B
41	bge-reranker-v2-m3	√	x	x	x	x	√	v0	https://huggingface.co/BAAI/bge-reranker-v2-m3
42	bge-base-en-v1.5	√	x	x	x	x	√	v0	https://huggingface.co/BAAI/bge-base-en-v1.5
43	bge-base-zh-v1.5	√	x	x	x	x	√	v0	https://huggingface.co/BAAI/bge-base-zh-v1.5
44	bge-large-en-v1.5	√	x	x	x	x	√	v0	https://huggingface.co/BAAI/bge-large-en-v1.5
45	bge-large-zh-v1.5	√	x	x	x	x	√	v0	https://huggingface.co/BAAI/bge-large-zh-v1.5
46	bge-m3	√	x	x	x	x	√	v0	https://huggingface.co/BAAI/bge-m3

**Table 2** Supported multimodal models and their weight download addresses
No.	Model	FP16/BF16 Inference Supported	W4A16 Quantization Supported	W8A8 Quantization Supported	W8A16 Quantization Supported	kv-cache-int8 Quantization Supported	Address for Obtaining the Open-Source Weight	Remarks
1	qwen2-vl-2B	√	x	x	x	x	https://huggingface.co/Qwen/Qwen2-VL-2B-Instruct/tree/main	-
2	qwen2-vl-7B	√	x	x	x	x	https://huggingface.co/Qwen/Qwen2-VL-7B-Instruct/tree/main	-
3	qwen2-vl-72B	√	√	x	x	x	https://huggingface.co/Qwen/Qwen2-VL-72B-Instruct/tree/main https://huggingface.co/Qwen/Qwen2-VL-72B-Instruct-AWQ	The awq version only supports the eager mode. --enforce-eager
4	qwen2.5-vl-7B	√	x	x	x	x	https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct/tree/main	-
5	qwen2.5-vl-32B	√	x	x	x	x	https://huggingface.co/Qwen/Qwen2.5-VL-32B-Instruct/tree/main	-
6	qwen2.5-vl-72B	√	√	x	x	x	https://huggingface.co/Qwen/Qwen2.5-VL-72B-Instruct/tree/main https://huggingface.co/Qwen/Qwen2.5-VL-72B-Instruct-AWQ/tree/main	The awq version only supports the eager mode. --enforce-eager
7	internvl2.5-26B	√	x	x	x	x	https://huggingface.co/OpenGVLab/InternVL2_5-26B/tree/main	-
8	internvl2-llama3-76B-awq	√	x	x	x	x	https://huggingface.co/OpenGVLab/InternVL2-Llama3-76B-AWQ/tree/main	The awq version only supports the eager mode. --enforce-eager
9	gemma3-27B	√	x	x	x	x	https://huggingface.co/google/gemma-3-27b-it/tree/main	-
10	internvl3-8B	√	x	x	x	x	https://huggingface.co/OpenGVLab/InternVL3-8B/tree/main	-
11	internvl3-14B	√	x	x	x	x	https://huggingface.co/OpenGVLab/InternVL3-14B/tree/main	-
12	internvl3-38B	√	x	x	x	x	https://huggingface.co/OpenGVLab/InternVL3-38B/tree/main	-
13	internvl3-78B	√	x	x	x	x	https://huggingface.co/OpenGVLab/InternVL3-78B/tree/main	-