启动推理服务（Embedding&Rerank）

本章节主要介绍大语言模型的推理服务Embedding&Rerank启动方式，包括离线推理和在线推理2种方式。

无论是离线推理还是在线推理方式，以下的框架通用环境变量都需要设置。

框架通用环境变量如下：

# VPC网段
# 需用户手动修改，修改方式见下方注意事项
VPC_CIDR="7.150.0.0/16"  #7.150.0.0是Snt9b23的网段，Snt9b机型需要替换为7.242.0.0
VPC_PREFIX=$(echo "$VPC_CIDR" | cut -d'/' -f1 | cut -d'.' -f1-2)
POD_INET_IP=$(ifconfig | grep -oP "(?<=inet\s)$VPC_PREFIX\.\d+\.\d+" | head -n 1)
POD_NETWORK_IFNAME=$(ifconfig | grep -B 1 "$POD_INET_IP" | head -n 1 | awk '{print $1}' | sed 's/://')
echo "POD_INET_IP: $POD_INET_IP"
echo "POD_NETWORK_IFNAME: $POD_NETWORK_IFNAME" 
# 指定通信网卡
export GLOO_SOCKET_IFNAME=$POD_NETWORK_IFNAME
export TP_SOCKET_IFNAME=$POD_NETWORK_IFNAME
export HCCL_SOCKET_IFNAME=$POD_NETWORK_IFNAME
# 多机场景下配置
export RAY_EXPERIMENTAL_NOSET_ASCEND_RT_VISIBLE_DEVICES=1

# 开启显存优化
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
# 配置通信算法的编排展开位置在Device侧的AI Vector Core计算单元
export HCCL_OP_EXPANSION_MODE=AIV
# 指定可使用的卡，按需指定
export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3 
# 指定绑核，按需指定
export CPU_AFFINITY_CONF=1
export LD_PRELOAD=/usr/local/lib/libjemalloc.so.2:${LD_PRELOAD}
# 默认启用ACLGRAPH模式，指定启动插件ascend
export VLLM_PLUGINS=ascend_vllm
# 是否是使用 ACLGRAPH模式，如果启用ACLGRAPH模式，值为1；否则为0
export USE_ACLGRAPH=0
# 指定vllm后端 v0
export VLLM_USE_V1=0                                                                                                                             
# 指定vllm版本
export VLLM_VERSION=0.9.0

from vllm import LLM, SamplingParams

def main():
   # 用户可以改
    prompts = [
        "Hello, my name is",
        "The president of the United States is",
        "The capital of France is",
        "The future of AI is",
    ]

    # ${model_path}用户必须改，具体参考【启动在线推理服务】章节推理服务基础参数说明--model
    model_path = "${model_path}"
    llm = LLM(
        model=model_path,
        # tensor_parallel_size用户根据实际情况可以改，具体参考【启动在线推理服务】章节推理服务基础参数说明--tensor-parallel-size
        tensor_parallel_size=2,
        # block_size用户根据实际情况可以改，具体参考【启动在线推理服务】章节推理服务基础参数说明--block-size
        block_size=128,
        # max_num_seqs用户根据实际情况可以改，具体参考【启动在线推理服务】章节推理服务基础参数说明--max-num-seqs
        max_num_seqs=256,
        # max_model_len用户根据实际情况可以改，具体参考【启动在线推理服务】章节推理服务基础参数说明--max-model-len
        max_model_len=8192,
        # distributed_executor_backend用户根据实际情况可以改，具体参考【启动在线推理服务】章节推理服务基础参数说明--distributed-executor-backend
        distributed_executor_backend='ray'
    )


    outputs = llm.encode(prompts)

    # Print the outputs.
    for output in outputs:
        print(output.outputs.embedding)

if __name__=="__main__":
    main()

脚本中，未标注释部分，请勿更改。

参数说明：

${model_path}：替换成能访问的实际环境模型权重路径。

执行如下命令运行推理脚本：

python embeddings_offline_inference.py

编辑一个rerank python脚本rerank_offline_inference.py，脚本内容如下，运行该脚本使用ascend-vllm进行模型离线推理。

from vllm import LLM
# ${model_path}用户必须改，具体参考【启动在线推理服务】章节推理服务基础参数说明--model
path = "${model_path}"
def get_llm() -> LLM:
    """Initializes and returns the LLM model for Qwen3-Reranker."""
    return LLM(
        model=path,
        task="score",
        hf_overrides={
            "architectures": ["Qwen3ForSequenceClassification"],
            "classifier_from_token": ["no", "yes"],
            "is_original_qwen3_reranker": True,
        },
tensor_parallel_size=1
    )
prefix = '<|im_start|>system\nJudge whether the Document meets the requirements based on the Query and the Instruct provided. Note that the answer can only be "yes" or "no".<|im_end|>\n<|im_start|>user\n'
suffix = "<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n"
query_template = "{prefix}<Instruct>: {instruction}\n<Query>: {query}\n"
document_template = "<Document>: {doc}{suffix}"
def main() -> None:
    instruction = (
        "Given a web search query, retrieve relevant passages that answer the query"
    )
    queries = [
        "What is the capital of Beijing?",
    ]
    documents = [
        "The capital of China is Beijing.",
        "Gravity is a force that attracts two bodies towards each other. It gives weight to physical objects and is responsible for the movement of planets around the sun.",
    ]
    queries = [
        query_template.format(prefix=prefix, instruction=instruction, query=query)
        for query in queries
    ]
    documents = [document_template.format(doc=doc, suffix=suffix) for doc in documents]
    llm = get_llm()
    outputs = llm.score(queries, documents)
    print("-" * 30)
    print([output.outputs.score for output in outputs])
    print("-" * 30)
if __name__ == "__main__":
    main()

脚本中，未标注释部分，请勿更改。

参数说明：

${model_path}：替换成能访问的实际环境模型权重路径。

执行如下命令运行推理脚本：

python rerank_offline_inference.py

推荐通过OpenAI服务的API接口启动推理，单机单卡和单机多卡场景下的具体操作命令如下，可以根据参数说明修改配置。

source /home/ma-user/AscendCloud/AscendTurbo/set_env.bash

python -m vllm.entrypoints.openai.api_server \
--model /model/Qwen3-Reranker-8B \
--served-model-name Qwen3-Reranker-8B \
--max-num-seqs=256 \
--max-model-len=32768 \
--max-num-batched-tokens=32768 \
--tensor-parallel-size=1 \
--block-size=128 \
--host=0.0.0.0 \
--port=9001 \
--gpu-memory-utilization=0.95 \
--disable-log-requests \
--trust-remote-code

# 只有 Qwen3 rerank 需要设置 hf_overrides
# --hf_overrides '{"architectures": ["Qwen3ForSequenceClassification"],"classifier_from_token": ["no", "yes"],"is_original_qwen3_reranker": true}'

rerank接口示例如下:

curl -X POST http://${docker_ip}:9001/v1/rerank \
    -H "Content-Type: application/json" \
    -d '{
        "model": "/container_model/qwen3-reranker-8b",
        "query": "What is the capital of France?",
        "documents": [
            "The capital of France is Paris",
            "Reranking is fun!",
            "vLLM is an open-source framework for fast AI serving"
        ]
    }'

使用OpenAI启动服务（仅支持V0启动），embeddings接口示例如下:

curl -X POST http://${docker_ip}:9001/v1/embeddings \
    -H "Content-Type: application/json" \
    -d '{
        "model": "/container_model/qwen3-embedding-8b",
        "input":"I love shanghai"
    }'