Help Center/ Cloud Container Engine/ Best Practices/ Cloud Native AI/ Deploying ModelServing Using Mooncake

Updated on 2026-06-17 GMT+08:00

Deploying ModelServing Using Mooncake

This section deploys the PD disaggregation architecture (1 Prefill + 1 Decode) of the DeepSeek-R1-Distill-Qwen-1.5B model using the vLLM-ascend and Kthena inference platforms in an Ascend A3 cluster (single SuperPoD). This architecture physically isolates the Prefill and Decode phases and uses Mooncake Connector for KV cache transfer, significantly optimizing resource utilization and inference performance.

Background

Compared with traditional deployment modes, PD disaggregation (Mooncake and KV cache) has the following advantages:

Physical isolation: Prefill and Decode run on separate nodes, so there is no interference between them.
Reduced latency and jitter: Long prompt processing will not block the token generation for other requests.
High concurrency: Efficient KV-cache transfer optimizes resource scheduling.

The core components are described as follows:

Component	Description
vLLM-ascend	Ascend NPU-optimized vLLM
Kthena	Huawei Cloud model service orchestration platform, used to centrally manage ModelServing instances
Mooncake Connector	Used for efficient KV-cache transfer, enabling efficient sharing between Prefill and Decode nodes.

Flowchart

Click to enlarge

Prerequisites

Volcano Scheduler v1.20.15 or later has been installed, and Volcano has been set as the default scheduler.
Before the deployment, ensure that the network between nodes is normal. For details, see Verification Process.

Constraints

This verification process uses bare metal servers (BMSs). Running on VMs does not guarantee normal NPU network communication. You need to solve related problems by yourself.

Procedure

Prepare a model.
1. Download the model locally and place it in the /models/DeepSeek-R1-Distill-Qwen-1.5B directory on the node.
2. Decompress the model to the specified path.
```
unzip <downloaded-model-file> -d /models
```

Create a ConfigMap.

Create a config.yaml file to define the Prefill and Decode startup scripts. Change the startup script parameters based on the selected model. For details, see the official vLLM Ascend documentation.

kind: ConfigMap
apiVersion: v1
metadata:
  name: deepseek-pd-cm
data:
  prefill.sh: |
    nic_name="enp23s0f3"  # network card name
    local_ip=$POD_IP
    export HCCL_IF_IP=$local_ip
    export GLOO_SOCKET_IFNAME=$nic_name
    export TP_SOCKET_IFNAME=$nic_name
    export HCCL_SOCKET_IFNAME=$nic_name
    export OMP_PROC_BIND=false
    export OMP_NUM_THREADS=10
    export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
    export HCCL_BUFFSIZE=256
    export TASK_QUEUE_ENABLE=1
    export HCCL_OP_EXPANSION_MODE="AIV"
    export VLLM_USE_V1=1
    export MOONCAKE_ENGINE_ID="${GROUP_NAME}_${ROLE_ID}"
    vllm serve $MODEL_LOCATION \
      --host $POD_IP \
      --port "7100" \
      --data-parallel-size 4 \
      --data-parallel-size-local 4 \
      --data-parallel-address $POD_IP \
      --data-parallel-rpc-port 12321 \
      --tensor-parallel-size 2 \
      --seed 1024 \
      --served-model-name ds_r1 \
      --max-model-len 40000 \
      --max-num-batched-tokens 16384 \
      --max-num-seqs 8 \
      --enforce-eager \
      --trust-remote-code \
      --gpu-memory-utilization 0.9  \
      --no-enable-prefix-caching \
      --additional-config '{"recompute_scheduler_enable":true}' \
      --kv-transfer-config \
      '{"kv_connector": "MooncakeConnectorV1",
      "kv_role": "kv_producer",
      "kv_port": "28000",
      "engine_id": "'"${MOONCAKE_ENGINE_ID}"'",
      "kv_connector_module_path": "vllm_ascend.distributed.mooncake_connector",
      "kv_connector_extra_config": {
                "use_ascend_direct": true,
                "prefill": {
                        "dp_size": 4,
                        "tp_size": 2
                },
                "decode": {
                        "dp_size": 4,
                        "tp_size": 2
                }
          }
      }'
  decode.sh: |
    nic_name="enp23s0f3"  # network card name
    local_ip=$POD_IP
    export HCCL_IF_IP=$local_ip
    export GLOO_SOCKET_IFNAME=$nic_name
    export TP_SOCKET_IFNAME=$nic_name
    export HCCL_SOCKET_IFNAME=$nic_name
    export OMP_PROC_BIND=false
    export OMP_NUM_THREADS=10
    export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
    export HCCL_BUFFSIZE=600
    export TASK_QUEUE_ENABLE=1
    export HCCL_OP_EXPANSION_MODE="AIV"
    export VLLM_USE_V1=1
    export MOONCAKE_ENGINE_ID="${GROUP_NAME}_${ROLE_ID}"
    vllm serve $MODEL_LOCATION \
      --host $POD_IP \
      --port "7101" \
      --data-parallel-size 4 \
      --data-parallel-address $POD_IP \
      --data-parallel-rpc-port 12322 \
      --tensor-parallel-size 2 \
      --seed 1024 \
      --served-model-name ds_r1 \
      --max-model-len 40000 \
      --max-num-batched-tokens 256 \
      --max-num-seqs 40 \
      --trust-remote-code \
      --gpu-memory-utilization 0.94  \
      --no-enable-prefix-caching \
      --additional-config '{"recompute_scheduler_enable":true,"finegrained_tp_config": {"lmhead_tensor_parallel_size":2}}' \
      --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}' \
      --kv-transfer-config \
      '{"kv_connector": "MooncakeConnectorV1",
      "kv_role": "kv_consumer",
      "kv_port": "28100",
      "engine_id": "'"${MOONCAKE_ENGINE_ID}"'",
      "kv_connector_module_path": "vllm_ascend.distributed.mooncake_connector",
      "kv_connector_extra_config": {
                "use_ascend_direct": true,
                "prefill": {
                        "dp_size": 4,
                        "tp_size": 2
                },
                "decode": {
                        "dp_size": 4,
                        "tp_size": 2
                }
          }
      }'

ConfigMap contains the following key scripts:

prefill.sh (startup script in the Prefill phase)

# Core configuration
Network interface (nic_name): enp23s0f3 (You can run the ip route | grep default command on the node to obtain the value.)
Service port (port): 7100
KV port (kv_port): 28000
Data parallel size (data-parallel-size): 4
Tensor parallel size (tensor-parallel-size): 2

decode.sh (startup script in the Decode phase)

# Core configuration
Network interface (nic_name): enp23s0f3
Service port (port): 7101
KV port (kv_port): 28100
Maximum number of concurrent sequences (max-num-seqs): 40

Run the following command:
```
kubectl apply -f config.yaml
```

Deploy ModelServing.

Create the deepseek-serv.yaml file and define the Prefill and Decode nodes.

apiVersion: workload.serving.volcano.sh/v1alpha1
kind: ModelServing
metadata:
  name: deepseek-pd
  namespace: default
spec:
  schedulerName: volcano
  replicas: 1
  recoveryPolicy: ServingGroupRecreate
  template:
    restartGracePeriodSeconds: 60
    roles:
    - name: prefill
      replicas: 1
      workerReplicas: 0
      entryTemplate:
        spec:
          hostNetwork: true
          containers:
          - name: prefill
            image: quay.io/ascend/vllm-ascend:v0.13.0-a3
            command:
              - /bin/bash
            args:
              - '-c'
              - cd /workspace && ./prefill.sh
            env:
            - name: ROLE
              value: "prefill"
            - name: GROUP_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.labels['modelserving.volcano.sh/group-name']
            - name: ROLE_ID
              valueFrom:
                fieldRef:
                  fieldPath: metadata.labels['modelserving.volcano.sh/role-id']
            - name: POD_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
            - name: NODE_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.hostIP
            - name: MODEL_LOCATION
              value: /models/DeepSeek-R1-Distill-Qwen-1.5B
            - name: TP_SIZE
              value: "2"
            - name: DP_SIZE
              value: "4"
            readinessProbe:
              httpGet:
                path: /health
                port: 7100
                scheme: HTTP
              initialDelaySeconds: 60
              periodSeconds: 10
              timeoutSeconds: 2
              failureThreshold: 3
            resources:
              limits:
                cpu: '94'
                huawei.com/ascend-1980: '8'
                memory: 900Gi
              requests:
                cpu: '32'
                huawei.com/ascend-1980: '8'
                memory: 350Gi
            ports:
            - containerPort: 7100
              name: server
            volumeMounts:
            - name: model
              mountPath: /models
            - name: dshm
              mountPath: /dev/shm
            - name: hccn-conf
              mountPath: /etc/hccn.conf
            - name: hccn-tool
              mountPath: /usr/local/Ascend/driver/tools/hccn_tool
            - name: ascend-install-info
              mountPath: /etc/ascend_install.info
            - name: config
              mountPath: /workspace/prefill.sh
              subPath: prefill.sh
          volumes:
          - name: model
            hostPath:
              path: /models
              type: Directory
          - name: dshm
            emptyDir:
              medium: Memory
          - name: hccn-conf
            hostPath:
              path: /etc/hccn.conf
          - name: hccn-tool
            hostPath:
              path: /usr/local/Ascend/driver/tools/hccn_tool
          - name: ascend-install-info
            hostPath:
              path: /etc/ascend_install.info
          - name: config
            configMap:
              name: deepseek-pd-cm
              defaultMode: 0777
    - name: decode
      replicas: 1
      workerReplicas: 0
      entryTemplate:
        spec:
          hostNetwork: true
          containers:
          - name: decode
            image: quay.io/ascend/vllm-ascend:v0.13.0-a3
            command:
              - /bin/bash
            args:
              - '-c'
              - cd /workspace && ./decode.sh
            env:
            - name: ROLE
              value: "decode"
            - name: ENGINE_ID
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
            - name: NODE_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.hostIP
            - name: GROUP_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.labels['modelserving.volcano.sh/group-name']
            - name: ROLE_ID
              valueFrom:
                fieldRef:
                  fieldPath: metadata.labels['modelserving.volcano.sh/role-id']
            - name: MODEL_LOCATION
              value: /models/DeepSeek-R1-Distill-Qwen-1.5B
            - name: TP_SIZE
              value: "2"
            - name: DP_SIZE
              value: "4"
            readinessProbe:
              httpGet:
                path: /health
                port: 7101
                scheme: HTTP
              initialDelaySeconds: 60
              periodSeconds: 10
              timeoutSeconds: 2
              failureThreshold: 3
            ports:
            - containerPort: 7101
              name: server
            resources:
              limits:
                cpu: '94'
                huawei.com/ascend-1980: '8'
                memory: 900Gi
              requests:
                cpu: '32'
                huawei.com/ascend-1980: '8'
                memory: 350Gi
            volumeMounts:
            - name: model
              mountPath: /models
            - name: dshm
              mountPath: /dev/shm
            - name: hccn-conf
              mountPath: /etc/hccn.conf
            - name: hccn-tool
              mountPath: /usr/local/Ascend/driver/tools/hccn_tool
            - name: ascend-install-info
              mountPath: /etc/ascend_install.info
            - name: config
              mountPath: /workspace/decode.sh
              subPath: decode.sh
          volumes:
          - name: model
            hostPath:
              path: /models
              type: Directory
          - name: dshm
            emptyDir:
              medium: Memory
          - name: hccn-conf
            hostPath:
              path: /etc/hccn.conf
          - name: hccn-tool
            hostPath:
              path: /usr/local/Ascend/driver/tools/hccn_tool
          - name: ascend-install-info
            hostPath:
              path: /etc/ascend_install.info
          - name: config
            configMap:
              name: deepseek-pd-cm
              defaultMode: 0777

Deploy ModelServing.

kubectl apply -f deepseek-serv.yaml

The following table lists the key mount points.

Mount Point	Description
/models	Model file directory
/dev/shm	Shared memory, which is used for communication across processes
/etc/hccn.conf	NPU network configuration file
/workspace/prefill.sh or /workspace/decode.sh	Path of the startup script

Configure the load balancing proxy.

Download the proxy server script:

wget https://raw.githubusercontent.com/vllm-project/vllm-ascend/main/examples/disaggregated_prefill_v1/load_balance_proxy_server_example.py

Obtain the pod IP addresses:

kubectl get pods -owide

Example response:

NAME                        READY   STATUS    RESTARTS   AGE   IP             NODE           NOMINATED NODE   READINESS GATES
deepseek-pd-0-decode-0-0    1/1     Running   0          20h   192.168.0.25   192.168.0.25   <none>           <none>
deepseek-pd-0-prefill-0-0   1/1     Running   0          20h   192.168.0.25   192.168.0.25   <none>           <none>

Start the proxy server. Change the ports and IP addresses based on the deployment environment.

python3 load_balance_proxy_server_example.py \
  --port 8080 \
  --host 0.0.0.0 \
  --prefiller-hosts 192.168.0.25 \
  --prefiller-ports 7100 \
  --decoder-hosts 192.168.0.25 \
  --decoder-ports 7101

Perform verification test. Send a request through the proxy server port and IP address. Change the port and IP address based on the site requirements.

Send a test request.

curl -X POST http://192.168.0.25:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ds_r1",
    "messages": [
      {
        "role": "user",
        "content": "Hello, how are you?"
      }
    ],
    "max_tokens": 100
  }'

Information similar to the following is displayed:

{
  "id": "chatcmpl-53cf0580-0e68-4623-80aa-1cf0fd923034",
  "object": "chat.completion",
  "created": 1776425897,
  "model": "ds_r1",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Okay, so I just received a message from someone asking, \"Hello, how are you?\" I need to respond appropriately. Let me think about the best way to handle this.\n\nFirst, I should consider the context. The user is greeting me, which is friendly. They're probably new or just reaching out for the first time. I should keep it warm and open-ended to encourage them to share more.\n\nI should acknowledge their greeting and express my greeting in a friendly manner. Maybe something like,",
        "refusal": null,
        "annotations": null,
        "audio": null,
        "function_call": null,
        "tool_calls": [],
        "reasoning": null,
        "reasoning_content": null
      },
      "logprobs": null,
      "finish_reason": "length",
      "stop_reason": null,
      "token_ids": null
    }
  ],
  "service_tier": null,
  "system_fingerprint": null,
  "usage": {
    "prompt_tokens": 11,
    "total_tokens": 111,
    "completion_tokens": 100,
    "prompt_tokens_details": null
  },
  "prompt_logprobs": null,
  "prompt_token_ids": null,
  "kv_transfer_params": null
}

View the log output.

Proxy logs

INFO:     Started server process [417174]
INFO:     Waiting for application startup.
Initialized 1 prefill clients and 1 decode clients.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)
INFO:     192.168.0.25:53050 - "POST /v1/completions HTTP/1.1" 200 OK
INFO:     192.168.0.25:43946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     192.168.0.25:56470 - "POST /v1/completions HTTP/1.1" 200 OK
INFO:     192.168.0.25:36910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     192.168.0.25:51070 - "POST /v1/chat/completions HTTP/1.1" 200 OK

If the HTTP status code in the response is 200 OK, the request is successfully processed.

Prefill pod logs

View the Prefill pod logs:

kubectl logs deepseek-pd-0-prefill-0-0

Key information similar to the following is displayed. Check whether there are logs related to Engine 000 and Delaying free. If yes, the prefill computation is normal, and the KV cache has been generated and is ready for transfer.

(EngineCore_DP0 pid=142) INFO 04-17 11:38:17 [mooncake_connector.py:1062] Delaying free of 1 blocks for request chatcmpl-53cf0580-0e68-4623-80aa-1cf0fd923034
(APIServer pid=7) INFO:     192.168.0.25:52240 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=7) INFO:     192.168.1.191:34968 - "GET /metrics HTTP/1.1" 200 OK
(APIServer pid=7) INFO 04-17 11:38:24 [loggers.py:248] Engine 000: Avg prompt throughput: 1.1 tokens/s, Avg generation throughput: 0.1 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%, External prefix cache hit rate: 0.0%

Decode pod logs

View the Decode pod logs:

kubectl logs deepseek-pd-0-decode-0-0

Key information similar to the following is displayed. You can check the value of Avg generation throughput. If the value is greater than 0, the model is running properly.

(APIServer pid=8) INFO:     192.168.1.191:46710 - "GET /metrics HTTP/1.1" 200 OK
I0417 11:38:17.681255  1302 ascend_direct_transport.cpp:605] Transfer to:192.168.0.25:20294, cost: 5289 us
(Worker_DP0_TP0 pid=304) INFO 04-17 11:38:17 [mooncake_connector.py:561] KV cache transfer for request chatcmpl-53cf0580-0e68-4623-80aa-1cf0fd923034 took 5.85 ms (1 groups, 1 blocks). local_ip 192.168.0.25 local_device_id 0 remote_session_id 192.168.0.25:15910
I0417 11:38:17.700887  1272 ascend_direct_transport.cpp:605] Transfer to:192.168.0.25:21685, cost: 26734 us
(Worker_DP0_TP1 pid=307) INFO 04-17 11:38:17 [mooncake_connector.py:561] KV cache transfer for request chatcmpl-53cf0580-0e68-4623-80aa-1cf0fd923034 took 27.27 ms (1 groups, 1 blocks). local_ip 192.168.0.25 local_device_id 1 remote_session_id 192.168.0.25:16045
(APIServer pid=8) INFO:     192.168.0.25:47876 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=8) INFO:     192.168.0.25:47878 - "GET /health HTTP/1.1" 200 OK
(APIServer pid=8) INFO 04-17 11:38:25 [loggers.py:248] Engine 000: Avg prompt throughput: 1.1 tokens/s, Avg generation throughput: 10.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%, External prefix cache hit rate: 100.0%

Common Issues

Error AttributeError: 'Qwen2Config' object has no attribute 'head_dim'
If there is an error similar to AttributeError: 'Qwen2Config' object has no attribute 'head_dim', the head_dim parameter is missing in the current model configuration. This parameter defines the dimensionality of each attention head, which is a key configuration during model inference.

Manually edit the config.json file in the model directory and add the "head_dim": 128 field. If models of different scales are used, dynamically calculate the value based on the actual parameters. The calculation formula is as follows:
```
head_dim = hidden_size / num_attention_heads 
```

Parent Topic: Cloud Native AI

Previous topic: Building a Cloud-Native Large Model Monitoring Dashboard Based on Mainstream Inference Engines Such as vLLM and SGLang

Next topic: Best Practices for High-Performance EMS KV Cache in CCE Distributed Inference

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

For any further questions, feel free to contact us through the chatbot.

Chatbot