Deploying an AI Application Template

Introduction

AI application templates streamline model deployment via the AI Inference Framework add-on offered by Huawei Cloud CCE.

Traditional deployment of large AI models typically involves complex container configurations, hardware resource (GPU/NPU) scheduling, network gateway settings, and storage mounting. To significantly simplify large model deployment, a one-click inference preset template is now available.

• Preset configurations: The underlying layer includes predefined configurations for mainstream AI models (such as DeepSeek-R1), covering runtime parameters, NPU resource requirements, and hardware adaptation.
• Simplified operations: You can quickly convert a large model into a highly available, low-latency API inference service by simply selecting a template and performing basic configurations through the visualized frontend UI.

Advantages

• Out-of-the-box: No need to edit complex Kubernetes YAML files. Parameters are deeply optimized for Ascend hardware.
• One-click deployment: Complete full lifecycle management, from model loading and resource scheduling to service startup, by simply selecting options on the UI.
• High concurrency and low latency: The underlying layer automatically integrates with high-performance inference engines such as vLLM to provide production-grade inference capabilities.

Prerequisites

Before deployment, ensure the following requirements are met:

• Cluster: A CCE standard or Turbo cluster of v1.28 or later is available.
• Dependent add-on: Volcano Scheduler v1.21.7 or later is installed in the cluster.
• Network access: Inference nodes must have Internet access to pull images and models. For details, see Configuring Internet Access.

Procedure

1. Go to the AI Application Templates tab.
   1. Log in to the CCE console.
   2. In the navigation pane, choose AI Containers. Then, click the AI Application Templates tab.

2. On the AI Application Templates tab page, select the template to be deployed (for example, DeepSeek-R1-Distill-Qwen-1.5B used in this document) and click Deploy.
3. On the Deploy Application page, configure parameters.

   Table 1 Basic settings

   Parameter	Description
   Application Name	Unique ID of the AI application to be deployed.
   AI Application Template	A predefined application template. The selected template determines the underlying architecture (such as the DeepSeek-R1 model) and engine mode (such as vLLM PD disaggregation). CAUTION: Switching the application template will change or clear associated data settings. Proceed with caution.
   Image	When using a third-party image, ensure the pod can access the Internet. For details, see Using Third-Party Images. The vLLM inference engine image (version 0.13.0) optimized for Huawei Ascend NPUs is pre-installed in the environment.
   Cluster Name	Select the target cluster for service deployment. The cluster must have the Volcano Scheduler and NPU add-ons pre-installed. Otherwise, deployment will fail.
   Namespace	Select the namespace where the inference service will be deployed.
   Pods	Specify the number of inference pods to deploy.

4. Click Submit.

   You can view the workload information on the Inference Workloads tab page.

5. Configure a Service to expose the deployed model. This example uses NodePort access. In the selector, set modelserving.volcano.sh/name to the inference workload name and modelserving.volcano.sh/role to proxy. Set both the container port and the service port to 8181. For details, see NodePort.

   

6. Verify the model by sending a standard POST request to the port exposed by the Service.

   curl -X POST http://<node-IP>:<node-port>/v1/chat/completions \
      -H "Content-Type: application/json" \
      -d '{
        "model": "ds_r1",
        "messages": [
          {
            "role": "user",
            "content": "Hello, how are you?"
          }
        ],
        "max_tokens": 100
      }'

   If the model is running correctly, the response returns JSON data containing choices and message fields, where the content field holds the model's generated reply.

   {
     "id": "chatcmpl-753ceb4c-7aa5-4a4f-94b5-dc791c*****",
     "object": "chat.completion",
     "created": 1779936213,
     "model": "ds_r1",
     "choices": [
       {
         "index": 0,
         "message": {
           "role": "assistant",
           "content": "Alright, someone just said \"Hello, how are you?\" I should respond in a friendly and approachable way.\n\nI need to keep it simple and open-ended to encourage them to share more.\n\nMaybe ask them how they're doing or if they have any questions they want to discuss.\n\nThat should make the conversation feel natural and helpful.\n</think>\n\nHello! I'm just a computer program, so I don't have feelings, but thanks for asking! How can I assist you today?",
           "refusal": null,
           "annotations": null,
           "audio": null,
           "function_call": null,
           "tool_calls": [],
           "reasoning": null,
           "reasoning_content": null
         },
         "logprobs": null,
         "finish_reason": "stop",
         "stop_reason": null,
         "token_ids": null
       }
     ],
     "service_tier": null,
     "system_fingerprint": null,
     "usage": {
       "prompt_tokens": 11,
       "total_tokens": 109,
       "completion_tokens": 98,
       "prompt_tokens_details": null
     },
     "prompt_logprobs": null,
     "prompt_token_ids": null,
     "kv_transfer_params": null
   }