Help Center/ ModelArts/ Getting Started/ Getting Started with Notebook: Deploying the Qwen3-8B Model and Setting Up a Chatbot in One Click

Updated on 2026-11-10 GMT+08:00

Getting Started with Notebook: Deploying the Qwen3-8B Model and Setting Up a Chatbot in One Click

Scenario

For AI developers, traditional local development environments suffer from low efficiency in deploying and testing AI models due to limited resources and incomplete toolchains. For example, when you try to deploy the Qwen3-8B model, the local environment may not provide sufficient computing resources or support efficient model debugging and testing. To improve the development efficiency of AI models without increasing costs, ModelArts development environment leverages cloud-native resource utilization and integrates development toolchains, aiming to provide users with a superior cloud-based AI development experience. This document teaches you how to efficiently deploy Qwen3-8B locally. It uses a vLLM-Ascend inference engine to provide API services on Huawei Ascend NPUs. The process, from loading the model to calling the RESTful API, is shown using JupyterLab. Lastly, the Gradio component creates the chatbot's frontend UI, allowing quick setup.

This case takes about 20 minutes to complete. The time required depends on the model download speed.

Concepts

vLLM-Ascend

vLLM-Ascend is a key hardware adaptation plug-in designed to efficiently introduce the large language model (LLM) framework vLLM into the Huawei Ascend NPU ecosystem. It serves as a bridge between the core logic of vLLM and Huawei NPU hardware. Through decoupling, it enables seamless integration and efficient execution of vLLM's core optimization techniques (such as PagedAttention memory management, continuous batch processing, and fast quantization) on the Ascend platform.

Qwen3-8B

Qwen3-8B is the open-source language model in the Qwen3 series, setting industry benchmarks with its outstanding performance and efficient resource utilization. This model excels in multiple benchmark tests.

Prerequisites

You have registered a Huawei Cloud account, completed real-name authentication, and been granted related permissions. For details, see 1. Prerequisites: Accounts and Permissions.

Constraints

This case is only available in the new console.

Billing

Running notebook instances in ModelArts development environments use compute and storage resources, which are billed. For details about the billing, see Development Environment Billing Item.

Step 1: Quickly Create and Run a Notebook Instance

On the Development Practice tab page, click Develop in the Qwen3-8B Deployment and vllm-ascend-based Chatbot Development card, and open in an existing notebook instance or create a notebook instance as required.

Figure 1 Opening an existing notebook or creating a notebook
Click to enlarge

Opening an existing notebook
If you have created a notebook, click Open. The system displays the notebooks that meet the recommended flavor or image. You can select the target notebook from the notebook list and click Open.

Creating a notebook

If you have not created any notebook or the notebook list does not contain a notebook that meets the requirements of this example, click Create Notebook. On the Create Notebook page, the parameters that meet the requirements of this example are preset. You can view the configuration fee and click OK. In the billing notification dialog box, read the billing information carefully and click OK. The creation takes 2 to 5 minutes.

The following is an example of creating a notebook:

Figure 2 Creating a notebook
Click to enlarge

**Table 1** Parameters for creating a notebook
Parameter	Description	Example Value
Name	The name of the notebook instance. The system automatically generates a default name, which you can modify based on your needs. Naming rules: The name can only contain digits, letters, underscores (_), and hyphens (-). The length cannot exceed 128 characters and cannot be empty.	notebook-2e37
Select Image	Select a ModelArts official image.	Default value
Resource Pool Type	Public resource pools are billed based on the runtime of your notebook instances. Dedicated resource pool: The resources provided in a dedicated resource pool are exclusive and more controllable. Using a dedicated resource pool: Click Dedicated Resource Pool. On the right of Select Resource Pool, click . On the Select Dedicated Resource Pool page, select the target dedicated resource pool and click OK. For details about dedicated resource pool parameters, see Creating a Notebook Instance. If you do not have a dedicated resource pool, click Buy Dedicated Resource Pool in the lower part of the Select Dedicated Resource Pool page to purchase a dedicated resource pool. For details about the parameters for purchasing a dedicated resource pool, see Creating a Standard Dedicated Resource Pool.	Public resource pool
Instance Specifications	*1 ascend-snt9b1 \| 24 vCPUs \| 192 GiB \| ARM (modelarts.bm.npu.arm.1snt9b1)** or higher specifications are recommended.	Default value
Disk Capacity	20 GB or larger is recommended.	20
Auto Stop	Enable this feature to set an auto-stop method and duration. The instance automatically stops once the threshold is exceeded. (There may be a 2–5 minute delay, which is billed normally.)	Enable this feature and set the duration to 1 hour.

(Optional) If the Select Kernel dialog box is displayed, select PyTorch-2.7.1 and click Select.

Step 2: Download the Qwen3-8B Model

Run the following command to install ModelScope:
```
!pip install "modelscope<=1.33.0"
```
On the left of the command, click to run the command.
Run the following command to download the Qwen3-8B model:
```
!modelscope download --model 'Qwen/Qwen3-8B' --local_dir ./Qwen3-8B
```
- The time required depends on the model download speed.
- Running this command repeatedly will perform integrity check and incremental download. If the download times out, run this command again.
After the download is complete, the Qwen3-8B model will appear in the Qwen3-8B folder in the directory on the left.

Figure 3 Qwen3-8B folder

Step 3: Install vLLM and vLLM-Ascend

Perform step 1 first. After step 1 succeeds, perform step 2. If the operation sequence is incorrect, the versions may not match, affecting subsequent operations.

Run the following command to install vLLM:
In this case, the inference version supports vLLM 0.11.0, which works with vLLM-Ascend 0.11.0 for A2 hardware. The --use-deprecated=legacy-resolver option is used to prevent the Torch version from being updated because vLLM-Ascend depends on Torch 2.7.1.
```
!pip install vllm==0.11.0 torch==2.7.1 pycountry==26.2.16 --use-deprecated=legacy-resolver
```
Run the following command to install vLLM-Ascend:
vLLM-Ascend 0.11.0 depends on vLLM 0.11.0.
```
!pip install vllm-ascend==0.11.0
```

Step 4: Start the Inference Service

Set the inference service port, model path, and model name.

import os
# Set the port of the inference service, which can be customized.
os.environ['port'] = '9001'
port = os.environ['port']
# Set the path for storing the model.
os.environ['model_path'] = './Qwen3-8B'
# Model name, which can be customized.
os.environ['model_name'] = 'qwen3'
model_name = os.environ['model_name']

Run the following commands to start the inference service in the background:

%%bash

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export MODEL_PATH=$model_path
export PORT=$port
export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=10
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export HCCL_BUFFSIZE=256
export TASK_QUEUE_ENABLE=1
export HCCL_OP_EXPANSION_MODE="AIV"
export VLLM_USE_V1=1
export LD_LIBRARY_PATH=/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/mooncake:$LD_LIBRARY_PATH

nohup vllm serve ${MODEL_PATH} \
  --port ${PORT} \
  --served-model-name ${model_name} \
  --max-model-len 5500 \
  --max-num-batched-tokens 40960 \
  --gpu-memory-utilization 0.90 \
  --enable-prefix-caching \
  --data-parallel-size 1 \
  --tensor-parallel-size 1 \
  --enforce-eager > vllm_qwen3.log 2>&1 &

Run the following command to view the startup log:
After executing the startup command, run the command below multiple times to check the logs. The service has started successfully when Application startup complete is displayed.
```
%%bash
# View the startup result and run logs of vLLM.
tail -n 100 vllm_qwen3.log
```

Check whether the service is running properly.

Once the inference service has started, run the script to verify its status. You can modify the question text in the content field.

import subprocess
import json

data = {
    "messages": [{"role": "user", "content": "How many days are there in a week?"}],
    "stream": False,
    "max_tokens": 300
}

result = subprocess.run([
    "curl", "-X", "POST",
    f"http://127.0.0.1:{port}/v1/chat/completions",
    "-H", "Content-Type: application/json",
    "-d", json.dumps(data)
], capture_output=True, text=True)

if result.returncode != 0:
    print("Request failed. Error message:")
    print(result.stderr)
else:
    try:
        response = json.loads(result.stdout)
        if "choices" in response and len(response["choices"]) > 0:
            message = response["choices"][0]["message"]["content"]
            print("Model's reply:")
            print(message.strip())
        else:
            print("No valid reply content found. Complete response:")
            print(json.dumps(response, indent=2, ensure_ascii=False))
    except json.JSONDecodeError:
         print("Failed to parse the JSON response. Original output:"))
        print(result.stdout)

Example output:

Model's reply:
<think>
Well, the user asked, "How many days are there in a week?" This seems like a very basic question, but it might require careful consideration. First, I need to confirm what the user's needs are. They might be learning about basic time units, or they might be doing a task that requires knowing the number of days in a week, such as scheduling appointments or planning activities. It's also possible that the user is not very familiar with the time system, or they want to confirm whether the number of days in a week is consistent across different cultures.

Next, I need to recall the standard weekly structure. Generally, a week consists of seven days, which is common in most countries, such as Monday to Sunday. However, there may be variations. For example, in some places, Sunday may be considered the start of the week, while in others, Monday is the start. However, the international standard usually considers Sunday as the end of the week and Monday as the start. This may vary by region, but the number of days remains seven.

Then, I need to consider the user's possible background. If the user is a child, a simpler explanation may be needed, such as using examples to explain the name of each day or using daily activities as examples. If the user is an adult, a direct answer of seven days may suffice, but additional information can also be provided, such as differences in various cultures or the application of a week in different calendars, such as the Gregorian calendar and the lunar calendar.

Additionally, the user may have deeper needs. For example, they may be planning a week's activities and need to know the schedule for each day; or they may be learning a language and need to know the names of the days of the week in English; or they may be interested in time management and want to learn how to efficiently use a week's time.

Step 5: Use the Chatbot for Q&A

Install Gradio to provide the UI display functionality.
```
!pip install gradio==5.50.0
```

Set the Chatbot UI port and obtain the proxy URL.

server_port = 7890 # Chatbot port, which can be customized.

import os

instance_id = os.environ["INSTANCE_ID"]
proxy_port = f"/proxy/{server_port}/"

try:
    from modelarts.session import Session
    from modelarts.config.auth import auth_by_apig

    project_id = os.getenv("PROJECT_ID")
    notebook_info = auth_by_apig(Session(), "GET", f"/v1/{project_id}/notebooks/{instance_id}")
    url = notebook_info.get("endpoints", [{}])[0].get("uri")
    proxy_url = url.replace("/lab", proxy_port)
except Exception:
    lab_url = os.getenv("LABCONSOLE_URL")
    if lab_url:
        url = lab_url.replace("console/lab", instance_id)
    else:
        region = os.environ["REGION_NAME"]
        url = f"https://authoring-modelarts.{region}.huaweicloud.com/{instance_id}"
    proxy_url = f"{url}{proxy_port}"

Start the chatbot UI.

Execute the script below. The UI has started successfully when Running on local URL: http://127.0.0.1:{server_port} is displayed.

import json
import os


import gradio as gr
import requests
server_name = "0.0.0.0"

def chat_with_openai_stream(message, history):
    url = f"http://127.0.0.1:{port}/v1/chat/completions"


    messages = []
    for history_item in history:
        messages.append({"role": history_item["role"], "content": history_item["content"]})
    messages.append({"role": "user", "content": message})

    payload = {
        "model": model_name,
        "messages": messages,
        "max_tokens": 2048,
        "stream": True # Enable streaming response.
    }

    try:
        response = requests.post(url, json=payload, stream=True, timeout=120)
        response.raise_for_status()

        full_answer = ""    # Accumulated: extracted [pure formal reply] content
        full_think = ""     # Accumulated: extracted [pure thinking] content
        has_think = False # Flag: Whether thinking content is detected. If not, the thinking module is not displayed.
        think_over = False # Whether thinking is complete.
        think_start = False # Whether thinking has started.


        # Process the streaming response line by line.
        for line in response.iter_lines():
            if line:
                line_str = line.decode('utf-8')

                if line_str.startswith('data: '):
                    line_str = line_str[6:]

                if line_str.strip() == '[DONE]':
                    continue

                try:
                    data = json.loads(line_str)
                    if 'choices' in data and len(data['choices']) > 0:
                        delta = data['choices'][0].get('delta', {})
                        current_content = delta.get('content', '')

                        if current_content:
                            if current_content == '<think>':
                                has_think = True
                                think_start = True
                                continue

                            if think_start:
                                current_content = current_content.replace('\n\n', '<br>').replace('\n', '<br>')
                                full_think += current_content

                            if has_think and current_content == '</think>':
                                think_over = True
                                think_start = False
                                continue

                            if think_over:
                                full_answer += current_content

                            if not has_think:
                                full_answer += current_content

                            if has_think and full_think:
                                # With thinking content: Thinking text (gray text + thinking prefix) + Formal reply, displayed in separate lines for clear display
                                show_text = f"<span style='color:#888; font-size:0.9em;'> [Model thinking] {full_think}</span>\n\n{full_answer}"

                            if not has_think:
                                # No thinking content: Only formal replies are displayed, which are exactly the same as the effect of your original code, without any unnecessary content.
                                show_text = full_answer

                            # Streaming response, retaining the good typewriter-like word-by-word effect.
                            yield show_text

                except json.JSONDecodeError:
                    continue
                except Exception as e:
                    print(f"An error occurred while processing streaming data: {e}")
                    continue

    except Exception as e:
        error_msg = f"Error: {str(e)}"
        yield error_msg

examples = [
    "Hello, please introduce yourself.",
    "What is 32 multiplied by 12?"
]

# Create a Gradio interface.
demo = gr.ChatInterface(
    type="messages",
    fn=chat_with_openai_stream,
    title="Chat",
    description="Chat with the local Qwen-3 model.",
    flagging_mode="manual",
    flagging_options=["Like", "Spam", "Inappropriate", "Other"],
    save_history=True,
    examples=examples
)

# Start the application.
gradio_share = os.environ.get("GRADIO_SHARE", "0").lower() in ["true", "1"]
demo.launch(
    share=gradio_share,
    inline=False,
    server_name=server_name,
    server_port=server_port,
    root_path=proxy_url
)
print(f"You are advised to open the link below in your browser for the best experience.\n {proxy_url}")
print(f"* (If {proxy_url} cannot be accessed, modify the proxy_url prefix by referring to the current notebook link.)")

Display the Chatbot UI.
Run the following script to display the Chatbot UI, where you can start chatting:
```
from IPython.display import IFrame, display
display(IFrame(src=proxy_url, width="100%", height="800"))
```
Figure 4 Chatbot

More Examples

The new ModelArts console provides a variety of introductory notebook examples. To explore them, log in to the new ModelArts console, click Getting Started from the navigation pane on the left, and click on any example card under the Development Practices tab to view its detailed description. You can also click Develop directly on an example card to quickly implement the corresponding feature. The names of some development practice cases are as follows:

Rapid Supervised Fine-Tuning (SFT) and Gradio Visualization Using MindSpeed-LLM
Qwen3 Training Profiling and TensorBoard Visualization
AI Video Generation Based on Ascend and ComfyUI
LLM Inference PD Ratio Tool: Instant Ratio Calculation
Interactive LoRA Fine-Tuning via LLama-Factory for Chinese Historical Character Role-Playing
Model Performance and Precision Evaluation Based on Ascend
GPU-to-NPU Migration: Handwritten Digit Recognition
One-Click Model Quantization Based on Ascend
Image Generation via SDXL Turbo and SDXL ReVision Based on ComfyUI

Click to enlarge

Previous topic: GPU-to-NPU Migration: Handwritten Digit Recognition

Next topic: Getting Started with Training: Training a Model with One Click