Building an AI Chatbot with the ChatGLM-6B Model on a GPU ECS

Scenarios

ChatGLM is a large language model based on the General Language Model (GLM) architecture. It is widely used in enterprise-grade conversational scenarios such as intelligent customer service, business Q&A, knowledge retrieval, and office assistants. It helps organizations build high-quality AI interaction capabilities at a low cost and accelerate the intelligent upgrade of business processes.

Architecture

Figure 1 Deploying a ChatGLM-6B chatbot on a GPU ECS

Advantages

By deploying ChatGLM on a Huawei Cloud GPU ECS, you can get a high-performance dedicated compute environment. It supports local model deployment, ensures data security, and offers predictable, stable performance with on-demand scalability. This helps enterprises quickly build AI conversational capabilities for efficient deployment and continuous innovation.

Resource Planning

Contact Huawei Cloud technical support to select GPU ECSs suitable for your deployment.

Prerequisites

• The rule listed in Table 3 has been added to the security group that the target ECS belongs to. For operation details, see Configuring Security Group Rules.

  Table 3 Security group rules

  Direction	Priority	Action	Type	Protocol & Port	Source
  Inbound	1	Allow	IPv4	TCP: 8000	0.0.0.0/0

Process

To build an AI chatbot with the ChatGLM-6B model on a Linux ECS, do as follows:

1. Create a GPU ECS.
2. Check the GPU driver and CUDA versions.
3. Initialize the data disk to store the model file and related resources.
4. Create a conda virtual environment.
5. Download the ChatGLM-6B model.
6. Download the ChatGLM demo.
7. Start the AI chatbot in CLI mode.
8. Start the AI chatbot in web mode.

Procedure

1. Create a GPU ECS.

   Here, we use the pi2.2xlarge.4 flavor and the Ubuntu OS as an example. When creating the ECS, pay attention to the parameters listed below. For details about other parameters, see Purchasing an ECS in Custom Config Mode.

   1. Instance: Select pi2.2xlarge.4. For more recommended specifications, see Table 2.
   2. Image: You are advised to select the Huawei Cloud EulerOS 2.0 or Ubuntu 22.04 public image that does not have drivers installed for GPU ECSs.
      Figure 2 Selecting an image
      
   3. EIP: Select Auto assign to enable public network access. The EIP is used to download the environment dependencies and call model APIs for inference.
   4. Security Group: Allow inbound traffic on TCP port 8000 to access Stable Diffusion WebUI. For operation details, see Configuring Security Group Rules.
   5. Storage & Backup: To ensure service stability and data security, you are advised to deploy the system disk and data disk separately.
      • System disk: 40 GiB, used to install the OS, driver, and runtime environment.
      • Data disk: 100 GiB, used to store the model file, generated images, and related service data.

2. Check the GPU driver and CUDA versions.

   Query GPU details.

   nvidia-smi

   Install the driver of version 535 and CUDA of 12.2. For details, see Manually Installing a Tesla Driver on a GPU-accelerated ECS.

3. Initialize the data disk to store the model file and related resources.

   After the GPU ECS is created, initialize the data disk by referring to Initializing a Linux Data Disk. In this example, we mount the data disk to the /mnt/model directory. You can change the directory as required.

   

4. Create a conda virtual environment.
   1. Download the miniconda installation package.

      wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh

   2. Install miniconda.

      bash Miniconda3-latest-Linux-x86_64.sh

   3. Add the conda environment variable to the startup file.

      echo 'export PATH="$HOME/miniconda3/bin:$PATH"' >> ~/.bashrc
      source ~/.bashrc

   4. Create a virtual environment based on Python 3.10.

      conda create -n chatglm python=3.10
      conda activate chatglm

5. Download the ChatGLM-6B model.
   1. Create a script for downloading the model.

      vim download_models.py

      Press i to enter insert mode and add the following content:

      from modelscope import snapshot_download
      model_dir = snapshot_download('ZhipuAI/ChatGLM-6B', cache_dir='/mnt/model', revision='master')

      ZhipuAI/ChatGLM-6B indicates the model name, and /mnt/model indicates the directory where the model is stored. You can change the directory as required.

      After the content is added, press Esc to exit insert mode. Type :wq and press Enter to save the file and exit.

   2. Download the ChatGLM model.

      pip install modelscope
      python download_models.py

      Wait until the model is downloaded. If Download model 'ZhipuAI/ChatGLM-6B' successfully. is displayed in the command output, the model is downloaded.

   

   ModelScope is an open-source model community in China, which provides faster access in China. If you are using the model outside China, obtain it from Hugging Face.

6. Download the ChatGLM demo.
   1. Download the ChatGLM demo file.

      git clone https://github.com/zai-org/ChatGLM-6B.git

   2. Install the dependencies required by the demo.

      cd ChatGLM-6B/
      sed -i '/torch/d' requirements.txt
      pip install torch==2.1.2+cu121 torchvision==0.16.2+cu121 --index-url https://download.pytorch.org/whl/cu121
      pip install -r requirements.txt

      Wait until the required dependencies are downloaded.

7. Start the AI chatbot in CLI mode.
   Run the following commands to start the AI chatbot in CLI mode:

   sed -i 's|"THUDM/chatglm-6b"|"/mnt/model/ZhipuAI/ChatGLM-6B"|g' cli_demo.py
   python cli_demo.py

   

   The sed command is used to replace the model pulled from Hugging Face with the local model. Use the path configured in downloading the ChatGLM-6B model as the path of the local model.

8. Start the AI chatbot in web mode.
   1. Run the following commands to start the AI chatbot in web mode:

      pip install streamlit
      pip install streamlit_chat
      export STREAMLIT_SERVER_HEADLESS=1
      sed -i 's|"THUDM/chatglm-6b"|"/mnt/model/ZhipuAI/ChatGLM-6B"|g' web_demo2.py
      streamlit run web_demo2.py --server.port 8000

      

   2. Use a local browser to access http://<EIP>:8000 and chat with the AI chatbot.

      

   The model has been deployed and verified. You can use the ECS CLI or access http://<EIP>:8000 to chat with the chatbot.