Deploying a Model from the vLLM-Ascend Open-Source Community on the ModelArts Inference Platform

Use Case

When you need to deploy LLMs from open-source communities such as Hugging Face and ModelScope to the Ascend NPU environment of the ModelArts inference platform, you may face the following challenges: large model weights, adaptation of the inference framework to the Ascend CANN operator, configuration of tensor parallelism and data parallelism parameters for multi-node multi-PU deployment, and inconsistent service status caused by improper health check configuration. Based on the vLLM-Ascend inference framework and ModelArts real-time service deployment capabilities, you can obtain models, prepare images, configure scripts, and deploy services in one-stop mode. This greatly simplifies the deployment of open-source models in Ascend environments.

Typical scenarios include:

• Ascend NPU-based model inference: Open-source LLMs such as GLM, Qwen, and DeepSeek are deployed on Ascend devices to provide text generation and Q&A capabilities.
• Multi-node multi-PU parallel inference: The tensor parallelism (TP) and data parallelism (DP) policies are used to split a large model for deployment on multiple NPUs, breaking the single-PU memory limit.
• Quantized model inference: Quantized models such as W8A8 are deployed to reduce memory usage and inference latency, while improving throughput.
• Service-oriented general-purpose model inference: Any open-source model supported by vLLM-Ascend is encapsulated into an OpenAI-compatible API for service systems to call.

Solution Process

To obtain a model from an open-source community and deploy it on the ModelArts inference platform, perform the following steps:

1. Prepare model weights: Download model weights from ModelScope or Hugging Face and upload them to an OBS bucket.
2. Prepare the deployment image: Download the vLLM-Ascend image from Quay.io and push it to an SWR image repository.
3. Prepare the startup script: Compile the startup script and health check script based on the model adaptation document and upload them to an OBS bucket.
4. Deploy the inference service: Configure a real-time service on ModelArts, mount the model weights and startup script, and start the inference service.
5. Verify the service: Call the inference service via APIs to verify service availability.

Constraints

• The model must be from the vLLM-Ascend official model list. Otherwise, you need to verify the model adaptation.
• During multi-node deployment, you need to ensure that the network communication is normal, as HCCL communication depends on the network between nodes.

Resource and Cost Planning

Table 1 lists the resources and cost planning used in this case.

Preparations

Before the deployment, ensure that you have:

• Enabled ModelArts, OBS, and SWR.
• Created a ModelArts dedicated resource pool with Snt9b NPU resources.
• Created an OBS bucket and an SWR organization, which are in the same region as ModelArts.
• Created an ECS. For details, see Purchasing and Using a Linux ECS.

Step 1: Download the Model Weight and Upload It to OBS

1. Access the vLLM-Ascend model adaptation description list, confirm that the target model is supported, and check the model weight requirements.

   GLM-5.2-W8A8 adaptation document: https://docs.vllm.ai/projects/ascend/en/latest/tutorials/models/GLM5.2.html.

2. Download the model weight from ModelScope or Hugging Face.

   Download the GLM-5.2-W8A8 model weight from ModelScope: https://www.modelscope.cn/models/Eco-Tech/GLM-5.2-w8a8/files

   

   Ensure that the downloaded model weight file is complete and contains the weight file (.safetensors), configuration file (config.json), and tokenizer.

3. Upload the downloaded model weight to the models directory in the OBS bucket.

   OBS path planning for the model weight file:

   ├── <OBS-bucket>
   │   └── models
   │       └── <model-name>                    # Directory where the model weight is located, which is mounted to the container during deployment.

   

   It takes a long time to upload large model weight files to OBS. Models with tens of billions of parameters are about 20 GB, and models with hundreds of billions of parameters are about 200 GB. The upload time depends on the network bandwidth.

Step 2: Download the Inference Image and Push It to SWR

Download the vLLM-Ascend image from the official image repository and push it to SWR through an ECS node.

1. Choose the inference image required by the target model based on the vLLM-Ascend model adaptation description.

   Take GLM-5.2 as an example, the inference image name is quay.io/ascend/vllm-ascend:glm5.2 for the A2 series.

2. Remotely log in to the ECS and check whether Docker has been installed.

   # Check whether Docker is installed.
   docker -v

   If Docker is not installed, run the following command:

   yum install -y docker-engine.aarch64 docker-engine-selinux.noarch docker-runc.aarch64

3. Use Docker to download the image on the ECS.

   docker pull --platform linux/arm64 <inference-image-name>

   The inference image name must be that obtained in 1.

   Example:

   docker pull --platform linux/arm64 quay.io/ascend/vllm-ascend:glm5.2

   

   If an Arm-powered image is used by the Ascend device, you must specify --platform linux/arm64 when downloading the image. Otherwise, the inference fails due to image architecture mismatch.

   If the download is successful, the download status is displayed:

   Status: Downloaded newer image for quay.io/ascend/vllm-ascend:glm5.2

   Troubleshooting

   • If the --platform parameter is unavailable when you run the pull command, rectify the fault by referring to FAQ: What Can I Do If "The --platform Parameter Is Not Supported" Is Displayed When I Run the pull Command?.
   • If ECS cannot connect to Quay.io when you run the pull command, rectify the fault by referring to FAQ: What Can I Do If the pull Respond Times Out Because the ECS Network Cannot Access the Image Repository?.
4. Log in to SWR, choose My Images, and click Upload Through Client in the upper right corner. In the displayed dialog box, click Generate Login Command to obtain the Docker login command.

   Login command example:

   docker login -u <region>@<account-AK> -p <temporary-token> <image-address>

   For details, see Pushing an Image.

5. Enter the obtained login command on the ECS to log in to Docker.

   If Login Succeeded is returned, the login is successful.

6. Tag the inference image.

   docker tag <inferenc-image-name> <image-address>/<organization-name>/<image-name>:<version>

   Table 2 Tag parameters

   Parameter	Description
   Inference Image Name	Actual image version to be pushed to SWR, for example, quay.io/ascend/vllm-ascend:glm5.2.
   Image Address	SWR image address, that is, the domain name at the end of the login command.
   Organization Name	SWR organization name created in preparations, for example, modelarts.
   Image Name	Custom image name, which can be the same as the actual image name, for example, vllm-ascend.
   Version	Custom version, which can be the same as the actual image version, for example, glm5.2.

   Example:

   docker tag quay.io/ascend/vllm-ascend:glm5.2 swr.xxx.example.com/modelarts/vllm-ascend:glm5.2

7. Push the tagged inference image to SWR.

   docker push <image-address>/<organization-name>/<image-name>:<version>

   Example:

   docker push swr.xxx.example.com/modelarts/vllm-ascend:glm5.2

   If the push is successful, the push progress of each layer and the final digest are displayed.

   glm5.2: digest: xxx:xxxxx size: xxxx

8. Check whether the image is uploaded on the SWR console. To do so, choose My Images > Private Images, and check whether the image has been pushed.
   Figure 1 Checking the image
   

Step 3: Compile a Script and Upload It to OBS

1. Confirm the startup command requirements of the target model based on the vLLM-Ascend model adaptation description.

   GLM-5.2-W8A8 adaptation document: https://docs.vllm.ai/projects/ascend/en/latest/tutorials/models/GLM5.2.html

2. Compile the get_master_ip.py tool for obtaining the master IP address from the global rank table. This tool is used to automatically detect the master/slave role during multi-node PD deployment. During single-node deployment, that is, there is only one inference unit (DP = 1), there is no need to distinguish the master and slave roles. Therefore, this script is not required.

   import json
   import os
   import time
   import sys
   
   GLOBAL_RANK_TABLE_ENV = 'GLOBAL_RANK_TABLE_FILE_PATH'
   
   def wait_and_get_completed_global_rank_table():
       global_rank_table_path = os.getenv(GLOBAL_RANK_TABLE_ENV)
       if not global_rank_table_path:
           print('read env "{}" failed'.format(GLOBAL_RANK_TABLE_ENV))
           return
       while True:
           try:
               rank_table = get_rank_table(global_rank_table_path)
               if rank_table is not None:
                   return rank_table
           except Exception as e:
               print(e)
           time.sleep(1)
   
   def get_rank_table(global_rank_table_path=None):
       if global_rank_table_path is None:
           global_rank_table_path = os.getenv(GLOBAL_RANK_TABLE_ENV)
       with open(global_rank_table_path, 'r') as file:
           buf = file.read()
       rank_table = json.loads(buf)
       if rank_table["status"] == "completed":
           return rank_table
       else:
           return None
   
   def get_master_ip():
       try:
           wait_and_get_completed_global_rank_table()
           global_rank_table_path = os.getenv(GLOBAL_RANK_TABLE_ENV)
           with open(global_rank_table_path, 'r') as file:
               buf = file.read()
           rank_table = json.loads(buf)
           master = rank_table['server_group_list'][0]['server_list'][0]
           return master['server_ip']
       except Exception as e:
           print(e)
   
   def get_dp_start_rank():
       try:
           wait_and_get_completed_global_rank_table()
           global_rank_table_path = os.getenv(GLOBAL_RANK_TABLE_ENV)
           pod_ip = os.getenv("POD_IP")
           if not pod_ip:
               print("read env POD_IP failed")
               return 0
           with open(global_rank_table_path, 'r') as file:
               buf = file.read()
           rank_table = json.loads(buf)
           rank = 0
           for group in rank_table.get('server_group_list', []):
               for server in group.get('server_list', []):
                   if server.get('server_ip') == pod_ip or server.get('server_id') == pod_ip or server.get('container_ip') == pod_ip:
                       return rank
                   rank += 1
           print(f"Warning: POD_IP {pod_ip} not found in rank table, fallback to rank 0")
           return 0
       except Exception as e:
           print(f"get_dp_start_rank error: {e}")
           return 0
   
   if __name__ == "__main__":
       if len(sys.argv) > 1 and sys.argv[1] == "--dp-rank":
           print(get_dp_start_rank())
       else:
           print(get_master_ip())

3. Compile the model startup script. For details about the script content, see the deployment part in the vLLM-Ascend model adaptation description. The following provides script examples in different deployment modes.
   • Multi-node PD deployment: The master/slave role needs to be detected. The following uses glm_5_2.sh (the model startup script used in this case) as an example.

     #!/bin/sh
     CUR_DIR=$(cd "$(dirname "$0")" && pwd)
     
     # ============ Model-specific parameters ============
     MODEL_PATH="/weight/"
     SERVED_MODEL_NAME="glm-52"
     PORT="7000"
     TENSOR_PARALLEL_SIZE=8
     DATA_PARALLEL_SIZE=2
     DATA_PARALLEL_SIZE_LOCAL=1
     MAX_MODEL_LEN=40000
     MAX_NUM_BATCHED_TOKENS=4096
     GPU_MEMORY_UTILIZATION=0.95
     MAX_NUM_SEQS=16
     QUANTIZATION="ascend"
     EXTRA_ARGS="--enable-expert-parallel \
       --safetensors-load-strategy 'prefetch' \
       --block-size 128 \
       --async-scheduling \
       --additional-config '{\"fuse_muls_add\": true, \"multistream_overlap_shared_expert\": true, \"ascend_compilation_config\": {\"enable_npugraph_ex\": true}}' \
       --compilation-config '{\"cudagraph_mode\": \"FULL_DECODE_ONLY\"}' \
       --speculative-config '{\"num_speculative_tokens\": 5, \"method\": \"deepseek_mtp\"}'"
     
     # Network settings
     nic_name=$(ifconfig | grep -B1 "$POD_IP" | grep -o "^\w*")
     master_ip=$(PYTHONUNBUFFERED=1 python3 "$CUR_DIR"/get_master_ip.py)
     dp_start_rank=$(PYTHONUNBUFFERED=1 python3 "$CUR_DIR"/get_master_ip.py --dp-rank)
     echo "master ip:${master_ip}, local ip: ${POD_IP}, nic:${nic_name}, dp_rank:${dp_start_rank}"
     
     # Determine the node role.
     if [ "$POD_IP" = "$master_ip" ]; then
         echo "Current node is MASTER"
         slave_opts="--api-server-count 1"
     else
         echo "Current node is SLAVE"
         nohup python -m http.server 8080 > http_server.log 2>&1 &
         slave_opts="--headless"
     fi
     
     # Environment variable configuration
     export HCCL_OP_EXPANSION_MODE="AIV"
     export HCCL_IF_IP=$POD_IP
     export GLOO_SOCKET_IFNAME=$nic_name
     export TP_SOCKET_IFNAME=$nic_name
     export HCCL_SOCKET_IFNAME=$nic_name
     export VLLM_RPC_TIMEOUT=360000
     export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=3000
     export HCCL_EXEC_TIMEOUT=200
     export HCCL_CONNECT_TIMEOUT=120
     export OMP_PROC_BIND=false
     export OMP_NUM_THREADS=10
     export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
     export ACL_OP_INIT_MODE=1
     export TASK_QUEUE_ENABLE=1
     export CPU_AFFINITY_CONF=1
     export VLLM_ENGINE_READY_TIMEOUT_S=1200
     
     # Build quantization parameters (optional, only for quantized models).
     quant_opts=""
     if [ -n "$QUANTIZATION" ]; then
         quant_opts="--quantization ${QUANTIZATION}"
     fi
     
     # Start the service.
     vllm serve ${MODEL_PATH} \
       --host 0.0.0.0 \
       --port ${PORT} \
       --max_model_len ${MAX_MODEL_LEN} \
       --max-num-batched-tokens ${MAX_NUM_BATCHED_TOKENS} \
       --served-model-name ${SERVED_MODEL_NAME} \
       --seed 1024 \
       --gpu-memory-utilization ${GPU_MEMORY_UTILIZATION} \
       --max-num-seqs ${MAX_NUM_SEQS} \
       --data-parallel-size ${DATA_PARALLEL_SIZE} \
       --data-parallel-size-local ${DATA_PARALLEL_SIZE_LOCAL} \
       --data-parallel-start-rank ${dp_start_rank} \
       --data-parallel-address ${master_ip} \
       --data-parallel-rpc-port 13389 \
       --tensor-parallel-size ${TENSOR_PARALLEL_SIZE} \
       ${quant_opts} ${EXTRA_ARGS} \
       $slave_opts

   • Single-node deployment: The master/slave role does not need to be distinguished. The model startup script is simpler. The following uses minitron-8b-base.sh as an example.

     #!/bin/sh
     
     # ============ Model-specific parameters ============
     MODEL_PATH="/weight/"
     SERVED_MODEL_NAME="minitron-8b-base"
     PORT="8000"
     TENSOR_PARALLEL_SIZE=1
     MAX_MODEL_LEN=4096
     GPU_MEMORY_UTILIZATION=0.9
     EXTRA_ARGS="--enforce-eager"
     
     # Start the service.
     vllm serve ${MODEL_PATH} \
       --host 0.0.0.0 \
       --port ${PORT} \
       --max_model_len ${MAX_MODEL_LEN} \
       --served-model-name ${SERVED_MODEL_NAME} \
       --gpu-memory-utilization ${GPU_MEMORY_UTILIZATION} \
       --tensor-parallel-size ${TENSOR_PARALLEL_SIZE} \
       ${EXTRA_ARGS}

   Table 3 Startup script parameters

   Variable	Description	Example Value
   MODEL_PATH	Model weight mounting path, which must be the same as that on ModelArts.	/weight/
   SERVED_MODEL_NAME	Inference service name, which is used during API calling. It is recommended that the value be the same as the model name.	glm-52
   TENSOR_PARALLEL_SIZE	Number of TPs, which must be the same as the number of NPUs used by each inference unit during deployment.	8
   DATA_PARALLEL_SIZE	Number of DPs, which must be the same as the total number of instances used during deployment (Total number of instances = Number of inference units x Number of instances per inference unit).	2
   DATA_PARALLEL_SIZE_LOCAL	Number of DPs on a single node, which must be the same as the number of instances per inference unit during deployment.	1
   MAX_MODEL_LEN	Maximum context length, which is configured based on the vLLM-Ascend model adaptation description.	40,000
   MAX_NUM_BATCHED_TOKENS	Maximum number of tokens per batch, which can be configured as required.	4,096
   GPU_MEMORY_UTILIZATION	Memory usage, which is configured based on the vLLM-Ascend model adaptation description.	0.95
   MAX_NUM_SEQS	Maximum number of concurrent sequences, which can be adjusted based on service concurrency requirements.	16
   QUANTIZATION	Quantization mode. For W8A8 models, ascend is used. For non-quantized models, this variable can be deleted.	ascend
   EXTRA_ARGS	Model-specific optimization parameters, including expert parallelism, speculative decoding, and operator fusion. Replace or delete the parameters based on the model adaptation document.	--enable-expert-parallel ...
   PORT	Listening port of the vLLM inference service. The value must be the same as the container port during deployment. Ensure that there is no port conflict.	7000

4. Compile the check.sh health check script. The following provides script examples for different deployment modes. The value of PORT must be the same as that in the model startup script.
   • Multi-node PD deployment: You need to call get_master_ip.py (health check script used in this case) to obtain the master IP address.

     #!/bin/bash
     PROBE_TYPE=$1
     
     # The port number must be the same as that in the model startup script.
     PORT="7000"
     
     if [[ $PROBE_TYPE == "startup" ]]; then
         if pgrep -f "vllm serve" > /dev/null; then
         exit 0
         fi
         exit 1
     fi
     
     master_ip=$(PYTHONUNBUFFERED=1 python3 /code/get_master_ip.py)
     
     if [[ $PROBE_TYPE == "readiness" ]]; then
         HEALTH_URL="http://${master_ip}:{PORT}/health"
         status_code=$(curl -s -o /dev/null -w "%{http_code}" $HEALTH_URL)
         if [ "$status_code" -eq 200 ]; then
           exit 0
         else
           echo "readiness check failed" >&2
           exit 1
         fi
     fi
     
     if [[ $PROBE_TYPE == "liveness" ]]; then
         if pgrep -f "vllm serve" > /dev/null; then
         exit 0
         fi
         exit 1
     fi

   • Single-node deployment: The master/slave role does not need to be distinguished. The health check script is simpler.

     #!/bin/bash
     PROBE_TYPE=$1
     
     # The port number must be the same as that in the model startup script.
     PORT="8000"
     
     if [[ $PROBE_TYPE == "startup" ]]; then
         if pgrep -f "vllm serve" > /dev/null; then
         exit 0
         fi
         exit 1
     fi
     
     if [[ $PROBE_TYPE == "readiness" ]]; then
         HEALTH_URL="http://${POD_IP}:${PORT}/health"
         status_code=$(curl -s -o /dev/null -w "%{http_code}" $HEALTH_URL)
         if [ "$status_code" -eq 200 ]; then
           exit 0
         else
           echo "readiness check failed" >&2
           exit 1
         fi
     fi
     
     if [[ $PROBE_TYPE == "liveness" ]]; then
         if pgrep -f "vllm serve" > /dev/null; then
         exit 0
         fi
         exit 1
     fi

5. Upload the startup script and health check script to the code directory of the OBS bucket.

   OBS path planning for the script file:

   ├── <OBS-bucket>
   │   └── models
   │       └── <model-name>                    # Directory where the model weight is located, which is mounted to the container during deployment.
   │   └── code
   │       ├── get_master_ip.py            # Obtain the master IP tool from the global rank table (only for multi-node PD deployment).
   │       ├── glm_5_2.sh                  # Model startup script
   │       └── check.sh                    # Health check script

Step 4: Deploy the Inference Service

Configure and start a real-time service on ModelArts.

1. Log in to the ModelArts console.
2. (Recommended) Warm up the model weight in the dedicated resource pool in advance for faster deployment.
   

   For first-time deployment of the inference service, you need to download the model weight from OBS to the node. You can load the model in advance using resource pool warmup as it takes a long time to load large models.

   1. In the navigation pane on the left, choose Resource Management > Dedicated Compute Resources > Resource Pools.
   2. Click the target dedicated resource pool to access its details page.
   3. In the Warmup tab, click Add Warmup Task. In the displayed dialog box, set the parameters. The table below lists only key parameters. For details, see Adding a Model Warmup Task to a Dedicated Resource Pool.

      Table 4 Key parameters for adding a model warmup task

      Parameter	Description	Example Value
      Weight Path	Select the OBS path of the model weight file to be warmed up, that is, the path in Step 1: Download the Model Weight and Upload It to OBS.	obs://xxx/models/
      File Space (GB)	Enter the size of the model weight file to be warmed up. After Weight Path is configured, click Obtain weight file size to obtain the recommended value. Generally, the predicted value is slightly greater than the actual size of the model directory.	780

   4. After the configuration is complete, click OK to submit the model warmup task.

      When Status changes to Ready, the model warmup is complete.

3. Return to the ModelArts main menu and choose Model Inference > Real-Time Inference from the navigation pane. The real-time service management page is displayed.
4. Click Deploy in the upper right corner. The Deploy Real-Time Service page is displayed.
5. On the displayed page, configure service information. The table below only lists key parameters. For details, see Configuring Service Information.

   Table 5 Key parameters of service information

   Category	Parameter	Description	Example Value
   Basic Information	Service Name	Custom service name. You are advised to add the model name to this value.	glm_52_w8a8
   Network Settings	Service Access Method	Select Default to use ModelArts capabilities.	Default
   	Service Protocol	Select HTTP and set the request timeout interval as required.	HTTP
   	Authentication Mode	Select an authentication mode as required. In the test environment, you can select None to directly call APIs.	None

6. Click Next to configure service deployment. The table below only lists key parameters. For details, see Deploying a Real-Time Inference Service Using Multi-Node PD Co-location.

   Table 6 Key parameters for deployment configuration

   Category	Parameter	Description	Example Value
   Resource Settings	Resource Pool Type	Select Dedicated Resource Pool to obtain exclusive resources.	Dedicated Resource Pool
   Model Settings	Model Source	Select Custom Model.	Custom Model
   	Storage Type	Select Pre-warmed model.	Pre-warmed model
   	Pre-warmed Model	Select a pre-warmed model.	glm52w8a8
   	Mount Path	Mount path of the model weight in the container, which must be the same as that in the startup script glm_5_2.sh.	/weight/
   Unit Settings	Deployment Mode	Select a model deployment mode. Basic mode: Single-node deployment. An inference unit is configured. Role detection and PD communication are not required. Multi-role: Multi-node PD deployment. Multiple inference units are configured. The get_master_ip.py script is required to check the master and slave roles.	Multi-role
   	Unit Name	Customized inference unit name.	role-0
   	Specification Type	Select Custom.	Custom
   	NPU	Number of NPUs used by each inference unit, which must be the same as the value of TENSOR_PARALLEL_SIZE in the model startup script, for example, glm_5_2.sh.	8
   	CPU	Set this parameter based on the resource pool specifications.	144
   	Memory	Set this parameter based on the resource pool specifications.	512,000 MB
   	Single-Replica Instances	Number of instances on each inference unit, which must be the same as the value of DATA_PARALLEL_SIZE_LOCAL in the model startup script, for example, glm_5_2.sh.	1
   	Image Type	Select Custom Images.	Custom Images
   	Select Image	Select a private image from SWR, that is, the image in Step 2: Download the Inference Image and Push It to SWR.	vllm-ascend:glm52
   	Mount File Storage	Mount the code folder in OBS to the container. Under File Storage, set Storage Type to Object Storage Service — Bucket, Storage Location to code, and Mount Path to /code/.	See the parameter description.
   	Health Check	Configure a health check. Startup Probe: Check whether the vLLM process is started. Set Check Method to Command and enter bash /code/check.sh startup. Readiness Probe: Check whether the inference service can receive requests (only the /health API on the master node is checked). Set Check Method to Command and enter bash /code/check.sh readiness. Liveliness Probe: Check whether the vLLM process is running. Set Check Method to Command and enter bash /code/check.sh liveness.	See the parameter description.
   	Boot Command	Enter the boot command and mount the /code/ directory. After the container is started, the code path in OBS is mounted to this path. In the boot command, use the boot script in this path to start the vLLM service.	bash /code/glm_5_2.sh
   	Add Inference Unit	During multi-node PD deployment, add inference units as required and configure the parameters by referring to the first inference unit. Customize the unit name, for example, role-1. Set same values for other parameters.	-
   Deployment Management Settings	Container Protocol	Network transmission protocol of the container.	HTTP
   	Container Port	Listening port of the image, which must be the same as the value of port in the model startup script, for example, glm_5_2.sh.	7000

7. Click Next and confirm the configuration details.
8. Click Deploy to start the real-time service.
9. When the inference service is in the Running state, click the service name to access its details page, and click the Logs tab.
10. In the displayed tab, check the deployment log. If the information below is displayed, the inference service is deployed.

    INFO: Application startup complete.

Step 5: Verify Service Availability via APIs

Verify the availability of the inference service via APIs.

1. Click the deployed inference service in the list to access its details page.
2. Click the Prediction tab.
3. Set the request parameters.

   Example:

   • Request method: POST
   • Request URL: /v1/completions
   • Data format: stream
   • Request body:

     {
         "model": "glm-52",
         "prompt": "The future of AI is",
         "max_completion_tokens": 50,
         "temperature": 0
     }

4. Click Predict and view the returned result. If the service is normal, the model name, model information, and answer are returned.

   For the first request, the model needs to be compiled and the cache needs to be warmed up. It is normal that the response is slow. Subsequent requests are responded to normally.

FAQ: What Can I Do If "The --platform Parameter Is Not Supported" Is Displayed When I Run the pull Command?

If the error below is reported when you run the pull command, you need to enable the Docker experiment feature before using --platform.

"--platform" is only supported on a Docker daemon with experimental features enabled

Enable the experiment mode:

1. Edit /etc/docker/daemon.json:

   {
     "experimental": true
   }

2. Restart Docker:

   systemctl restart docker

3. Run the pull command again.

FAQ: What Can I Do If the pull Respond Times Out Because the ECS Network Cannot Access the Image Repository?

If the error below is reported when you run the pull command, ECS cannot connect to Quay.io. In this case, download the image from a server that can connect to Quay.io, export the image, and upload it to the current ECS.

Error response from daemon: Get "https://quay.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Solution:

1. Check the network between the ECS and Quay.io. If a timeout error is reported, the ECS is isolated from the network and cannot directly access the image repository.

   curl -I https://quay.io

2. Check the model deployment environment.
   • If the model is to be deployed in a LAN, go to the next step. Use another ECS that supports public network access to download the image, export the image, and upload it to the ECS in the LAN.
   • If the model is to be deployed on a public network, bind an EIP to the current ECS and perform Step 2: Download the Inference Image and Push It to SWR again to continue the deployment.
3. Use another ECS that supports public network access to download the image.
   1. Remotely log in to the ECS and check whether Docker has been installed.

      # Check whether Docker is installed.
      docker -v

      If Docker is not installed, run the following command:

      yum install -y docker-engine.aarch64 docker-engine-selinux.noarch docker-runc.aarch64

   2. Use Docker to download the image on the ECS.

      docker pull --platform linux/arm64 <inference-image-name>

      The inference image name must be that obtained in 1.

      Example:

      docker pull --platform linux/arm64 quay.io/ascend/vllm-ascend:glm5.2

4. After the image is downloaded, export it as a vllm-ascend.tar file.

   docker save <inference-image-name> > vllm-ascend.tar

   Example:

   docker save quay.io/ascend/vllm-ascend:v0.22.1rc1 > vllm-ascend.tar

5. Upload vllm-ascend.tar to the ECS in the LAN. You can use physical transmission methods such as SCP, FTP, portable hard disk, or USB flash drive, or use OBS obsutil to upload and download data.

6. On the ECS in the LAN, load the image to the local Docker.

   docker load < vllm-ascend.tar

   If the operation is successful, the name of the loaded image is displayed.

   Loaded image: quay.io/ascend/vllm-ascend:v0.22.1rc1

7. Push the loaded image to SWR on the intranet. For details, see 4 to 8.
8. Continue with the model deployment.