Preparing an Inference Image

Inference Image Specifications

The image used for deploying a service must meet the following requirements.

1. The inference API defined by the image must support the HTTP, HTTPS, WS, or WSS protocol. If the HTTPS or WSS protocol is used, TLS authentication can be enabled.
2. The custom metric collection API defined by the image must support the HTTP or HTTPS protocol. If the HTTPS protocol is used, two-way authentication cannot be enabled.
3. The HTTP request health check API defined by the image must support the HTTP or HTTPS protocol. If the HTTPS protocol is used, two-way authentication cannot be enabled.
4. The API protocol and port number can be configured separately for the inference API, custom metric collection API, and health check API defined by the image. You can allocate ports in the image in advance.
5. There are no restrictions on the command used for health checks defined in the image; it can be customized as needed.
6. Real-time services support code and key mounting. If code or key configuration is required in an image, you can plan the mount path in the image in advance.
7. The inference platform does not limit image size. However, the image must be smaller than the container disk size you choose when buying the resource pool. If it is larger, the image will not load when starting the inference service.
8. The inference platform has no specific user requirements for starting the container. You can use either the root or ma-user account, based on your image settings.

Complete Process of Creating an Inference Image

You can pull existing inference service images via ECS or CCE, or build a custom inference service image, and then push it to SWR. When deploying an inference service, simply enter the corresponding SWR image address.

A CCE cluster is used here for image creation because Containerd is pre-installed and ready for use on CCE nodes by default. To use an ECS to create images, ensure the following requirements are met:

• Architecture & OS: Select the CPU architecture based on your specific needs. EulerOS is the recommended image.
• Runtime: Containerd must be installed and functional on the ECS.

Step 1: Creating a Single-Node CCE Cluster

1. Create a single-node CCE cluster of the Containerd container on the CCE console.
   1. Log in to the CCE console. In the upper left corner of the page, click and select a region for your cluster. The closer the selected region is to the region where resources are deployed, the lower the network latency and the faster the access.

      Click Buy Cluster. If you use CCE for the first time, create an agency following the instructions.

   2. Configure the basic information. Key parameters are as follows. For details about more parameters, see Buying a CCE Standard/Turbo Cluster.
      • Type: CCE Standard Cluster
      • Cluster Version: v1.33
      • Max Worker Nodes: 50
      • Master Nodes: Single
      • Network Model: Tunnel network
      • Container CIDR Block: Auto select

      Configure other parameters as required or retain the default settings.

2. Create a node.
   1. Log in to the CCE console, choose Clusters, and click the name of the cluster created in the previous step to access the cluster console.
   2. In the navigation pane, choose Nodes, click the Nodes tab, click Create Node in the upper right corner, and configure the node parameters. For more parameter descriptions, see Creating a Node.
      • Login mode: Select Password and enter the password.
      • Storage Settings: Set the system disk and data disk sizes based on the image size.

      Retain the default settings for other parameters.

   3. Click Next: Confirm, read the usage note, confirm the information, and click Submit.

Step 2: Creating an Image Organization

Go to the SWR console. On the Overview page, click Create Organization in the upper right corner, enter an organization name, and click OK.

Figure 1 Creating an image organization

Step 3: Logging In to SWR on a CCE Node

1. Go to the EIP console. On the EIPs page, click Buy EIP to create a temporary EIP.

   For details, see Assigning an EIP.

2. Go to the CCE console, click Clusters and then the target cluster name to access the cluster details page, choose Nodes > Nodes, and click the name of the node created in Step 1: Creating a Single-Node CCE Cluster. In the displayed dialog box, click OK. The ECS console is displayed.
3. Click EIPs and click Bind EIP. In the displayed dialog box, select the created EIP and click OK.
4. Use PuTTY to log in to the CCE node as user root through the EIP. For details, see Logging In to a Node.
   Figure 2 Successful login to the CCE node
   

Step 4: (Optional) Writing a Dockerfile

If you already have an inference service image, you can skip this step. Alternatively, you can refer to this step to write a Dockerfile for your inference image.

When creating an inference service image, you are advised to build upon the official images provided by ModelArts. Essential software such as drivers, CUDA, and CANN are pre-installed in official images, saving time and effort. If the official images do not meet your requirements, you can also build based on other base images, provided they comply with the inference image specifications outlined above.

For details about official ModelArts images, see ModelArts Preset Images. Official ModelArts images are hosted in SWR and can be retrieved using the following commands:

# View the list of available official images on the ModelArts console's Image Management page.
# Example: Pull an image. The image path is an example.
docker pull swr.{region}.myhuaweicloud.com/atelier/pytorch_ascend:pytorch_2.7.1-cann_8.5.2-py_3.12-hce_2.0.2512-aarch64-snt9b-20260417112518-aabfd52
# Replace {region} with the actual region code, for example, cn-north-4 or cn-southwest-2.

When building upon an official NPU inference image, the following specifications are pre-configured in the official image and do not require manual handling:

• Basic system: preset
• CANN software package: pre-installed and environment variables configured
• Python: pre-installed

You only need to focus on the following core specifications:

• Inference port: EXPOSE 8080
• Inference & health check APIs: Implemented within the inference service script
• Startup entrypoint: Set the inference service startup command using ENTRYPOINT
• Model load path: Ensure the model loading directory exists, e.g., /home/ma-user/model
• Dependency pre-installation: pip install --no-cache-dir
• Log output: Output logs to stdout in the inference service code.

Below is a complete Dockerfile example, with comments on key lines indicating the corresponding specification requirements:

# Pull the official ModelArts base image ${base_image}
FROM ${base_image}
# Switch to the root user for installation operations
USER root
# Install additional system dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
    vim \
    curl \
    && rm -rf /var/lib/apt/lists/*
# Install additional Python packages (inference framework and dependencies). Packages and versions here are examples only. Install them as needed
RUN pip install --no-cache-dir \
    transformers==4.30.0 \
    accelerate==0.21.0 \
    fastapi==0.100.0 \
    uvicorn==0.23.0
# Copy the inference service script. inference_server.py is used here as an example. Replace it with the actual one.
COPY inference_server.py /home/ma-user/inference_server.py
# Ensure correct permissions for the ma-user directory
RUN chown -R ma-user:ma-user /home/ma-user
# Ensure the model load path directory exists, e.g., /home/ma-user/model
RUN mkdir -p /home/ma-user/model && chown ma-user:ma-user /home/ma-user/model
# Expose the inference service port
EXPOSE 8080
# Switch back to ma-user
USER ma-user
# Set working directory
WORKDIR /home/ma-user
# Set the inference service startup command
ENTRYPOINT ["python", "/home/ma-user/inference_server.py"]
# Build the new image ${image}
docker build -t ${image}

Step 5: Uploading an Inference Image to the CCE Node

1. Upload the prepared image to the CCE node.

   Run the following command on the CCE node:

   ctr image import <image-package-name>

   Parameters:

   <image-package-name>: image package name, including the file name extension, For example, helloworld-aarch64_1.0.20250322104447.tar.

   Example:

   ctr image import helloworld-aarch64_1.0.20250322104447.tar

   Figure 3 Image package imported
   
2. Tag the loaded image package.

   Run the following command on the CCE node:

   ctr image tag <original-package-path> <new-package-path>

   Parameters:

   <original-package-path>: Run ctr image list to obtain the original path. For example, swr.example.myhuaweicloud.com/itep-test-mock/helloworld-aarch64:1.0.20250322104447.

   Figure 4 Obtaining the original package path
   

   <new-package-path>: SWR address/organization name/image name/tag, for example, swr.ma-region-3.ma03.external.com/test/helloworld-aarch64:1.0.20250322104447.

   Example:

   ctr image tag swr.cn-north-7.myhuaweicloud.com/itep-test-mock/helloworld-aarch64:1.0.20250322104447 swr.ma-region-3.ma03.external.com/test/helloworld-aarch64:1.0.20250322104447

   Figure 5 Image package tagged
   

Step 6: Uploading the Image to SWR

Run the following command on the CCE node:

ctr image push --user <resource_space_name>@<AK>:<login_key>  <region_SWR_address>/<organization_name>/<image_name>:<tag>

Parameters:

• <resource_space_name>: Log in to the management console, click the username in the upper right corner, and click My Settings. In the Resource Spaces tab, query the resource space corresponding to the current region, for example, ma-region-3.
• To obtain the <AK>, follow these steps:
  1. Log in to the console, click the username in the upper right corner, and click My Settings.
  2. On the My Settings page, click Access Keys.
  3. Click Add Access Key to create an AK/SK.
  4. Click OK to download the access key file.
  5. After the file is downloaded, obtain the AK/SK information from the credentials file.
• <login_key>: Log in to a Linux PC and run the command below to obtain the login key. Replace $AK and $SK with the actual AK/SK in the credentials file.

  printf "$AK" | openssl dgst -binary -sha256 -hmac "$SK" | od -An -vtx1 | sed 's/[ \n]//g' | sed 'N;s/\n//'

• <region_SWR_address>: SWR address of the corresponding region, for example, swr.ma-region-3.ma03.external.com.
• <organization_name>: Name of the created organization, for example, test.
• <image_name>:<tag>: Custom image name, for example, helloworld-aarch64:1.0.20250322104447.

Example:

ctr image push --user ma-region-3_test_01@<AK>:<login-key>  swr.ma-region-3.ma03.external.com/test/helloworld-aarch64:1.0.20250322104447

Figure 6 Image uploaded to SWR

If error "SWR domain name x509: certificate has expired or is not yet valid" is reported during image pulling or pushing, add -k to the end of the push or pull command.

Follow-Up Operation

Deploying a Real-Time Inference Service Using a Single Node: Deploy your model as a real-time service on ModelArts and use it for predictions.