Creating a Custom Image on ECS and Using It

Custom Image Build Process

1. Prepare a Linux server. In this example, an ECS is used.
2. Create a custom image on the ECS. The Dockerfile sample file is provided.
3. Upload the created image to SWR.
4. Register an SWR image on ModelArts.
5. Create a notebook instance and verify the new image.

Specifications for Custom Images

The base image for creating a custom image must meet either of the following conditions:

• It is an open-source image from the official website of Ascend or Docker Hub and it meets the following OS constraints:

  x86: Ubuntu 22.04 or Ubuntu20.04

  ARM: EulerOS 2.10.x, EulerOS 2.9.x, EulerOS 2.8.x, HCE 3.0, or HCE 2.0

• If an image error occurs due to unmet requirements, check the image specifications and rectify the fault by referring to Troubleshooting for Custom Images in Notebook Instances. If the fault persists, contact technical support.

This example installs PyTorch 2.7, FFmpeg 4, and GCC 8 on the preset PyTorch image from ModelArts to create a new image for AI development.

Step 1: Preparing an ECS and Configuring the Environment

Prepare a server with Docker enabled. If no such a server is available, create an x86 ECS, buy an EIP, and install required software on it. ModelArts provides Ubuntu scripts to facilitate Docker installation.

• Operations on a local Linux server are identical to those on an ECS server. For details, see this case.
• The Docker image architecture must be the same as the host architecture.

1. Log in to the ECS console and purchase an ECS.
   • Quick purchase: In the OS area, select Ubuntu 22.04. For details, see Purchasing an ECS in Quick Config Mode.
     Figure 1 OS
     

   • Custom purchase: In the OS area, select Ubuntu 22.04. In the Storage & Backup area, set the system disk size to 100 GiB. For details, see Purchasing an ECS in Custom Config Mode.
     Figure 2 OS
     
     Figure 3 Storage & Backup
     

2. Purchase an EIP and bind it to the ECS. For details, see Binding an EIP.
3. Log in to the ECS and configure the Docker environment.
   1. Run the following command on the ECS to download the configuration script: Only Ubuntu scripts are supported.

      wget https://cnnorth4-modelarts-sdk.obs.cn-north-4.myhuaweicloud.com/modelarts/custom-image-build/install_on_ubuntu1804.sh

   2. Run the following command on the ECS to configure the environment:

      bash install_on_ubuntu1804.sh

      Figure 4 Configuration succeeded
      

      source /etc/profile

      The installation script is executed to:

      1. Install Docker.
      2. If the Docker ECS runs on GPUs, install nvidia-docker2 to mount the GPUs to the Docker container.

Step 2: Creating a Custom Image

This section describes how to edit a Dockerfile, use it to create an image, and use the created image to create a notebook instance. For details about the Dockerfile, see Dockerfile reference.

1. Obtain the base image. (Skip this step if you are using a third-party image.)

   For addresses of the unified images provided by ModelArts, see ModelArts Unified Images or ModelArts Preset Images.

2. Access SWR.
   1. Log in to the SWR console, choose Dashboard in the navigation pane, and click Generate Login Command in the upper right corner. On the displayed page, click the copy button.
      Figure 5 Obtaining the login command
      
      

      You can select a validity period as required. To obtain a long-term valid login command, see Obtaining a Long-Term Login or Image Push/Pull Command. After you obtain a long-term valid login command, your temporary login commands will still be valid as long as they are in their validity periods.

   2. Run the login command on the machine where the container engine is installed. The message "Login Succeeded" will be displayed upon a successful login.
3. Pull a ModelArts image or a third-party image.

   The following uses a ModelArts image as an example. If you use a third-party image, replace the image address with the actual one. For details about how to obtain a ModelArts image address, see ModelArts Unified Images or ModelArts Preset Images.

   docker pull swr.cn-north-4.myhuaweicloud.com/atelier/pytorch_cuda:pytorch_1.12.1-cuda_10.2-py_3.9.11-ubuntu_20.04-x86_64-20260425154604-d9a1f9f

4. Compile a Dockerfile.

   A Dockerfile is a text file with instructions to build a Docker image. It defines the build process, such as the base image, software to install, files to copy, and environment variables.

   Run the vim command to create a Dockerfile. If a ModelArts unified image is used, see Dockerfile on a ModelArts Base Image for details about the Dockerfile.

   If a third-party image is used, add user ma-user whose UID is 1000 and user group ma-group whose GID is 100. For details, see Dockerfile on a Non-ModelArts Base Image.

   In this case, PyTorch 2.7.0, FFmpeg 4, and GCC 8 will be installed on a PyTorch image to build an AI image.

   vim Dockerfile

5. Build an image.

   Run the docker build command to build a new image from the Dockerfile. Command:

   docker build -t {region_information}/{organization_name}/{image_name}:{version_name} -f {Dockerfile_name} .

   Table 1 Parameters in the command

   Parameter	Description	Example Value
   -t	Specifies the new image path, including region information, organization name, image name, and version. Set this parameter based on the actual requirements. You are advised to use a complete SWR address for subsequent debugging and registration.	-
   {region_information}	The format is swr.{regionName}.myhuaweicloud.com. You can obtain the value of regionName from the project column on the API Credentials page.	swr.cn-north-4.myhuaweicloud.com
   {organization_name}	You can view existing organizations or create an organization through SWR Organization Management page.	dev-custom
   {image_name}	Custom image name.	pytorch2_7
   {version_name}	Custom version name.	v1
   -f	Specifies the Dockerfile name. Set this parameter based on the actual requirements.	-
   {Dockerfile_name}	Custom Dockerfile name.	Dockerfile
   .	Indicates that the file is in the current directory.	-

   The following is an example command. It takes about one hour to build the image, but the time varies based on the image size.

   docker build -t swr.cn-north-4.myhuaweicloud.com/dev-custom/pytorch2_7:v1 -f Dockerfile .

   Figure 6 Custom image built successfully
   

Step 3: Pushing the Image to SWR

After the image debugging is complete, push the image to SWR.

1. Log in to the SWR console and run the docker push command to push the image. For details, see Pushing an Image.
   The command format is as follows. For details about the command parameters, see Table 1.

   docker push {SWR_address}/{organization_name}/{image_name}:{version_name}

   Example:

   docker push swr.cn-north-4.myhuaweicloud.com/dev-custom/pytorch2_7:v1

2. In the navigation pane of the SWR console, choose My Images. Refresh the page to view the new image. Click the image name to view its details.
   Figure 7 Uploading the image to SWR
   

Step 4: Registering the Image with ModelArts

After an image is debugged, register it with ModelArts image management so that the image can be used in ModelArts.

1. Log in to the ModelArts console. In the navigation pane on the left, perform the following operations:
   • New console: Choose Asset Management > Images. Click the Registered Images tab.
   • Old console: Choose Asset Management > Image Management. Click the Registered Images tab.
2. Click Register. Set SWR Source to the image pushed to SWR in step 3.

   When you register an image, ensure that the architecture and type are the same as those of the image source. Otherwise, the creation fails.

   • Architecture: Run the docker run --rm {image_name} uname -m command to view the architecture. Replace image_name with the actual image name. Example:

     docker run --rm swr.cn-north-4.myhuaweicloud.com/dev-custom/pytorch2_7:v1 uname -m

     As shown in the following figure, the architecture of the image is x86.

     Figure 8 Viewing the image architecture
     
   • Type: Determine if the CUDA or CANN package is installed when pulling the image. The CUDA package indicates GPU, and the CANN package indicates NPU.

     For example, the address for pulling the image is swr.cn-north-4.myhuaweicloud.com/atelier/pytorch_cuda:pytorch_1.12.1-cuda_10.2-py_3.9.11-ubuntu_20.04-x86_64-20260425154604-d9a1f9f. From cuda_10.2, it can be seen that CUDA is installed, so the type is GPU.

   • Specification: If the type is NPU, choose a specification. The specification's model must match the image model.

     Get the specification from the image address. For example, if the address is swr.cn-north-4.myhuaweicloud.com/atelier/pytorch_ascend:pytorch_2.7.1-cann_8.5.2-py_3.12-euler_2.10.11-aarch64-snt9b-20260417112518-aabfd52, the image specification is snt9b.

3. On the Registered Images page, you can view the registered image.

Step 5: Creating a Notebook Instance and Using a Custom Image

1. In the navigation pane of the ModelArts console, perform the following operations as required:
   • New console: Choose Model Build > Notebook.
   • Old console: Choose Development Space > Notebook.
2. In the upper right corner of the Notebook page, click Create. In the Environment section, choose Custom Images, click , select the image registered in step 4, and set WebIDE to JupyterLab.
   For details about the parameters for creating a notebook instance, see Creating a Notebook Instance.

   Figure 9 Creating a notebook instance
   

   If the notebook instance status is Running, the notebook instance has been created.

3. On the Notebook page, click Access Environment in the Operation column. In the Access Method dialog box, click Access on the right of JupyterLab Access.
4. In the notebook area on the ModelArts Launcher page, click PyTorch to create an IPYNB file, import the PyTorch library, and view the PyTorch version.

   import torch
   print(torch.__version__)

   Figure 10 Creating an IPYNB file
   

   As shown in the Dockerfile, PyTorch version 2.7.0 is installed.

5. Open another terminal and run the following commands to check the FFmpeg and GCC versions:

   ffmpeg -v 
   gcc -v

   Figure 11 Checking the FFmpeg and GCC versions
   

Dockerfile on a ModelArts Base Image

Run the vim command to create a Dockerfile. If the base image is provided by ModelArts, the content of the Dockerfile is as follows:

# Replace ${base_image_name} after FROM with the image name obtained from docker pull.
FROM swr.cn-north-4.myhuaweicloud.com/atelier/pytorch_cuda:pytorch_1.12.1-cuda_10.2-py_3.9.11-ubuntu_20.04-x86_64-20260425154604-d9a1f9f
USER root
# section1: Configure apt sources
RUN apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 3B4FE6ACC0B21F32 && \
    mv /etc/apt/sources.list /etc/apt/sources.list.bak && \
    echo -e "deb http://repo.huaweicloud.com/ubuntu/ bionic main restricted\ndeb http://repo.huaweicloud.com/ubuntu/ bionic-updates main restricted\ndeb http://repo.huaweicloud.com/ubuntu/ bionic universe\ndeb http://repo.huaweicloud.com/ubuntu/ bionic-updates universe\ndeb http://repo.huaweicloud.com/ubuntu/ bionic multiverse\ndeb http://repo.huaweicloud.com/ubuntu/ bionic-updates multiverse\ndeb http://repo.huaweicloud.com/ubuntu/ bionic-backports main restricted universe multiverse\ndeb http://repo.huaweicloud.com/ubuntu bionic-security main restricted\ndeb http://repo.huaweicloud.com/ubuntu bionic-security universe\ndeb http://repo.huaweicloud.com/ubuntu bionic-security multiverse" > /etc/apt/sources.list && \
    apt-get update
# section2: Download the FFmpeg and GCC dependency packages.
RUN apt-get -y -f install && \
    apt-get -y autoremove && \
    apt-get -y autoclean && \
    apt-get update && \ 
    echo Y | apt-get -y install openssh-server && \
    apt-get -y install ffmpeg && \
    apt-get -y install gcc-8 g++-8 && \
    update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-8 80 --slave /usr/bin/g++ g++ /usr/bin/g++-8 && \
    if [ -f $HOME/.pip/pip.conf ]; then rm $HOME/.pip/pip.conf; fi
USER ma-user
# section3: Configure sources as required. Tsinghua sources are configured in this example.
RUN echo -e "channels:\n  - defaults\nshow_channel_urls: true\ndefault_channels:\n  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main\n  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/r\n  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/msys2\ncustom_channels:\n  conda-forge: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud\n  msys2: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud\n  bioconda: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud\n  menpo: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud\n  pytorch: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud\n  pytorch-lts: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud\n  simpleitk: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud" > $HOME/.condarc && \
    echo -e "[global]\nindex-url = https://pypi.tuna.tsinghua.edu.cn/simple\n[install]\ntrusted-host = https://pypi.tuna.tsinghua.edu.cn" > $HOME/.pip/pip.conf
# section4: create a conda environment(only support python=3.11.10) and install pytorch2.7.0
# PyTorch, torchvision, and torchaudio versions must be compatible. To create a custom Python environment and use the notebook feature, download ipykernel.
RUN source /home/ma-user/anaconda3/bin/activate && \
    conda create -y --name pytorch_2_7 python=3.11.10 && \
    conda activate pytorch_2_7 && \
    pip install torch==2.7.0 torchvision==0.22.0 torchaudio==2.7.0 && \
    conda activate pytorch_2_7 && \
    pip install ipykernel && \
    
    conda deactivate

Dockerfile on a Non-ModelArts Base Image

If a third-party image is used, add user ma-user whose UID is 1000 and user group ma-group whose GID is 100 to the Dockerfile. If UID 1000 or GID 100 in the base image has been used by another user or user group, delete the user or user group. The user and user group have been added to the Dockerfile in this case. You can directly use them.

You only need to set the user ma-user whose UID is 1000 and the user group ma-group whose GID is 100, and grant the read, write, and execute permissions on the target directory to user ma-user.

Run the vim command to create a Dockerfile and add a third-party (non-ModelArts) image as the base image, for example, Ubuntu 20.04. The following provides two Dockerfile file examples for your reference.

• Example 1: Basic configuration, that is, adding a specified user to a user group.

  FROM ubuntu:20.04
  # Create ma-user UID=1000 GID=100
  USER root
  RUN default_user=$(getent passwd 1000 | awk -F ':' '{print $1}') || echo "uid: 1000 does not exist" && \
      default_group=$(getent group 100 | awk -F ':' '{print $1}') || echo "gid: 100 does not exist" && \
      if [ ! -z "${default_user}" ] && [ "${default_user}" != "ma-user" ]; then \
          userdel -r ${default_user}; \
      fi && \
      if [ ! -z "${default_group}" ] && [ "${default_group}" != "ma-group" ]; then \
          groupdel -f ${default_group}; \
      fi && \
      if ! getent group 100 > /dev/null; then \
          groupadd -g 100 ma-group; \
      fi && \
      useradd -d /home/ma-user -m -u 1000 -g 100 -s /bin/bash ma-user && \
      chmod -R 750 /home/ma-user

• Example 2: Basic configuration + configuration of related tools and environments.
  • Basic configuration: adding a specified user to a user group
  • (Optional) Example of configuring related tools and environments:
    • Configure the APT source to accelerate software package download.
    • Install Miniconda and configure the Conda and pip sources.
    • Create a Conda environment that contains specific Python packages (such as PyTorch and TorchVision).
    • Install third-party packages, for example, FFmpeg and GCC 8, to support multimedia processing and compilation tasks.

      FROM ubuntu:20.04
      # section1: Create the ma-user user group and user.
      USER root
      RUN default_user=$(getent passwd 1000 | awk -F ':' '{print $1}') || echo "uid: 1000 does not exist" && \
          default_group=$(getent group 100 | awk -F ':' '{print $1}') || echo "gid: 100 does not exist" && \
          if [ ! -z "${default_user}" ] && [ "${default_user}" != "ma-user" ]; then \
              userdel -r ${default_user}; \
          fi && \
          if [ ! -z "${default_group}" ] && [ "${default_group}" != "ma-group" ]; then \
              groupdel -f ${default_group}; \
          fi && \
          if ! getent group 100 > /dev/null; then \
              groupadd -g 100 ma-group; \
          fi && \
          useradd -d /home/ma-user -m -u 1000 -g 100 -s /bin/bash ma-user && \
          chmod -R 750 /home/ma-user
      
      # section2: Use the standard focal source.
      RUN mv /etc/apt/sources.list /etc/apt/sources.list.bak && \
          printf "deb http://archive.ubuntu.com/ubuntu/ focal main restricted universe multiverse\n\
          deb http://archive.ubuntu.com/ubuntu/ focal-updates main restricted universe multiverse\n\
          deb http://archive.ubuntu.com/ubuntu/ focal-backports main restricted universe multiverse\n\
          deb http://archive.ubuntu.com/ubuntu focal-security main restricted universe multiverse\n" > /etc/apt/sources.list && \
          apt-get update && \
          apt-get install -y zip wget ca-certificates && \ 
          apt update && echo Y | apt install openssh-server
      RUN rm /bin/sh && ln -s /bin/bash /bin/sh
      
      # section3: Install Miniconda3 as user ma-user.
      USER ma-user
      RUN cd /home/ma-user/ && \
          wget --no-check-certificate https://mirrors.tuna.tsinghua.edu.cn/anaconda/miniconda/Miniconda3-latest-Linux-x86_64.sh && \
          bash Miniconda3-latest-Linux-x86_64.sh -b -p /home/ma-user/anaconda3 && \
          rm -rf Miniconda3-latest-Linux-x86_64.sh
      
      # section4: Configure Tsinghua mirrors for conda and pip + Initialize conda
      RUN mkdir -p /home/ma-user/.pip && \
          echo -e "channels:\nshow_channel_urls: true\ndefault_channels:\n  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main\n  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/r\n  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/msys2\nremote_read_timeout: 120" > /home/ma-user/.condarc && \
          echo -e "[global]\nindex-url = https://pypi.tuna.tsinghua.edu.cn/simple\n[install]\ntrusted-host = pypi.tuna.tsinghua.edu.cn" > /home/ma-user/.pip/pip.conf && \
          /home/ma-user/anaconda3/bin/conda init bash && \
          source /home/ma-user/anaconda3/bin/activate base && \
          conda config --remove-key channels || true && \
          conda config --append channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main && \
          conda config --append channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/r && \
          conda config --append channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/msys2 && \
          conda tos accept --override-channels -c https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main && \
          conda tos accept --override-channels -c https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/r && \
          conda tos accept --override-channels -c https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/msys2
      
      # section5: Create a dedicated conda virtual environment for PyTorch. If you need to build a Python environment and use the notebook feature, download ipykernel.
      RUN /bin/bash -c '\
          source /home/ma-user/anaconda3/bin/activate base && \
          conda create -y --override-channels \
          -c https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main \
          -c https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/r \
          -c https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/msys2 \
          -n pytorch_2_7 python=3.11.10 && \
          conda run -n pytorch_2_7 \
          pip install --index-url https://pypi.tuna.tsinghua.edu.cn/simple --trusted-host pypi.tuna.tsinghua.edu.cn \
          torch==2.7.0 torchvision==0.22.0 torchaudio==2.7.0 ipykernel==6.7.0 \
      '
      
      # section6: Fix the dependency conflict, install the system package by phase, and add the command for fixing the dependency.
      USER root
      RUN apt-get update && apt-get --fix-broken install -y
      RUN apt-get install -y --no-install-recommends libc6-dev libstdc++-8-dev
      RUN apt-get install -y --no-install-recommends ffmpeg gcc-8 g++-8
      # Create symbolic links to set gcc-8 and g++-8 as the default gcc and g++.
      RUN ln -s /usr/bin/gcc-8 /usr/bin/gcc && \
          ln -s /usr/bin/g++-8 /usr/bin/g++
      RUN apt-get autoremove -y && apt-get autoclean