Creating a Custom Training Image (PyTorch + Ascend)

This section describes how to create an image and use it for training on ModelArts. The AI engine used in the image is PyTorch, and the resources used for training are powered by Ascend in a dedicated resource pool.

Prerequisites

A Linux virtual or physical server with Docker 18.09.7 or later versions installed is available. The server can access the Internet and function as an image creation node.

Run the docker pull, apt-get update/upgrade, and pip install commands to check whether the node can access an external open-source software repository. If so, the node can access the Internet.

The preceding servers must be Arm64-powered.
It is a good practice to install Ubuntu 18.04 on the image creation node.
In this section, the /opt directory is used for the image creation task. Ensure that the available storage of this directory is greater than 30 GB.
For details about how to install Docker, see Install Docker Engine on Ubuntu. Miniconda and TFLite installation packages are provided by third parties. ModelArts is not responsible for their security issues. If you have security requirements, harden the security of these packages, release them as files with the same names, and upload them to the image creation node.

Creating a Custom Image

Check the Docker engine version.
```
docker version | grep -A 1 Engine
```
The command output is as follows:
```
Engine:   
 Version: 18.09.0
```
Use the Docker engine of the preceding version or later to create a custom image.
Create a folder named context.
```
mkdir -p context
```

Obtain the pip.conf file. In this example, the pip source provided by Huawei Mirrors is used, which is as follows:

[global]
index-url = 
https://repo.huaweicloud.com/repository/pypi/simple
trusted-host = 
repo.huaweicloud.com
timeout = 120

Obtain the APT source file Ubuntu-Ports-bionic.list. In this example, the APT source provided at Huawei Mirrors is used. Run the following command to obtain the APT source file:
```
wget -O Ubuntu-Ports-bionic.list --no-check-certificate 
https://repo.huaweicloud.com/repository/conf/Ubuntu-Ports-bionic.list
```
Download the Ascend-cann-nnae_7.0.0_linux-aarch64.run, torch-2.1.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl, and torch_npu-2.1.0.post7-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl installation files.
- Download the Ascend-cann-nnae_7.0.0_linux-aarch64.run file. Click the following link based on your user type. Search for CANN7 for version and click the CANN7.0.0 link. On the displayed page, search for Ascend-cann-nnae_7.0.0_linux-aarch64.run and download it.
  - For enterprise users, click here for download.
  - For carrier users, click here for download.
- Click here to download the torch-2.1.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl file.
- Click here to download the torch_npu-2.1.0.post7-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl file.
ModelArts supports only the commercial CANN edition.
Download the Miniconda3 installation file.

Click here to download the Miniconda3-py39_24.5.0-0 installation file (for Python 3.9).

Download Python of a different version from Miniconda3 File List. The MindSpore version must correspond to the Python version.

Store the pip source file, .list file, .run file, .whl file, and Miniconda3 installation file in the context folder, which is as follows:

context 
├── Ascend-cann-nnae_7.0.0_linux-aarch64.run 
├── torch-2.1.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 
├── Miniconda3-py39_24.5.0-0-Linux-aarch64.sh 
├── torch_npu-2.1.0.post7-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
├── pip.conf 
└── Ubuntu-Ports-bionic.list

Write the Dockerfile of the container image.

Create an empty file named Dockerfile in the context folder and copy the following content to the file:

# The server on which the container image is created must access the Internet.
FROM ubuntu:18.04 AS builder

# The default user of the base container image is root.
# USER root

# Install OS dependencies.
COPY Ubuntu-Ports-bionic.list /tmp
RUN cp -a /etc/apt/sources.list /etc/apt/sources.list.bak && \
    mv /tmp/Ubuntu-Ports-bionic.list /etc/apt/sources.list && \
    echo > /etc/apt/apt.conf.d/00skip-verify-peer.conf "Acquire { https::Verify-Peer false }" && \
    apt-get update && \
    apt-get install -y \
    # utils
    ca-certificates vim curl \
    # CANN 7.0.0
    gcc g++ make cmake zlib1g zlib1g-dev openssl libsqlite3-dev libssl-dev libffi-dev unzip pciutils net-tools libblas-dev gfortran libblas3 libopenblas-dev \
    # MindSpore 2.2.0
    libgmp-dev && \
    apt-get clean && \
    mv /etc/apt/sources.list.bak /etc/apt/sources.list && \
    # Grant the write permission of the parent directory of the CANN 7.0.0 installation directory to ma-user.
    chmod o+w /usr/local

RUN useradd -m -d /home/ma-user -s /bin/bash -g 100 -u 1000 ma-user

# Configure the default user and working directory of the container image.
USER ma-user
WORKDIR /home/ma-user

# Use the PyPI configuration obtained at the open-source image website.
RUN mkdir -p /home/ma-user/.pip/
COPY --chown=ma-user:100 pip.conf /home/ma-user/.pip/pip.conf

# Copy the installation files to the /tmp directory in the base container image.
COPY --chown=ma-user:100 Miniconda3-py39_24.5.0-0-Linux-aarch64.sh /tmp

# https://conda.io/projects/conda/en/latest/user-guide/install/linux.html#installing-on-linux
# Install Miniconda3 in the /home/ma-user/miniconda3 directory of the base container image.
RUN bash /tmp/Miniconda3-py39_24.5.0-0-Linux-aarch64.sh -b -p /home/ma-user/miniconda3

ENV PATH=$PATH:/home/ma-user/miniconda3/bin

# Install the CANN 7.0.0 Python dependency package.
RUN pip install numpy~=1.19.2 decorator~=4.4.0 sympy~=1.5.1 cffi~=1.12.3 protobuf~=3.13.0 \
    attrs pyyaml pathlib2 scipy requests psutil absl-py

# Install CANN 7.0.0 in /usr/local/Ascend.
COPY --chown=ma-user:100 Ascend-cann-nnae_7.0.0_linux-aarch64.run /tmp
RUN chmod +x /tmp/Ascend-cann-nnae_7.0.0_linux-aarch64.run && \
    echo Y|/tmp/Ascend-cann-nnae_7.0.0_linux-aarch64.run --install --install-path=/usr/local/Ascend

# Install PyTorch 2.1.0.
COPY --chown=ma-user:100 torch-2.1.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl /tmp
RUN chmod +x /tmp/torch-2.1.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl && \
    pip install /tmp/torch-2.1.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl

# Install touch-npu.
COPY --chown=ma-user:100 torch_npu-2.1.0.post7-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl /tmp
RUN chmod +x /tmp/torch_npu-2.1.0.post7-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl && \
    pip install /tmp/torch_npu-2.1.0.post7-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl

# Create the container image.
FROM ubuntu:18.04

# Install OS dependencies.
COPY Ubuntu-Ports-bionic.list /tmp
RUN cp -a /etc/apt/sources.list /etc/apt/sources.list.bak && \
    mv /tmp/Ubuntu-Ports-bionic.list /etc/apt/sources.list && \
    echo > /etc/apt/apt.conf.d/00skip-verify-peer.conf "Acquire { https::Verify-Peer false }" && \
    apt-get update && \
    apt-get install -y \
    # utils
    ca-certificates vim curl \
    # CANN 7.0.RC1
    gcc g++ make cmake zlib1g zlib1g-dev openssl libsqlite3-dev libssl-dev libffi-dev unzip pciutils net-tools libblas-dev gfortran libblas3 libopenblas-dev \
    # MindSpore 2.2.0
    libgmp-dev && \
    apt-get clean && \
    mv /etc/apt/sources.list.bak /etc/apt/sources.list

RUN useradd -m -d /home/ma-user -s /bin/bash -g 100 -u 1000 ma-user

# Copy the directories from the builder stage to the directories with the same name in the current container image.
COPY --chown=ma-user:100 --from=builder /home/ma-user/miniconda3 /home/ma-user/miniconda3
COPY --chown=ma-user:100 --from=builder /home/ma-user/Ascend /home/ma-user/Ascend
COPY --chown=ma-user:100 --from=builder /home/ma-user/var /home/ma-user/var
COPY --chown=ma-user:100 --from=builder /usr/local/Ascend /usr/local/Ascend

# Configure the preset environment variables of the container image.
# Configure CANN environment variables.
# Configure Ascend driver environment variables.
# Set PYTHONUNBUFFERED to 1 to prevent log loss.
ENV PATH=$PATH:/usr/local/Ascend/nnae/latest/bin:/usr/local/Ascend/nnae/latest/compiler/ccec_compiler/bin:/home/ma-user/miniconda3/bin \
    LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/Ascend/driver/lib64:/usr/local/Ascend/driver/lib64/common:/usr/local/Ascend/driver/lib64/driver:/usr/local/Ascend/nnae/latest/lib64:/usr/local/Ascend/nnae/latest/lib64/plugin/opskernel:/usr/local/Ascend/nnae/latest/lib64/plugin/nnengine \
    PYTHONPATH=$PYTHONPATH:/usr/local/Ascend/nnae/latest/python/site-packages:/usr/local/Ascend/nnae/latest/opp/op_impl/built-in/ai_core/tbe \
    ASCEND_AICPU_PATH=$ASCEND_AICPU_PATH:/usr/local/Ascend/nnae/latest \
    ASCEND_OPP_PATH=$ASCEND_OPP_PATH:/usr/local/Ascend/nnae/latest/opp \
    ASCEND_HOME_PATH=$ASCEND_HOME_PATH:/usr/local/Ascend/nnae/latest \
    PYTHONUNBUFFERED=1

# Configure the default user and working directory of the container image.
USER ma-user
WORKDIR /home/ma-user

For details about how to write a Dockerfile, see official Docker documents.

Verify that the Dockerfile has been created. The following shows the context folder:

context
├── Ascend-cann-nnae_7.0.0_linux-aarch64.run
├── Dockerfile
├── torch-2.1.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
├── torch_npu-2.1.0.post7-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
├── Miniconda3-py39_24.5.0-0-Linux-aarch64.sh
├── pip.conf
└── Ubuntu-Ports-bionic.list

Create the container image. Run the following command in the directory where the Dockerfile is stored to create a container image:
```
docker build . -t pytorch:2.1.0-cann7.0.0
```
If "connection refused" or "Client.Timeout exceeded" is reported when you access https://registry-1.docker.io/v2/ during image creation, configure the Docker proxy.

vi /etc/docker/daemon.json

Add the following content to the file and save the file:

{

"registry-mirrors":[

"https://docker.m.daocloud.io",

"https://docker.jianmuhub.com",

"https://huecker.io",

"https://dockerhub.timeweb.cloud",

"https://dockerhub1.beget.com",

"https://noohub.ru"]

}

Run the systemctl daemon-reload and systemctl restart docker commands in sequence.

Recreate
The following log shows that the image has been created.
```
Successfully tagged pytorch:2.1.0-cann7.0.0
```

Uploading an Image to SWR

Log in to the SWR console and select a region. It must share the same region with ModelArts. Otherwise, the image cannot be selected.
Click Create Organization in the upper right corner and enter an organization name to create an organization. Customize the organization name. Replace the organization name deep-learning in subsequent commands with the actual organization name.
Click Generate Login Command in the upper right corner to obtain the login command. In this example, the temporary login command is copied.
Log in to the local environment as user root and enter the copied temporary login command.

Upload the image to SWR.

Run the docker tag command to add tags to the uploaded image:

# Replace the region, domain, as well as organization name deep-learning with the actual values.
sudo docker tag pytorch:2.1.0-cann7.0.0 swr.{region-id}.{domain}/deep-learning/pytorch:2.1.0-cann7.0.0

Run the following command to upload the image:

# Replace the region, domain, as well as organization name deep-learning with the actual values.
sudo docker push swr.{region-id}.{domain}/deep-learning/pytorch:2.1.0-cann7.0.0

After the image is uploaded, choose My Images in navigation pane on the left of the SWR console to view the uploaded custom images.

Creating a Training Job on ModelArts

Log in to the ModelArts management console, check whether access authorization has been configured for your account. For details, see Configuring Agency Authorization for ModelArts with One Click. If you have been authorized using access keys, clear the authorization and configure agency authorization.
In the navigation pane on the left, choose Model Training > Training Jobs. Then, click Create Training Job.
On the displayed page, configure parameters and click Submit.
- Algorithm Type: Custom algorithm
- Boot Mode: Custom image
- Image: swr.cn-north-4.myhuaweicloud.com/deep-learning/pytorch:2.1.0-cann7.0.0
- Code Directory: directory where the boot script file is stored in OBS, for example, obs://test-modelarts/pytorch/demo-code/. The training code is automatically downloaded to the ${MA_JOB_DIR}/demo-code directory of the training container. demo-code (customizable) is the last-level directory of the OBS path.
- Boot Command: /home/ma-user/miniconda3/bin/python ${MA_JOB_DIR}/demo-code/pytorch-verification.py. demo-code (customizable) is the last-level directory of the OBS path.
- Resource Pool: Dedicated resource pool
- Resource Type: Ascend with the required driver and firmware version
- Job Log Path: OBS path to stored training logs, for example, obs://test-modelarts/pytorch/log/.
Confirm the configurations of the training job and click Submit.
Wait until the training job is created.
After you submit the job creation request, the system will automatically perform operations on the backend, such as downloading the container image and code directory and running the boot command. A training job requires a certain period of time for running. The duration ranges from dozens of minutes to several hours, depending on the service logic and selected resources. You can view log information on the job details page.