Creating and Uploading a Custom Image

In this example, the Dockerfile file is used to customize an image.

A Linux x86_x64 host is used here. You can purchase an ECS of the same specifications or use an existing local host to create a custom image.

1. Install Docker. For details, see https://docs.docker.com/engine/install/binaries/#install-static-binaries.

   The following uses the Linux x86_64 architecture as an example to describe how to obtain the Docker 19.03.2 installation package. Run the following command to install the Docker software:

   tar -xzvf docker-19.03.2.tgz
   cp docker/* /usr/bin/

   If the docker images command is successfully executed, Docker has been installed. In this case, skip this step.

2. Obtain a basic image.

   A custom image used for a training job must be compiled based on a basic image. For details about the formats of basic image names, see Basic Image Package. Run the following command to obtain a basic image for custom images:

   docker pull swr.<region>.myhuaweicloud.com/<image org>/custom-<processor type>-[<cuda version>]-base:<image tag>

   In addition, you can run the docker images command to view the local image list.

3. (Optional) Modify the basic image for debugging. If you need to modify the PyPi mirror and apt source based on the network status, perform the following steps to modify the file content in the container:

   The following uses the basic image swr.cn-north-4.myhuaweicloud.com/modelarts-job-dev-image/custom-cpu-base:1.3 as an example.

   1. Run the following command to start the container and allocate a pseudo TTY to connect to the stdin of the container:

      docker run -ti ${Container image} bash

   2. The context is in the container.
      1. Configure the PyPi mirror in the basic image based on the network status to accelerate the download of Python packages. The PyPi mirror is set to the one provided by Huawei DevCloud by default. You can run the following command in the container to view the PyPi mirror. If you want to use another PyPi mirror, change the value of index-url in the command to the value of your desired PyPi mirror.

         cat /root/.pip/pip.conf
         [global] 
         index-url = http://repo.myhuaweicloud.com/repository/pypi/simple 
         format = columns 
         [install] 
         trusted-host=repo.myhuaweicloud.com

      2. Configure the Ubuntu16.04 apt source in the basic image based on the network status to accelerate the download of apt packages. The apt source in the basic image is provided by Ubuntu 16.04. Run the following command in the container to change the default source to the apt source of DevCloud:

         If you want to use another apt source, change http://repo.myhuaweicloud.com in the command to your desired source.

         cp -a /etc/apt/sources.list /etc/apt/sources.list.bak
         sed -i "s@http://.*archive.ubuntu.com@http://repo.myhuaweicloud.com@g"  /etc/apt/sources.list 
         sed -i "s@http://.*security.ubuntu.com@http://repo.myhuaweicloud.com@g"  /etc/apt/sources.list

   3. After the modification, run the following command to exit the pseudo TTY:

      exit

   4. The context is in the host. Run the following command to save the modified container as an image:

      docker commit ${container-id} swr.cn-north-4.myhuaweicloud.com/modelarts-job-dev-image/custom-cpu-base:1.3-updated

      In the preceding command, ${container-id} indicates the ID of the modified container, which can be obtained by running the docker ps -a command. The modified container is named swr.cn-north-4.myhuaweicloud.com/modelarts-job-dev-image/custom-cpu-base:1.3-updated. If you need to use the image of the modified container for build tasks, use the new container image name.

   5. Delete the container that has exited.

      docker rm ${container-id}

4. Compile a Dockerfile for building a custom Python 3.6.5 + TensorFlow 1.13.2 image.

   The file name is tf-1.13.2.dockerfile. In this Dockerfile example, Minicoda and Python are reinstalled. Therefore, you need to reinstall the Python dependency package. The basic image contains the following default Python packages:

   • boto3 (1.7.29)
   • botocore (1.10.81)
   • docutils (0.14)
   • futures (3.2.0)
   • jmespath (0.9.3)
   • netifaces (0.10.7)
   • python-dateutil (2.7.5)
   • pyzmq (17.1.2)
   • s3transfer (0.1.13)
   • setuptools (20.7.0)
   • six (1.11.0)
   • wheel (0.29.0)
   In this example, Boto3 is installed. You can add the installation of other dependency packages based on site requirements. For details about how to compile the Dockerfile file, see the Dockerfile Reference.

   FROM swr.cn-north-4.myhuaweicloud.com/modelarts-job-dev-image/custom-cpu-base:1.3
   RUN mkdir -p /tmp/install && cd /tmp/install && \
   curl -o Miniconda3-4.5.4-Linux-x86_64.sh -k https://repo.anaconda.com/miniconda/Miniconda3-4.5.4-Linux-x86_64.sh && \
   bash Miniconda3-4.5.4-Linux-x86_64.sh -b && \
   root/miniconda3/bin/pip install tensorflow==1.13.2 && \
   pip install boto3==1.7.29 && \
   rm -rf /tmp/install
   
   ENV PATH=/root/miniconda3/bin/:$PATH

5. Create a custom image.

   In the following example, the image is in the cn-north-4 region and belongs to the deep-learning-diy organization. Run the following command in the directory where the tf-1.13.2.dockerfile file resides:

   1	docker build -f tf-1.13.2.dockerfile . -t swr.cn-north-4.myhuaweicloud.com/deep-learning-diy/tf-1.13.2:latest

6. Push the image to SWR. For details about how to upload an image, see the Software Repository for Container User Guide.

   The prerequisite is that you have created an organization and obtained the SWR login command. In the following example, the image is located in the cn-north-4 region and belongs to the deep-learning-diy organization. Run the following command to push the image to SWR:

   1	docker push swr.cn-north-4.myhuaweicloud.com/deep-learning-diy/tf-1.13.2:latest

   swr.cn-north-4.myhuaweicloud.com/deep-learning-diy/tf-1.13.2:latest is the SWR URL of the custom image.

Standalone Training

1. Upload training code mnist_softmax.py and training data to OBS. Store the code and data in the code root directory so that they can be directly downloaded to the container.

   The root directory obs://deep-learning/new/mnist/ is used as an example.

   The training code file is obs://deep-learning/new/mnist/.

   The data is stored in obs://deep-learning/new/mnist/minist_data.

2. Create a training job using a custom image. Set Data Storage Location and Training Output Path based on site requirements. Set Image Path, Code Directory, and Boot Command as follows:
   • Image Path: Enter the SWR URL of the uploaded image.
   • Code Directory: Enter the OBS path for storing the training code, that is, the code root directory in 1.

     Before a training job is started, ModelArts automatically recursively downloads all content in the code directory to the local path of the container. The local path of the container is /home/work/user-job-dir/${Last level of the code root directory}/. For example, if Code Directory is set to obs://deep-learning/new/mnist, the local path is /home/work/user-job-dir/mnist/, and the code boot file is /home/work/user-job-dir/mnist/mnist_softmax.py.

   • Boot Command: bash /home/work/run_train.sh python /home/work/user-job-dir/mnist/mnist_softmax.py --data_url /home/work/user-job-dir/mnist/mnist_data

     /home/work/user-job-dir/mnist/mnist_softmax.py is the code boot file, and --data_url /home/work/user-job-dir/mnist/mnist_data is the data storage path.

3. After the training job is created, the code directory is downloaded, the custom image is reviewed, and the training job is completed in the background. Generally, training jobs run for a period of time, which may be several minutes or tens of minutes depending on the amount of data and resources you select. After the program is executed successfully, the log similar to the following is outputted:
   Figure 1 Run log information