Creating a Custom Image in a Notebook Instance Using Dockerfile

Scenario

For AI engines that are not supported by ModelArts, you can import the models you compile to ModelArts from custom images for creating AI applications.

This section describes how to use a base image in ModelArts notebook to create an inference image, use the image to create an AI application, and deploy the application as a real-time service.

Procedure:

Step 1 Creating an Image in a Notebook Instance. For details, see Specifications for Custom Images.
Step 2 Registering the Image in Image Management.
Step 3 Changing and Debugging the Image in a Notebook Instance.
Step 4 Using a Debugged Image for Inference Deployment.

Step 1 Creating an Image in a Notebook Instance

This section uses a TensorFlow base image provided in ModelArts as an example to create an image in ModelArts notebook.

Log in to the ModelArts management console. In the navigation pane on the left, choose Permission Management. Check whether the access authorization has been configured. If not, configure access authorization by referring to Configuring Agency Authorization for ModelArts with One Click.
In the navigation pane on the left, choose Development Workspace > Notebook.
Click Create Notebook in the upper right corner. Configure the parameters on the displayed page.
1. Configure basic information of the notebook instance, including its name, description, and auto stop status.
2. Select an image and configure resource specifications for the instance.
  - Image: Select a public image that can run on CPUs, for example, pytorch1.8-cuda10.2-cudnn7-ubuntu18.04.
  - Resource Type: Select a public resource pool or a dedicated resource pool. A public resource pool is used as an example.
  - Type: GPU is recommended.
  - Flavor: GP Tnt004 is recommended.
Click Next. Confirm the information and click Submit.
Switch to the notebook instance list. The notebook instance is being created. It will take several minutes before its status changes to Running.
Access the running notebook instance.
Figure 1 Accessing a notebook instance
Click to upload the Dockerfile and model package file to the notebook instance. The default working directory of the instance is /home/ma-user/work/.
For details about the Dockerfile, see Dockerfile Template. Prepare the model package file. For details, see Sample Model Package File.

Figure 2 Uploading Dockerfile and model package file
Start the Terminal. Decompress model.zip and delete it.
```
# Decompress the ZIP file.
unzip model.zip
```
Figure 3 Decompressing model.zip on the Terminal

Open a new IPYNB file, start the image creation script, and specify the paths to the Dockerfile and image.

Figure 4 Starting the image creation script

The image creation script is as follows:

from modelarts.image_builder import ImageBuilder
from modelarts.session import Session
session = Session()

image = ImageBuilder(session=session,
   dockerfile_path="/home/ma-user/work/Dockerfile",
   image_url="custom_test/pytorch1.8:1.0.0",# custom_test is the organization name, pytorch1.8 is the image name, and 1.0.0 is the tag.
   context="/home/ma-user/work")
result = image.build_push()

Wait until the image is created. The image will be automatically pushed to SWR.

Figure 5 Image being created

Dockerfile Template

The following provides a sample Dockerfile, which can be saved as another Dockerfile. For details about the available base images, see Preset Dedicated Images for Inference.

FROM swr.cn-north-4.myhuaweicloud.com/atelier/tensorflow_2_1:tensorflow_2.1.0-cuda_10.1-py_3.7-ubuntu_18.04-x86_64-20221121111529-d65d817

# Create a soft link from /home/ma-user/anaconda/lib/python3.7/site-packages/model_service to /home/ma-user/infer/model_service, which is the built-in inference framework code directory.
# if the installed python version of this base image is python3.8, you should create a soft link from '/home/ma-user/anaconda/lib/python3.8/site-packages/model_service' to '/home/ma-user/infer/model_service'.
USER root
RUN ln -s /home/ma-user/anaconda/lib/python3.7/site-packages/model_service  /home/ma-user/infer/model_service
USER ma-user

# here we supply a demo, you can change it to your own model files
ADD model/  /home/ma-user/infer/model/1
USER root
RUN chown -R ma-user:ma-group  /home/ma-user/infer/model/1
USER ma-user

# default MODELARTS_SSL_CLIENT_VERIFY switch is "true". In order to debug, we set it to be "false"
ENV MODELARTS_SSL_CLIENT_VERIFY="false"

# change your port and protocol here, default is 8443 and https
# ENV MODELARTS_SERVICE_PORT=8080
# ENV MODELARTS_SSL_ENABLED="false"

# add pip install here
# RUN pip install numpy==1.16.4
# RUN pip install -r requirements.txt

# default cmd, you can chage it here
# CMD sh /home/ma-user/infer/run.sh

Sample Model Package File

A model file in the model package file model.zip must be prepared by yourself. The following uses a handwritten digit recognition model as an example.

The inference script file customize_service.py must be available in the model directory for model pre-processing and post-processing.

Figure 6 Model directory of the inference model

For details about the inference script customize_service.py, see Specifications for Writing a Model Inference Code File customize_service.py.

The content of the customize_service.py file used in this case is as follows:

import logging
import threading

import numpy as np
import tensorflow as tf
from PIL import Image

from model_service.tfserving_model_service import TfServingBaseService


class mnist_service(TfServingBaseService):

    def __init__(self, model_name, model_path):
        self.model_name = model_name
        self.model_path = model_path
        self.model = None
        self.predict = None

        # Load the model in saved_model format in non-blocking mode to prevent blocking timeout.
        thread = threading.Thread(target=self.load_model)
        thread.start()

    def load_model(self):
        # Load the model in saved_model format.
        self.model = tf.saved_model.load(self.model_path)

        signature_defs = self.model.signatures.keys()

        signature = []
        # only one signature allowed
        for signature_def in signature_defs:
            signature.append(signature_def)

        if len(signature) == 1:
            model_signature = signature[0]
        else:
            logging.warning("signatures more than one, use serving_default signature from %s", signature)
            model_signature = tf.saved_model.DEFAULT_SERVING_SIGNATURE_DEF_KEY

        self.predict = self.model.signatures[model_signature]

    def _preprocess(self, data):
        images = []
        for k, v in data.items():
            for file_name, file_content in v.items():
                image1 = Image.open(file_content)
                image1 = np.array(image1, dtype=np.float32)
                image1.resize((28, 28, 1))
                images.append(image1)

        images = tf.convert_to_tensor(images, dtype=tf.dtypes.float32)
        preprocessed_data = images

        return preprocessed_data

    def _inference(self, data):

        return self.predict(data)

    def _postprocess(self, data):

        return {
            "result": int(data["output"].numpy()[0].argmax())
        }

Step 2 Registering the Image in Image Management

Register the custom image created in Step 1 Creating an Image in a Notebook Instance in ModelArts image management for future use.

Log in to the ModelArts management console. In the navigation pane on the left, choose Image Management. Then, click Register on the displayed page.
Specify the image source, select the architecture and type, and click Register.
- SWR Source: The image source is swr.cn-north-4-myhuaweicloud.com/custom_test/tensorflow2.1:1.0.0, where custom_test/tensorflow2.1:1.0.0 is the image path set in the image creation script in 8.
- Architecture: Select X86_64.
- Type: Select CPU.
Figure 7 Registering an image
View the registered image on the Image Management page.

Step 3 Changing and Debugging the Image in a Notebook Instance

Debug the created custom image for inference. Then, import the debugged image to ModelArts AI applications and deploy it as a real-time service.

Log in to the ModelArts console. In the navigation pane on the left, choose Development Workspace > Notebook. Stop the notebook instance created in Step 1 Creating an Image in a Notebook Instance.
Locate the target notebook instance in the list and choose More > Change Image in the Operation column. In the displayed dialog box, set Change Image to Private image, and select the image registered in Step 2 Registering the Image in Image Management, as shown in Figure 8.
Figure 8 Changing an Image
Start the notebook instance and access it. Go to the Terminal page, run the boot script run.sh in the working directory, and run the model for prediction. In a base image, run.sh is used as the boot script by default.
Figure 9 Running the boot script
Upload an image with a handwritten digit to the notebook instance for prediction.
Figure 10 Handwritten digit

Figure 11 Uploading an image for prediction
Open a new Terminal and run the following command for prediction:
```
curl -kv -F 'images=@/home/ma-user/work/test.png' -X POST http://127.0.0.1:8080/
```
Figure 12 Prediction

If the model file or inference script file is modified during debugging, restart the run.sh script. To do so, run the following command to stop Nginx and then run the run.sh script:
```
# Obtain the Nginx process.
ps -ef |grep nginx 
# Stop all Nginx-related processes.
kill -9 {Process ID}
# Execute run.sh.
sh run.sh
```
You can also run the pkill nginx command to stop all Nginx processes.
```
# Stop all Nginx processes.
pkill nginx
# Execute run.sh.
sh run.sh
```
Figure 13 Restarting run.sh

Step 4 Using a Debugged Image for Inference Deployment

Import the custom image debugged in Step 3 Changing and Debugging the Image in a Notebook Instance to AI applications and deploy it as a real-time service.

Log in to the ModelArts console. In the navigation pane on the left, choose AI Applications. Click Create Applications on the displayed page.
Configure the parameters for the AI application.
- Meta Model Source: Select Container image.
- Container Image Path: Click and select the created custom image.
- Container API: Select HTTPS.
- host: Set to 8443.
- Deployment Type: Select Real-Time Services.
Enable APIs, edit the API, and click Save. Specify a file as the input. The following shows a code example.
Figure 14 API definition

The API definition is as follows:
```
[{
 "url": "/",
 "method": "post",
 "request": {
  "Content-type": "multipart/form-data",
  "data": {
   "type": "object",
   "properties": {
    "images": {
     "type": "file"
    }
   }
  }
 },
 "response": {
  "Content-type": "applicaton/json",
  "data": {
   "type": "object",
   "properties": {
    "result": {
     "type": "integer"
    }
   }
  }
 }
}]
```
After enabling this function, you can edit RESTful APIs to define the AI application input and output formats.
- If you edit APIs when creating an AI application, the system will automatically identify the prediction type after the created AI application is deployed.
- If you do not edit APIs when creating an AI application, you will be required to select a request type for prediction after the created AI application is deployed. The request type can be application/json or multipart/form-data. Select a proper type based on the meta model.
  - If you select application/json, enter code for text prediction.
  - If you select multipart/form-data, configure the request parameter. The request parameter value is the KEY value in the Body tab when GUI-based software (Postman as an example) is used for prediction. It is also the parameter name in the prediction request sent by running the cURL command.
After the APIs are configured, click Create now. Wait until the AI application runs properly.
Locate the created AI application in the list and click Deploy in the Operation column. In the displayed version list, locate the target version and click Real-Time Services in the Operation column.
On the Deploy page, configure the key parameters as follows:
Name: Enter a custom real-time service name or use the default name.

Resource Pool: Select a public resource pool.

AI Application Source and AI Application and Version: The AI application and version will be automatically selected.

Specifications: Select specifications with limited time offer.

Retain default settings for other parameters.

If free specifications have been sold out, use charged CPU specifications instead. Resource fees are displayed on the GUI.
After configuring the parameters, click Next, confirm parameter settings, and click Submit.
In the navigation pane on the left, choose Service Deployment > Real-Time Services. When the service status changes to Running, the service is deployed. Click Predict in the Operation column. The Prediction page on the service details page is displayed. Upload an image for prediction.
Figure 15 Prediction