Specifications for Compiling Inference Code

1. In the model inference code file customize_service.py, add a child model class. This child model class inherits properties from its parent model class. For details about the import statements of different types of parent model classes, see Table 1.

   Table 1 Import statements of different types of parent model classes

   Model Type	Parent Class	Import Statement
   TensorFlow	TfServingBaseService	from model_service.tfserving_model_service import TfServingBaseService
   PyTorch	PTServingBaseService	from model_service.pytorch_model_service import PTServingBaseService
   MindSpore	SingleNodeService	from model_service.model_service import SingleNodeService

2. The following methods can be rewritten:

   Table 2 Methods to be rewritten

   Method	Description
   __init__(self, model_name, model_path)	Initialization method, which is suitable for models created based on deep learning frameworks. Models and labels are loaded using this method. This method must be rewritten for models based on PyTorch and Caffe to implement the model loading logic.
   __init__(self, model_path)	Initialization method, which is suitable for models created based on machine learning frameworks. The model path (self.model_path) is initialized using this method. In Spark_MLlib, this method also initializes SparkSession (self.spark).
   _preprocess(self, data)	Preprocess method, which is called before an inference request and is used to convert the original request data of an API into the expected input data of a model
   _inference(self, data)	Inference request method. You are advised not to rewrite the method because once the method is rewritten, the built-in inference process of ModelArts will be overwritten and the custom inference logic will run.
   _postprocess(self, data)	Postprocess method, which is called after an inference request is complete and is used to convert the model output to the API output

   

   • You can choose to rewrite the preprocess and postprocess methods to implement preprocessing of the API input and postprocessing of the inference output.
   • Rewriting the init method of the parent model class may cause an AI application to run abnormally.
3. The attribute that can be used is the local path where the model resides. The attribute name is self.model_path. In addition, PySpark-based models can use self.spark to obtain the SparkSession object in customize_service.py.
   

   An absolute path is required for reading files in the inference code. You can obtain the local path of the model from the self.model_path attribute.

   • When TensorFlow, Caffe, or MXNet is used, self.model_path indicates the path of the model file. See the following example:

     # Store the label.json file in the model directory. The following information is read:
     with open(os.path.join(self.model_path, 'label.json')) as f:
         self.label = json.load(f)

   • When PyTorch, Scikit_Learn, or PySpark is used, self.model_path indicates the path of the model file. See the following example:

     # Store the label.json file in the model directory. The following information is read:
     dir_path = os.path.dirname(os.path.realpath(self.model_path))
     with open(os.path.join(dir_path, 'label.json')) as f:
         self.label = json.load(f)

4. data imported through the API for pre-processing, actual inference request, and post-processing can be multipart/form-data or application/json.
   • multipart/form-data request

     curl -X POST \
       <modelarts-inference-endpoint> \
       -F image1=@cat.jpg \
       -F images2=@horse.jpg

     The corresponding input data is as follows:

     [
        {
           "image1":{
              "cat.jpg":"<cat.jpg file io>"
           }
        },
        {
           "image2":{
              "horse.jpg":"<horse.jpg file io>"
           }
        }
     ]

   • application/json request

      curl -X POST \
        <modelarts-inference-endpoint> \
        -d '{
         "images":"base64 encode image"
         }'

     The corresponding input data is python dict.

      {
         "images":"base64 encode image"
      }

TensorFlow Inference Script Example

The preceding code example resizes images imported to the user's form to adapt to the model input shape. The 32×32 image is read from the Pillow library and resized to 1×784 to match the model input. In subsequent processing, convert the model output into a list for the RESTful API to display.

XGBoost Inference Script Example

For details about the inference code of other machine learning engines, see PySpark and Scikit-learn.

# coding:utf-8
import collections
import json
import xgboost as xgb
from model_service.python_model_service import XgSklServingBaseService


class UserService(XgSklServingBaseService):

    # request data preprocess
    def _preprocess(self, data):
        list_data = []
        json_data = json.loads(data, object_pairs_hook=collections.OrderedDict)
        for element in json_data["data"]["req_data"]:
            array = []
            for each in element:
                array.append(element[each])
                list_data.append(array)
        return list_data

    #   predict
    def _inference(self, data):
        xg_model = xgb.Booster(model_file=self.model_path)
        pre_data = xgb.DMatrix(data)
        pre_result = xg_model.predict(pre_data)
        pre_result = pre_result.tolist()
        return pre_result

    # predict result process
    def _postprocess(self, data):
        resp_data = []
        for element in data:
            resp_data.append({"predict_result": element})
        return resp_data

Inference Script Example of the Custom Inference Logic

Customize a dependency package in the configuration file by referring to Example of a Model Configuration File Using a Custom Dependency Package. Then, use the following code example to load the model in saved_model format for inference.

The logging module of Python used by the base inference image uses the default log level Warning. Only warning logs can be queried by default. To query INFO logs, set the log level to INFO in the code.

MindSpore Inference Script Example

• For details about how to convert a model, see Model Conversion. The inference script is as follows:

  from __future__ import absolute_import
  from __future__ import division
  from __future__ import print_function
  
  import json
  import os
  
  import numpy as np
  from PIL import Image
  from hiai.nn_tensor_lib import NNTensor
  from hiai.nntensor_list import NNTensorList
  from model_service.hiai_model_service import HiaiBaseService
  
  class DemoService(HiaiBaseService):
    def __init__(self, *args, **kwargs):
  	# The .om file in the model package directory is loaded by default.
  	super(DemoService, self).__init__(*args, **kwargs)
  	self.labels_list = None
  	self.is_multilabel = False
  
    def _preprocess(self, data):
  	preprocessed_data = {}
  	images = []
  	for k, v in data.items():
  	  for file_name, file_content in v.items():
  	  image = Image.open(file_content)
  	  image = np.array(image)  # NHWC
  	  # AIPP should use RGB format.
  	  # mean reg is applied in AIPP.
  	  # Transpose is applied in AIPP
  	  tensor = NNTensor(image)
  	  images.append(tensor)
  	  tensor_list = NNTensorList(images)
  	preprocessed_data['images'] = tensor_list
  	return preprocessed_data
  
    def _inference(self, data, image_info=None):
  	result = {}
  	for k, v in data.items():
  	  result[k] = self.model.proc(v)
  	return result
  
    def _postprocess(self, data):
  	# Add post-processing.
  	return str(data)