Specifications for Writing a Model Inference Code File

Specifications for Writing Inference Code

1. In the model inference code file customize_service.py, add a child model class. This child model class inherits properties from its parent model class. For details about the import statements of different types of parent model classes, see Table 1. The ModelArts environment has already configured the necessary Python packages for import statements, so you do not need to install them separately.

   Table 1 Parent class and import statement of each model type

   Model Type	Parent Class	Import Statement
   TensorFlow	TfServingBaseService	from model_service.tfserving_model_service import TfServingBaseService
   PyTorch	PTServingBaseService	from model_service.pytorch_model_service import PTServingBaseService
   MindSpore	SingleNodeService	from model_service.model_service import SingleNodeService

2. The following methods can be overridden.

   Table 2 Methods to be overridden

   Method	Description
   __init__(self, model_name, model_path)	Initialization method, which is suitable for models created based on deep learning frameworks. Models and labels are loaded using this method. To implement model loading logic, override this method for PyTorch and Caffe-based models.
   __init__(self, model_path)	Initialization method, which is suitable for models created based on machine learning frameworks. This method initializes the model path (self.model_path). In Spark_MLlib, this method also initializes SparkSession (self.spark).
   _preprocess(self, data)	Preprocess method, which is called before an inference request and converts API request data into the model's expected input format.
   _inference(self, data)	Inference request method. You are advised not to override this method, as it will replace the built-in inference process in ModelArts with your custom logic.
   _postprocess(self, data)	Postprocess method, which is called after an inference request is complete and converts the model output to the API output.

   

   • You can override the preprocess and postprocess methods for preprocessing the API input and postprocessing the inference output.
   • Overriding the init method of the parent model class may cause an AI application to run abnormally.
3. The attribute that can be used is the local path to the model. The attribute name is self.model_path. Additionally, PySpark-based models can use self.spark to obtain the SparkSession object in customize_service.py.
   

   The inference code requires an absolute file path for reading files. You can obtain the local path to the model from the self.model_path attribute.

   • When TensorFlow, Caffe, or MXNet is used, self.model_path indicates the path to the model file. The following provides an example:

     # Reads the label.json file in the model directory.
     with open(os.path.join(self.model_path, 'label.json')) as f:
         self.label = json.load(f)

   • When PyTorch, Scikit_Learn, or PySpark is used, self.model_path indicates the path to the model file. The following provides an example:

     # Reads the label.json file in the model directory.
     dir_path = os.path.dirname(os.path.realpath(self.model_path))
     with open(os.path.join(dir_path, 'label.json')) as f:
         self.label = json.load(f)

4. The API accepts data in either multipart/form-data or application/json format for pre-processing, actual inference, and post-processing.
   • multipart/form-data request

     curl -X POST \
       <modelarts-inference-endpoint> \
       -F image1=@cat.jpg \
       -F images2=@horse.jpg

     The input data is as follows:

     [
        {
           "image1":{
              "cat.jpg":"<cat.jpg file io>"
           }
        },
        {
           "image2":{
              "horse.jpg":"<horse.jpg file io>"
           }
        }
     ]

   • application/json request

      curl -X POST \
        <modelarts-inference-endpoint> \
        -d '{
         "images":"base64 encode image"
         }'

     The input data is python dict.

      {
         "images":"base64 encode image"
      }

TensorFlow Inference Script Example

The preceding sample code resizes images imported to the user's form to adapt to the model input shape. The 32×32 image is read from the Pillow library and resized to 1×784 to match the model input. In subsequent processing, convert the model output into a list for the RESTful API to display.

Inference Script Example of Custom Inference Logic

Customize a dependency package in the configuration file by referring to Example of the Model Configuration File Using a Custom Dependency Package. Then, use the following code example to load the model in saved_model format for inference.

Python logging used by base inference images allows the display of only warning logs. To query INFO logs, set the log level to INFO in the code.