Operator Package Development Specifications

Python Operator Package Directory Specifications

Assume that the operator package name is video_clip.tar. The directory structure after the operator package is decompressed is as follows:

+--- video_clip # The directory name must be the same as the tar package name.
| +--- program_package # Python operator directory
| | +--- install.sh # (Optional) Installation script
| | +--- process.py # (Mandatory) Operator code

process.py File Development Specifications

The operator package must contain a script named process.py. The development modes are determined based on the value of auto-data-loading in the operator configuration file.

Mode 1 (auto-data-loading: true)

Applicable scenarios: This mode is recommended except for the following three scenarios:

1. Multimodal dataset: For example, datasets consisting of images and text, or videos and text.
2. The output dataset modality is not within the following range: text, image, video, and audio.
3. Scenario where the entire dataset needs to be used as the input, for example, deduplication operators.

The development specifications of process.py are as follows. The process.py file contains three classes.

1. PreProcess: (optional) operator preprocessing logic.

Before model inference, the operator offloads certain computations from the CPU to the GPU/NPU. Separating the operator preprocessing logic from CPU and GPU/NPU computations enhances GPU/NPU utilization.

2. Process: (mandatory) operator inference logic.

It is recommended that only the model loading and inference parts be included, and the preprocessing and postprocessing be written in PreProcess and PostProcess.

3. PostProcess: (optional) operator post-processing logic.

If the operator has heavy postprocessing logic that is still calculated on the CPU after inference, you are advised to split the logic and write the postprocessing logic in PostProcess.

The operator framework is called in the following sequence: preprocess -> process -> postprocess.

import pandas as pd
import ma_utils as utils
logger = utils.FileLogger.get_logger()

class PreProcess():
def __init__(self, args):
"""
:param args: operator parameters, including the service parameters of the operator and the default parameters obs_input_path (input OBS path) and obs_output_path (output OBS path).
"""
pass

def __call__(self, input: pd.DataFrame) -> pd.DataFrame:
"""
:param input: input parameter
Text-JSONL/CSV file:
The framework extracts the sample content of JSONL/CSV files and transfers the content to the operator. The operator is called once for each JSONL/CSV file. The field names in the DataFrame are the same as those defined in the JSONL/CSV files.
Text-other file:
The framework obtains the dataset file list and transfers the list to the operator. The operator is called once for each file. The field names in the DataFrame are file_path (full path of the input data on the operator, which is downloaded by the operator framework to the local directory of the operator) and file_name (data file name).
 
Image/Video/Audio-original file:
The framework obtains the dataset file list and transfers the list to the operator. The operator is called once for each file. The field names in the DataFrame are file_path (full path of the input data on the operator, which is downloaded by the operator framework to the local directory of the operator) and file_name (data file name).
 
Image/Video/Audio-Parquet file:
The framework extracts the sample content of Parquet files and transfers the content to the operator. The operator is called once for each Parquet file. The field names in the DataFrame are the same as those defined in the Parquet files.
There are two system predefined fields: file_path (full path of the input data on the operator (the operator framework downloads the dataset to the local directory of the operator)),
and file_name (file name of the data file (relative path of the file)).
:return: output parameter, data sample list DataFrame output by the operator, which is processed by the framework.
"""
pass


class Process():
def __init__(self, args):
"""
:param args: operator parameters, including the service parameters of the operator and the default parameters obs_input_path (input OBS path) and obs_output_path (output OBS path).
"""
pass

def __call__(self, input: pd.DataFrame) -> pd.DataFrame:
"""
:param input: input parameter
Text-JSONL/CSV file:
The framework extracts the sample content of JSONL/CSV files and transfers the content to the operator. The operator is called once for each JSONL/CSV file. The field names in the DataFrame are the same as those defined in the JSONL/CSV files.
Text-other file:
The framework obtains the dataset file list and transfers the list to the operator. The operator is called once for each file. The field names in the DataFrame are file_path (full path of the input data on the operator, which is downloaded by the operator framework to the local directory of the operator) and file_name (data file name).
 
Image/Video/Audio-original file:
The framework obtains the dataset file list and transfers the list to the operator. The operator is called once for each file. The field names in the DataFrame are file_path (full path of the input data on the operator, which is downloaded by the operator framework to the local directory of the operator) and file_name (data file name).
 
Image/Video/Audio-Parquet file:
The framework extracts the sample content of Parquet files and transfers the content to the operator. The operator is called once for each Parquet file. The field names in the DataFrame are the same as those defined in the Parquet files.
There are two system predefined fields: file_path (full path of the input data on the operator (the operator framework downloads the dataset to the local directory of the operator)),
and file_name (file name of the data file (relative path of the file)).
:return: output parameter, data sample list DataFrame output by the operator, which is processed by the framework.
"""
pass


class PostProcess():
def __init__(self, args):
"""
:param args: operator parameters, including the service parameters of the operator and the default parameters obs_input_path (input OBS path) and obs_output_path (output OBS path).
"""
pass

def __call__(self, input: pd.DataFrame) -> pd.DataFrame:
"""
:param input: input parameter
Text-JSONL/CSV file:
The framework extracts the sample content of JSONL/CSV files and transfers the content to the operator. The operator is called once for each JSONL/CSV file. The field names in the DataFrame are the same as those defined in the JSONL/CSV files.
Text-other file:
The framework obtains the dataset file list and transfers the list to the operator. The operator is called once for each file. The field names in the DataFrame are file_path (full path of the input data on the operator, which is downloaded by the operator framework to the local directory of the operator) and file_name (data file name).
 
Image/Video/Audio-original file:
The framework obtains the dataset file list and transfers the list to the operator. The operator is called once for each file. The field names in the DataFrame are file_path (full path of the input data on the operator, which is downloaded by the operator framework to the local directory of the operator) and file_name (data file name).
 
Image/Video/Audio-Parquet file:
The framework extracts the sample content of Parquet files and transfers the content to the operator. The operator is called once for each Parquet file. The field names in the DataFrame are the same as those defined in the Parquet files.
There are two system predefined fields: file_path (full path of the input data on the operator (the operator framework downloads the dataset to the local directory of the operator)),
and file_name (file name of the data file (relative path of the file)).
:return: output parameter, data sample list DataFrame output by the operator, which is processed by the framework.
"""
pass

Mode 2 (auto-data-loading: false)

This operation is applicable to the following scenarios:

1. Multimodal dataset: For example, datasets consisting of images and text, or videos and text.
2. The output dataset modality is not within the following range: text, image, video, and audio.

3. Scenario where the entire dataset needs to be used as the input, for example, deduplication operators.

The development specifications of process.py are as follows. The process.py file contains three classes.

1. PreProcess: (optional) operator preprocessing logic.

Before model inference, the operator offloads certain computations from the CPU to the GPU/NPU. Separating the operator preprocessing logic from CPU and GPU/NPU computations enhances GPU/NPU utilization.

2. Process: (mandatory) operator inference logic.

It is recommended that only the model loading and inference parts be included, and the preprocessing and postprocessing be written in PreProcess and PostProcess.

3. PostProcess: (optional) operator post-processing logic.

If the operator has heavy postprocessing logic that is still calculated on the CPU after inference, you are advised to split the logic and write the postprocessing logic in PostProcess.

The operator framework is called in the following sequence: preprocess -> process -> postprocess.

import pandas as pd

class Process():
def __init__(self, args):
"""
:param args: operator parameters, including the service parameters of the operator and the default parameters obs_input_path (input OBS path) and obs_output_path (output OBS path).
"""
pass

def __call__(self, input: pd.DataFrame):
"""
:param input: input parameter. In mode 2, input is an empty DataFrame.
:return: No return value.
"""
pass

Preconfigured Dependencies of the Operator Base Image

The following table lists the dependencies preconfigured in the base image of the operator.