Operator Base Classes and Differences

Operator Base Classes

Inheritance is a core concept in object-oriented programming: a subclass automatically acquires all attributes and methods of its parent class, while also being able to override parent class methods to implement custom logic. By inheriting an SDK base class, custom operators only need to implement the process method to complete the development of data processing logic.

OP (Parent base class, which cannot be used directly)
├── Mapper         — Row-level transformation/labeling operator
├── BatchMapper    — Batch transformation/labeling operator
├── Filter         — Row-level filtering operator
├── BatchFilter    — Batch filtering operator
└── Generator      — One-to-many operator (one sample generates multiple samples)

OP Parent Base Class Parameters

OP is the top-level base class for all operators and cannot be inherited directly. It defines the common attributes and methods of operators. Key parameter descriptions are shown in the following table.

Selection Suggestions

Decision process for choosing a base class:
  I want to transform/label data → Mapper (row-level) or BatchMapper (batch)
  I want to filter data → Filter (row-level) or BatchFilter (batch)
  I want to generate multiple items from one → Generator
Row-level vs. batch:
  No model inference needed → Row-level (Mapper/Filter), simpler logic
  Model inference needed → Batch (BatchMapper/BatchFilter), higher throughput

Dataset Types and Fields

The SDK has built-in dataset entity classes used to parse and construct sample data within operators.

Each dataset entity class provides three static methods: from_dict(sample): Converts a dictionary into an object. to_dict(obj): Converts an object into a dictionary. check(sample): Validates the sample format.

FAQs

• Can multiple classes be defined in process.py?

  Yes. However, only the three class names PreProcess, Process, and PostProcess will be recognized and executed. You can define helper classes and functions for these three classes to call.

• Can PreProcess and PostProcess inherit different base classes?

  Yes. The three processing stages can inherit different operator base classes; for example, PreProcess inherits Filter (filtering) and Process inherits Mapper (labeling).

• What is the difference between the return values of Mapper and BatchMapper's process methods?

  Mapper's process returns dict[str, Any], representing key-value pairs of newly added columns for a single sample; BatchMapper's process returns pa.Array, representing the results of an entire column of batch data.

• When does __return_extend_columns_type__() need to be implemented?

  It only needs to be implemented when the operator type is Mapper or BatchMapper to declare the types of newly added columns. Filter, BatchFilter, and Generator do not need to implement this method.

• How do I declare return types for multiple columns?

  @staticmethod 
  def __return_extend_columns_type__() -> daft.schema.Schema: 
      return daft.schema.Schema.from_pydict({ 
          'score': daft.DataType.int64(), 
          'label': daft.DataType.string(),
          'confidence': daft.DataType.float64(),
      })

• How does Generator's return value differ from other operators?

  Generator's process returns list[dict[str, Any]], where each element is a complete new sample (containing all columns), whereas Mapper only needs to return the newly added columns. Generator is suitable for scenarios where one sample generates multiple samples.