Overview

Core Features

Smart refining is a key part of ModelArts data engineering, designed to address the dual challenges of data quality and quantity in foundation model training. It transcends the boundaries of traditional data processing by seamlessly integrating rule-based data processing (cleaning, filtering, deduplication, and more) with LLM-based data synthesis (rewriting, expansion, polishing, and more).

Through visualized operator orchestration, you can drag and drop multiple processing and synthesis operators to build an automated pipeline, much like assembling building blocks. The system follows your predefined logic to filter and optimize massive raw datasets layer by layer, ultimately outputting high-quality datasets that meet rigorous training requirements.

Functional Architecture

Smart refining takes text, image, and video datasets as input. It constructs smart refining tasks by orchestrating various data processing and synthesis operators to produce refined datasets. For details, see Figure 1.

Figure 1 Smart refining functional architecture

Benefits

• Unified workflow: Orchestrates data processing and synthesis in a single pipeline, eliminating the need to switch between modules and reducing intermediate data transfers.
• Enhanced data quality: Ensures high-quality input for the synthesis stage through rigorous, multi-level filtering using processing operators.
• Flexible orchestration: Supports the free combination of dozens of operators to satisfy diverse scenarios, from simple cleaning to complex data augmentation.
• Efficient scale expansion: Enables high-efficiency training data expansion by performing synthetic rewriting on top of cleaned, high-quality data.
• Streamlined operation: Offers a visualized, "what you see is what you get" orchestration experience, removing the need for manual scripting.
• Workflow reproducibility: Supports saving and reusing refining templates to ensure consistency across different data processing tasks.