Help Center/ ModelArts/ Data Preparation/ Data Refining/ Smart Refining/ Application Scenarios
Updated on 2026-07-03 GMT+08:00
View PDF
Share

Application Scenarios

Smart Refining is an integrated data preparation solution designed to streamline both cleaning and synthesis for large model training. Whether you are processing raw pre-training corpora by removing HTML tags and garbled characters, enhancing sparse seed data for SFT instruction fine-tuning, or ensuring security compliance through privacy anonymization, Smart Refining simplifies the workflow. It orchestrates a wide range of data processing operators into a cohesive pipeline, transforming messy or incomplete inputs into high-quality, diverse, and secure training datasets ready for model development.

Smart refining provides strong data processing and flexible operations, but getting started can be tough. This guide covers common smart refining scenarios to help you finish tasks easily and get high-quality data fast, speeding up model development.

Typical Scenarios

Smart refining is used in these typical scenarios, each with recommended operator combinations. Choose a scenario based on your needs.

Scenario 1: Converting the Dataset Format

Description

ModelArts supports multiple dataset formats. You need to convert data from one format to another without additional data processing.

Recommended operator orchestration sequence

Start node → end node

Expected results

The input data is converted into output data in different formats (Alpaca, ShareGPT, or platform-compatible).

Scenario 2: Cleaning and Improving Quality of Raw Corpus

Description

Raw data from the internet, internal systems, or third parties often has a lot of noise. So, it needs to be systematically cleaned for model training. For details about common corpus issues, see Table 1.

Table 1 Typical data issues

Data Issue

Form

Impact on the Model

Duplicate information

A large amount of identical or similar content exists in the data.

The trained model is overfitting.

Garbled data

Encoding errors and abnormal characters exist in the data.

The semantic understanding of the model is polluted.

Sensitive and non-compliant information

Political, pornographic, or violent content exists in the data.

The model output has compliance risks.

Poor data quality

Sentences are not coherent, the logic is disordered, and sentences are incomplete.

The model generation quality is reduced.

Invalid data length

The data is too short to be meaningful or too long to be redundant.

The training efficiency is low.

Mixed data, not classified

Data from various domains is mixed and not classified by domain.

Unclassified domain data affects training efficiency.

Recommended operator orchestration sequence

Raw corpus → [Symbol standardization] → [Deduplication operator] → [Sensitive word filtering] → [Text length filtering] → [Incomplete sentence removal at paragraph ends] → [Pornographic text detection] → [Political text detection] → [Insult text detection operator] → [Pre-trained text classification] → Cleaned data

Expected results

  • The data repetition rate is reduced by more than 90%.
  • Low-quality samples are effectively removed.
  • 100% of sensitive and non-compliant content is filtered out.
  • The output data can be directly used for training or proceed to the next synthesis step.
  • The output data can be classified by domain.

Scenario 3: Expanding and Augmenting Training Data

The cost of obtaining high-quality labeled data is high, and the existing data volume is insufficient to train a model with good performance.

Applicable scenarios

  • Data is scarce in vertical domains.
  • The labeling cost is too high.
  • The data scale needs to be quickly expanded.
  • Data diversity is insufficient.

Recommended operator orchestration sequence

Raw data → Data cleaning → Data generation → Expanded data
Table 2 Operators to use

Operator

Function

Configuration Suggestion

Data cleaning

Ensures seed data quality.

Ensure strict screening criteria.

Data generation

Generates diverse expressions.

Select an appropriate rewriting strategy to generate diverse data.

Expected results

  • The data scale is expanded by 3 to 10 times.
  • The semantic consistency is maintained.
  • The expression diversity is improved.

Notes

  • The synthesis operator must be placed at the end of the workflow.
  • Only data synthesis of the same modality is supported.

Scenario 4: Preparing Data for SFT

Prepare high-quality datasets for SFT of foundation models.

Applicable scenarios

  • Fine-tuning of general assistant models
  • Customization of industry-specific models
  • Dialogue capability optimization
  • Task-oriented model training

Recommended operator orchestration process

Raw instruction data → Data cleaning → Text generation (optional) → Dataset generation
Table 3 Data format processing

Input Format

Processing Method

Output Format

Unstructured text

Format conversion operator

Alpaca/ShareGPT

Existing Alpaca

Quality filtering + rewriting

Optimized Alpaca

Existing ShareGPT

Quality filtering + rewriting

Optimized ShareGPT

Key quality control points

  • Instruction clarity check
  • Answer accuracy verification
  • Format consistency assurance

Scenario 5: Processing Multimodal Data

Process datasets that contain multiple modalities, such as images and videos.

Applicable scenarios

  • Video understanding data sorting

Recommended operator orchestration process (using images as an example)

Image dataset → Image deduplication → Image extraction → Image metadata filtering → Image detection → Processed data
Table 4 Key points for processing each modality

Modality

Key Point

Notes

Image

Size, format, and quality

Unified resolution

Video

Frame rate, resolution, and segment

Unified video encoding

Important constraints

A data processing task needs to be created for each modality separately.

Scenario 6: Ensuring Data Compliance and Security

Ensure that the training data complies with regulatory requirements and enterprise security policies.

Applicable scenarios

  • Personal information protection (GDPR/Personal Information Protection Law)
  • Sensitive data filtering

Recommended operator orchestration process

Raw data → Sensitive word filtering → Compliance data

Operators to use

Operator

Function

Compliance Requirements

Sensitive word filtering

Filters sensitive content in personal information.

Personal privacy requirements must be met.
Parent Topic: Smart Refining