Creating a Smart Refining Task

Scenarios

Smart Refining is an integrated data preparation solution designed to streamline both cleaning and synthesis for large model training. Whether you are processing raw pre-training corpora by removing HTML tags and garbled characters, enhancing sparse seed data for SFT instruction fine-tuning, or ensuring security compliance through privacy anonymization, Smart Refining simplifies the workflow. It orchestrates a wide range of data processing operators into a cohesive pipeline, transforming messy or incomplete inputs into high-quality, diverse, and secure training datasets ready for model development.

Prerequisites

• You have registered a Huawei ID and enabled Huawei Cloud services, performed real-name authentication, and ensure your account is not frozen or in arrears before using ModelArts. For details, see Signing Up for a HUAWEI ID and Enabling Huawei Cloud Services and Real-Name Authentication Introduction.
• You have configured an agency.

  Certain ModelArts functions require access to services like OBS. Before using ModelArts, ensure your account has been authorized to access these services.

• You have applied for the compute resources required for smart refining.
• If you need to use a custom dataset, import the dataset to ModelArts by referring to Creating a Data Connection.

Billing

Billing is based on the actual execution duration or usage of CPU operators.

Constraints

• Synthesis operator: The synthesis operator must be placed at the last node of the workflow. It cannot be inserted between processing operators or followed by other filtering operators.
• Modality:
  • Only intra-modal synthesis (e.g., text input to text output) is supported.
  • Cross-modal generation (such as generating Q&A pairs from input text or generating images from output text) is not yet supported.
• Functions:
  • Custom synthesis instructions (prompts) and the template functions are not supported.
  • Online debugging of synthesis tasks is not supported.
  • The output fields of synthesis operators are fixed, but the original fields in the input dataset are automatically retained.
• Data volume and quality inspection:
  • You cannot customize the number of output records of synthesized data. (Synthesized data is automatically generated based on the input.)
  • You cannot perform automatic quality inspection or filtering on the results.
  • The output is saved to a new dataset and is not automatically merged with the original dataset.

Creating a Smart Refining Task

1. Log in to the ModelArts console. In the navigation pane on the left, choose Data Preparation > Data Refining.
2. In the upper right corner, click Create Smart Refining Task, configure information, and click Next Step.

   Table 1 Parameters for creating a smart refining task

   Parameter		Description	Example Value
   Basic Information	Name	Custom task name. The default value is data-refine-YYYYMMDDHHMMSS. The name must start with a letter and end with a letter or digit. It can contain 2 to 64 characters. Only letters, digits, hyphens (-), and underscores (_) are allowed.	data-refine-20260423102952
   	Description	Description of a custom smart refining task. Only letters, digits, spaces, hyphens (-), underscores (_), commas (,), and periods (.) are allowed. It can contain a maximum of 200 characters.	-
   Selecting Dataset		Choose either ModelArts Preset Data or My Data. The supported dataset types are text, images, and videos. If you select My Data, upload a dataset. For details, see Creating a Data Connection.	Preset data code-alpaca
   Refinement Template		ModelArts offers ready-to-use smart refinement templates. These templates include preset service processing tools and parameters for specific applications. You can use these templates directly. If your application does not need the services in the refinement template, skip this step and click Next Step.	Single-turn Q&A process

3. Select and orchestrate data operators based on the specific scenario. After you select operators, the operator orchestration area is displayed on the right. Configure the operator sequence and parameters, and click Save and Next in the lower right corner.

   For details about the applications, see Application Scenarios. For details about the operators, see Preset Smart Refining Operators.

   Note: Data orchestration is the key and most complex part of smart refining. Many scenarios and constraints must be considered. For text data like single-turn or multi-turn dialogues (with or without persona settings), the start and end nodes (Start Node and End Node) handle both data input/output and format conversion. The following describes scenarios involving data format conversion:

   • When an arbitrary-format dataset enters the start node, it must be converted into the platform-compatible dataset format to enable processing by subsequent operators. Once all processing operators have completed their tasks, the end node will, by default, convert the output dataset back into the same format as the original input dataset.
   • The end node can be configured to output the dataset in any format, allowing you to choose the target format.
   • If no other operators are added between the start node and the end node, the system handles the task based on the following two conditions:
     • If the end node is set to the same format as the start node's input, no operations are performed on the dataset. In this case, the Save and Next button will be grayed out, preventing further configuration.
     • If the end node is set to a format different from the input, the task is treated as a pure format conversion. You can go to the next step to complete the subsequent configuration of the smart refining task.
   • After you click Save and Next, the current smart refining orchestration task, including the orchestration steps and previous configurations, will be saved. If the task is not complete, you can continue the subsequent configuration after the smart refining task is started next time.
4. Configure the generated dataset and resource information, and click Run.

   Table 2 Parameter description

   Parameter		Description	Example Value
   Generate Dataset	Dataset Name	Name of a custom dataset. The name must start with a letter and end with a letter or digit. It can contain 2 to 63 characters. Only letters, digits, hyphens (-), and underscores (_) are allowed.	dataset
   	Storage Location	Choose Object Storage Service – Bucket or Object Storage Service – Parallel File System as the storage type. Click to select an OBS storage address or manually enter an OBS storage address. The storage address must start with obs:// or / and end with a slash (/). It cannot contain double slashes (//) except in the prefix. For example, obs://bucketname/path/ or /bucketname/path/.	obs://bucketname/path/
   	Dataset Property	Configure dataset properties as required. You can configure tags by industry or language, or customize labels.	-
   	Description	Only letters, digits, spaces, hyphens (-), underscores (_), commas (,), and periods (.) are allowed. It can contain a maximum of 200 characters.	-
   	Dataset Status	Only published datasets can be used by downstream tasks such as model development and training. If you select Publish Dataset, the generated dataset is in the Online state on the Asset Management > Data > My Data page and can be directly used by downstream model training jobs. If you do not select Publish Dataset, the generated dataset will be in the Offline state on the Asset Management > Data > My Data page and cannot be directly used by downstream model training jobs. You need to manually publish the dataset before using it.	Select Publish Dataset.
   Resource Configuration	Resource Pool Type	Select a resource pool type as required.	Public resource pool
   	Reference Specifications for Execution CPU Operator Instance	Data operators need compute resources for processing. Choose CPU or NPU resources based on the task and operator type.	NPU (1 card) | (24 vCPUs) | MEMORY (192 GB)

   The smart refining task is complete when its Latest Status changes to Dataset Generation Success. The new data will be in Asset Management > Data > My Data.

Best Practice: Operator Orchestration Design Principles

Principle 1: Cleanse data before processing it.

Recommended sequence: Deduplicate → Format → Filter → Augment→ Synthesize

Principle 2: Reduce data before expanding it. Use the filter operator to reduce the data volume and then use the synthesis operator to expand it, improving the overall processing efficiency.

Principle 3: Place the synthesis operator at the end. The synthesis operator can only be used as the last processing step.

Principle 4: Maintain modal consistency. The entire workflow processes the same type of data and does not cross modalities.

Recommended Operator Orchestration Templates

Template 1: Basic data cleaning

Input → Format verification → Deduplication → Length filtering → Output

Template 2: Data cleaning + quality improvement

Input → Format verification → Deduplication → Sensitive word filtering → Quality score-based filtering → Output

Template 3: Data cleaning + synthesis and expansion

Input → Deduplication → Sensitive word filtering → Quality filtering → Q&A rewriting and synthesis → Output

Template 4: Full-process refinement

Input → Format conversion → Deduplication → Sensitive word filtering → Quality scoring → Length filtering → Rewriting and synthesis → Output

Follow-Up Operations

After a dataset is published, it can be directly used for model development, such as model training.