Getting Started with Data Refining: One-Click Data Refining
Scenario
To lower the usage threshold and enhance user experience, we provide one-click data refining cases on the ModelArts console. This guide helps you start using Huawei ModelArts' one-click data refining feature. It uses a single-turn Q&A template to clean, deduplicate, and standardize your data, creating a high-quality Q&A dataset ready for LLM fine-tuning and training.
For details, see Data Refining.
Prerequisites
- You have registered a Huawei Cloud account, completed real-name authentication, and been granted related permissions. For details, see 1. Prerequisites: Accounts and Permissions.
- You have created an OBS bucket and folder for storing the datasets generated by the refining task. For details, see Using OBS Console.
Cross-region OBS buckets are not supported. For sensitive data, encrypt before storing.
Constraints
Only the new console supports one-click data refining.
Billing
You are billed based on the compute and storage resources you use. For details about the billing, see Data Management Billing Items.
Procedure
- Log in to the ModelArts console.
- In the navigation pane, choose . In the Featured area, select Smart Refining for Text generation SFT Data and click Refine in the card.
- On the Create Refinement Task panel, configure related information.
Table 1 Parameters for creating a refining task Parameter
Description
Example Value
Task Name
Name of a refining task.
The system automatically generates a name. You can rename the task based on service requirements. The name can contain 1 to 128 characters. Only letters, digits, hyphens (-), and underscores (_) are allowed.
data-refine-20260706102429
Template Name
Refining template used for creating a refining task. Currently, only the text generation SFT process is supported. You can click the name of a template card to view its details.
For details about the data refining template, see Smart Refining Templates.
Text generation SFT processing workflow
Dataset
Raw dataset used for creating a one-click refining task. In this case, a preset dataset is used.
Preset dataset: Smol-contraints
Generated Dataset Name
Name of the generated dataset. The system automatically generates a name. You can rename the task based on service requirements. The name can contain 1 to 128 characters. Only letters, digits, hyphens (-), and underscores (_) are allowed.
pub_20260706_24307
Storage Location
OBS path for storing the dataset generated by the one-click refining task. You can directly enter an OBS path or click
to select a storage location. The storage address must start with obs:// or / and end with a slash (/). It cannot contain double slashes (//) except in the prefix. For example, obs://bucketname/path/ or /bucketname/path/.- Only the buckets in the current region can be selected. Encrypt sensitive data before saving it to your OBS bucket.
- Only folders within an OBS bucket can be selected.
obs://obstest0528/test/
Resource Allocation
In this case, a public resource pool is used. For details about resource pools, see What Are the Differences Between Public Resource Pools and dedicated resource pool in ModelArts?
Public resource pool
Reference Specifications for Execution CPU Operator Instance
CPU (4 vCPUs) | Memory (16 GB) or higher is recommended.
CPU (4 vCPUs) | Memory (16 GB)
- Confirm the configuration and price, and click OK. The Data Refining page is displayed.
It takes a period of time to create a data refining task. You can click the task name to view the basic information, running details, refining result, and operation records of the task. For more information, see Managing Smart Refining Tasks.
- The data refining task is complete when its Latest Status changes to Dataset Generation Success. The new data will be stored on the Assets > Data > My Data page.
- If the status of a data refining task changes to Failed or Abnormal, you can click the task name to go to the details page and view the running details, refining result, operation records, and resource usage to locate the fault.
- After the one-click data refining is successful, choose Assets > Data in the navigation pane on the left, click the My Data tab, and click Online in the Operation column of the dataset. In the dialog box that is displayed, click OK.
After a dataset is published, it can be directly used for model development, such as model training and model evaluation.
What is your overall rating for this page?
Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot