Data Preparation Functions

Function

During the development of large models, data quality and processing efficiency directly impact model performance. However, developers often face challenges such as difficult data acquisition, inconsistent data quality, and low processing efficiency. These challenges not only increase the cost of model training but also limit the generalization capability of models. How to efficiently prepare high-quality training data has become an urgent issue. ModelArts data preparation provides one-stop, full-process data processing and management services. With built-in industry-level data processing operators and automated pipelines, it systematically handles data acquisition, processing, and publishing. This helps you efficiently convert massive amounts of raw, multimodal data into highly available and pure training datasets, improving data quality and processing efficiency, significantly reducing model training costs, and enhancing model generalization capabilities.

Data Preparation Process

ModelArts provides end-to-end data development features. You can use Data Connection, Manual Calibration, and Smart Refining to develop model datasets. Smart Refining covers the entire data processing process, including data processing and synthesis, helping developers quickly generate datasets required for model development.

Figure 1 shows the overall data preparation process.

Figure 1 Data preparation process

• Data Connection: Data acquisition is the first step of data engineering. Data from different sources and in different formats can be imported to the platform. An original dataset can be generated. With this feature, you can easily import a large amount of data to the platform to prepare for subsequent smart refining and model development. For details, see Data Connection.
• Manual Calibration: You can manually calibrate datasets on the visualized labeling page, generate standard datasets in one-click mode, and synchronize the datasets to My Data for tasks such as smart refining. For details, see Manual Calibration.

• Smart Refining: provides one-stop operations such as data processing and data synthesis to ensure that the original data meets various service requirements and model training standards and to obtain a dataset that meets model development requirements. For details, see Smart Refining.

Data Asset Management

The data asset management module provides a one-stop multimodal data management center for developers. It breaks down data silos and implements full-link closed-loop management from data ingestion, version control, quality preview, to final calling. ModelArts manages multimodal data, including text, image, audio, and video data. It categorizes data into preset and user-defined assets based on the source, supporting both general capability development and specialized, domain-specific customizations. For details, see Data Asset Management.

Data Types Supported by the ModelArts Platform

ModelArts offers top data processing for text, images, audio, and videos. You can create custom datasets and work with popular formats like Alpaca and ShareGPT, making it easy to handle various data types.

The platform's smart refining and management capabilities provide you with comprehensive datasets for developing models.

Table 1 lists the data types supported by the platform. For details about the data format requirements of each type, see Dataset Format Requirements.