Help Center/ ModelArts/ Getting Started/ Getting Started with Data Connection: One-Click Data Connection
Updated on 2026-11-10 GMT+08:00
View PDF
Share

Getting Started with Data Connection: One-Click Data Connection

Scenarios

In data processing and model training scenarios, efficiently and accurately importing various datasets into ModelArts is essential to support subsequent smart refining and model training. Traditional data import methods, however, are often limited—supporting only basic task naming and offering minimal data format conversion capabilities. These constraints lead to difficulties in data import and slow processing efficiency, making the need for a more flexible and efficient data import solution increasingly urgent. ModelArts addresses these challenges with an enhanced data import feature that supports multiple common data types. You can now customize task names and descriptions during task creation. The platform also supports data format conversion and direct dataset publishing, significantly improving both the flexibility and efficiency of data processing. This enhancement meets the diverse needs of users during the data preparation phase.

Prerequisites

Constraints

  • The size of a data file or compressed package imported through a data connection cannot exceed 20 GB, and the total file size cannot exceed 20 GB.
  • When importing data from OBS, you can only specify a folder as the OBS path, not a file.

Billing

You are billed based on the used compute and storage resources. For details, see Data Management Billing Items.

Creating a Data Connection Task

  1. Log in to the ModelArts console.
  2. In the left navigation pane of the management console, choose Data Preparation > Data Connections.
  3. In the upper right corner, click Create Data Connection. On the displayed page, configure the parameters.
    Table 1 Parameters required for creating a data connection

    Parameter

    Description

    Example Value

    Basic Information

    Name

    Custom task name. The default value is data-connect-YYYYMMDDHHMMSS.

    Only letters, digits, hyphens (-), and underscores (_) are supported. Enter 1 to 128 characters.

    data-connect-20260425083507

    Description

    Description of a custom data connection. It can contain a maximum of 512 characters.

    -

    Data Import

    Modal

    Modality refers to the specific form or type in which data exists. Text, image, video, audio, and other types of data are supported. Select a modality, and the system shows its available data types.

    For details about the format requirements of each modality, see Dataset Format Requirements.

    Text

    Type

    Select the specific type of the dataset. The options vary depending on the modality.

    • If you select TEXT, you can choose from Base Text, Pre-trained Text, Text Generation SFT, and Reinforcement Learning.
      • Base Text: Centered around multi-format raw documents, supports document content extraction and parsing, and converts data into pre-training datasets or evaluation sets through platform data refinement.
      • Pre-trained Text: Centered around large-scale raw text corpora, suitable for direct use in model pre-training or refined into high-quality text generation SFT datasets.
      • Text Generation SFT: An annotated text dataset centered around Q&A pairs, containing instructions and standard responses. It adapts to SFT to meet training needs for dialogue generation, instruction alignment, and industry scenario-specific customization.
      • Reinforcement Learning: Tailored for reinforcement learning preference training and long CoT reasoning scenarios, adapting to GRPO and DAPO dedicated data formats for uploading and generating reinforcement learning training and evaluation datasets.
    • If you select IMAGE, you can choose Base Image.

      Base Image: Raw image datasets supporting various live-action, still-life, and scene images, suitable for visual pre-training and feature learning in multimodal models.

    • If you select VIDEO, you can choose Base Video.

      Base Video: Raw video datasets covering diverse real-world scenes, environments, and dynamic visual content, suitable for spatiotemporal visual pre-training and feature learning in multimodal models.

    • If you select AUDIO, you can choose Base Audio.

      Base Audio: Raw audio datasets recording human voices, ambient sounds, speech segments, etc., suitable for speech model pre-training and acoustic feature learning.

    • If you select OTHER, you can choose Customization.

      Customization: You can upload data in many non-standard formats, not limited to fixed data types. This flexibility suits different model training and evaluation needs.

    Base Text

    File Format

    Select a file format as required.

    -

    Source

    The supported sources may vary depending on the dataset type.

    Object Storage Service

    Storage Type

    If you select Object Storage Service or Local File as the source, you can select Object Storage Service – Bucket or Object Storage Service – Parallel File System.

    Object Storage Service – Bucket

    Storage Location
    • If you select Object Storage Service as the source, specify the OBS path where the data to be connected is stored for creating a dataset.
    • If you select Local File, specify the storage path on OBS where the local data will be uploaded. After the upload is complete, this path becomes the path of the dataset to be connected.

    Click to select an OBS storage address or manually enter an address. The storage address must start with obs:// or / and end with a slash (/). It cannot contain double slashes (//) except in the prefix. For example, obs://bucketname/path/ or /bucketname/path/.

    A maximum of 20 files can be uploaded. The size of a single file cannot exceed 1 GB. It is recommended that no more than 10 files be uploaded at a time.

    obs://bucketname/path/

    Data Source

    This parameter is only mandatory when Source is set to AI Datalake. Select the configured AI DataLake data source as needed. For details about how to create a data source, see Managing AI DataLake Data Sources.

    -

    Dataset

    This parameter is only mandatory when Source is set to AI Datalake. Click and select the target dataset.

    -

    Generate Dataset

    Dataset Name

    Name of a custom dataset. Only letters, digits, hyphens (-), and underscores (_) are supported. Enter 1 to 128 characters.

    dataset

    Dataset Property

    Click to configure dataset properties as required, such as the industry and language.

    -

    Description

    Description of a custom dataset. Only letters, digits, spaces, hyphens (-), underscores (_), commas (,), periods (.), brackets, colons (:), and commas are allowed. The value can contain up to 512 characters.

    -

    Dataset Status

    Only published datasets can be used by downstream tasks such as model training.

    • If you select Publish Dataset, the generated dataset is in the Online state on the Asset Management > Data > My Data page and can be directly used by downstream model training jobs.
    • If you do not select Publish Dataset, the generated dataset will be in the Offline state on the Asset Management > Data > My Data page and cannot be directly used by downstream model training jobs. You need to manually publish the dataset before using it.

    Select Publish Dataset.
  4. After setting the parameters, click Create Now in the lower right corner of the page. On the Data Connections page, you can view the task status of the dataset. If the status is Success, the data connection task is successful.
    Figure 1 Viewing the data connection task status

  5. After the dataset is generated, choose Asset Management > Data in the navigation pane and view the generated dataset in the My Data tab. For more information, see My Data.

Follow-Up Operations

After a data connection task is executed, it can be used for smart refining and model training.

FAQs

What should I do if creating a data connection fails with the task status "Failed" and the error "Internal service error. Please contact technical support. For more information, please check out Run Logs"? 
The asset manager service is abnormal. [{"error_code":"ModelArts.70512097","error_msg":"Sync dataset to data-management failed.","error_name":"Sync dataset to data-management failed.","error_solution":"Check if the data-management service is available or contact the data-management team. Sync dataset to data-management failed.","advice":"Check if the data-management service is available or contact the data-management team.","detail":"Sync dataset to data-management failed."}]
Figure 2 Running a data connection failed

Error code ModelArts.70512097 (failed to synchronize the dataset to the data management service) is usually caused by a duplicate generated dataset name. If the generated dataset has the same name as an existing dataset, data synchronization fails, triggering this error.

Solution: Re-create the data connection and change the generated dataset name to ensure that it does not duplicate the name of an existing dataset. If the error persists after you change the name, contact technical support.