Creating a Data Connection

Scenarios

In data processing and model training scenarios, efficiently and accurately importing various datasets into ModelArts is essential to support subsequent smart refining and model training. Traditional data import methods, however, are often limited—supporting only basic task naming and offering minimal data format conversion capabilities. These constraints lead to difficulties in data import and slow processing efficiency, making the need for a more flexible and efficient data import solution increasingly urgent. ModelArts addresses these challenges with an enhanced data import feature that supports multiple common data types. You can now customize task names and descriptions during task creation. The platform also supports data format conversion and direct dataset publishing, significantly improving both the flexibility and efficiency of data processing. This enhancement meets the diverse needs of users during the data preparation phase.

Prerequisites

• You have registered a Huawei ID and enabled Huawei Cloud services, performed real-name authentication, and ensure your account is not frozen or in arrears before using ModelArts. For details, see Signing Up for a HUAWEI ID and Enabling Huawei Cloud Services and Real-Name Authentication Introduction.
• Configure an agency.

  Certain ModelArts functions require access to services like OBS. Before using ModelArts, ensure your account has been authorized to access these services.

• Before creating an import job, prepare data based on Dataset Format Requirements.

Constraints

• The size of a data file or compressed package imported through a data connection cannot exceed 20 GB, and the total file size cannot exceed 20 GB.
• When importing data from OBS, you can only specify a folder as the OBS path, not a file.

Creating a Data Connection Task

1. Log in to the ModelArts console.
2. In the left navigation pane of the management console, choose Data Preparation > Data Connections.

3. In the upper right corner, click Create Data Connection. On the displayed page, configure the parameters.

   Table 1 Parameters required for creating a data connection

   Parameter		Description	Example Value
   Basic Information	Task Name	Custom task name. The default value is data-connect-YYYYMMDDHHMMSS. The name must start with a letter and end with a letter or digit. It can contain 2 to 64 characters. Only letters, digits, hyphens (-), and underscores (_) are allowed.	data-connect-20260425083507
   	Description	Description of a custom data connection. It can contain a maximum of 200 characters. Only letters, digits, spaces, hyphens (-), underscores (_), commas (,), periods (.), brackets, colons (:), and commas are allowed.	-
   Data Import	Dataset Type	Text, image, video, audio, and other types of datasets are supported. For details about the dataset format requirements, see Dataset Format Requirements.	Text Base text
   	File format	The file formats supported by different datasets vary. For details, see the console.	docx
   	Connection Method	The connection methods supported by different datasets vary. For details, see the console.	Object Storage Service
   	Storage Location	If Connection Method is set to Object Storage Service, you can select Object Storage Service – Bucket or Object Storage Service – Parallel File System. Click to select an OBS storage address or manually enter an address. The storage address must start with obs:// or / and end with a slash (/). It cannot contain double slashes (//) except in the prefix. For example, obs://bucketname/path/ or /bucketname/path/. Ensure individual files or compressed packages do not exceed 1 GB in size. For shared buckets from other users, you must enter the path. If Connection Method is set to Local upload, you can select Object Storage Service – Bucket or Object Storage Service – Parallel File System. Click to select an OBS storage address or manually enter an address. The storage address must start with obs:// or / and end with a slash (/). It cannot contain double slashes (//) except in the prefix. For example, obs://bucketname/path/ or /bucketname/path/. For shared buckets from other users, you must enter the path. A maximum of 20 files can be uploaded. The size of a single file cannot exceed 1 GB. It is recommended that no more than 10 files be uploaded at a time.	obs://bucketname/path/
   Generate Dataset	Dataset Name	Name of a custom dataset. The name must start with a letter and end with a letter or digit. It can contain 2 to 63 characters. Only letters, digits, hyphens (-), and underscores (_) are allowed.	dataset
   	Dataset Property	Click to configure dataset properties as required, such as the industry and language.	-
   	Description	Description of a custom dataset. Only letters, digits, spaces, hyphens (-), underscores (_), commas (,), periods (.), brackets, colons (:), and commas are allowed. The value can contain a maximum of 200 characters.	-
   	Dataset Status	Only published datasets can be used by downstream tasks such as model training. If you select Publish Dataset, the generated dataset is in the Online state on the Asset Management > Data > My Data page and can be directly used by downstream model training jobs. If you do not select Publish Dataset, the generated dataset will be in the Offline state on the Asset Management > Data > My Data page and cannot be directly used by downstream model training jobs. You need to manually publish the dataset before using it.	Select Publish Dataset.

4. After setting the parameters, click Create Now in the lower right corner of the page. On the Data Connections page, you can view the task status of the dataset. If the status is Success, the data connection task is successful.
   Figure 1 Viewing the data connection task status
   

5. After the dataset is generated, choose Asset Management > Data in the navigation pane and view the generated dataset in the My Data tab. For more information, see My Data.

Follow-Up Operations

After a data connection task is executed, it can be used for Smart Refining and model training.