Preparing Data

Uploading Data to OBS

There are many restrictions on using the OBS Console, and the OBS client is used to upload data. For more information about how to create a bucket and upload files, see Creating a Bucket and Uploading an Object.

Perform the following operations to import data to the dataset for model training and building.

1. Log in to OBS Console and create a bucket in the same region as ModelArts. If an available bucket exists, ensure that the OBS bucket and ModelArts are in the same region.
2. Upload the local data to the OBS bucket. If you have a large amount of data, use OBS Browser+ to upload data or folders. The uploaded data must meet the dataset requirements of the ExeML project.
   

   Upload data from unencrypted buckets. Otherwise, training will fail because data cannot be decrypted.

Requirements on Datasets

• Files must be in TXT or CSV format, and cannot exceed 8 MB.
• Use line feed characters to separate rows in files, and each row of data represents a labeled object.
• Currently, text classification supports only Chinese.

Requirements for Files Uploaded to OBS

• If you do not need to upload training data in advance, create an empty folder to store files generated in the future.

• If you need to upload files to be labeled in advance, create an empty folder and save the files in the folder. An example of the file directory structure is /bucketName/data/text.csv.
• A label name can contain a maximum of 32 characters, including letters, digits, hyphens (-), and underscores (_).
• If you want to upload labeled text files to the OBS bucket, upload them according to the following specifications:
  • The objects and files to be labels must be in the same directory. The objects must be in one-to-one relationship with the files. For example, if the object file name is COMMENTS_114745.txt, the label file name must be COMMENTS _114745_result.txt.

    Example of data files:

    ├─<dataset-import-path>
          │      COMMENTS_114732.txt 
          │      COMMENTS _114732_result.txt 
          │      COMMENTS _114745.txt 
          │      COMMENTS _114745_result.txt 
          │      COMMENTS _114945.txt 
          │      COMMENTS _114945_result.txt

  • The labeled objects and label files for text classification are text files, and correspond to each other based on rows. For example, the first row in a label file indicates the label of the first row in the labeled object.