Creating a Dataset (Old Version)

To manage data using ModelArts, you need to create a dataset first. Then you can perform operations on the dataset, such as labeling data, importing data, and publishing the dataset.

ModelArts provides both new and old versions of datasets.

The new version decouples dataset creation and labeling task creation. Datasets and labeling tasks are separately created.

For the old version, you need to create a labeling task when creating a dataset.

This section describes how to create a dataset of the old version.

Prerequisites

Before using the data management function, you need permissions to access OBS. This function cannot be used if you are not authorized to access OBS. Before using the data management function, go to the Settings page and complete access authorization using an agency.
You have created OBS buckets and folders for storing data. In addition, the OBS buckets and ModelArts are in the same region.
You have uploaded data to be used to OBS.

Procedure

Log in to the ModelArts management console. In the left navigation pane, choose Data Management > Datasets. The Datasets page is displayed.
Click Create Dataset. On the Create Dataset page, create datasets of different types based on the data type and data labeling requirements.
1. Set the basic information, the name and description of the dataset.
  Figure 1 Basic information about a dataset
2. Select a labeling scene and type as required. For details about the types supported by ModelArts, see Dataset Types.
  Figure 2 Selecting a labeling scene and type
3. Set the parameters based on the dataset type. For details, see the parameters of the following dataset types:
4. Click Create in the lower right corner of the page.
  After the dataset is created, the dataset management page is displayed. You can perform the following operations on the dataset: label data, publish dataset versions, manage dataset versions, modify the dataset, import data, and delete the dataset. For details about the operations supported by different types of datasets, see Functions Supported by Different Types of Datasets.

Images (Image Classification, Object Detection, and Image Segmentation)

Figure 3 Parameters of datasets for image classification and object detection

**Table 1** Dataset parameters
Parameter	Description
Input Dataset Path	Select the OBS path to the input dataset. NOTE: When you create a dataset, data in the OBS path will be imported to the dataset. If you modify data in OBS, the data in the dataset will be inconsistent with that on OBS. As a result, some data may be unavailable. If you need to modify data in a dataset, use the Synchronizing Data Sources or Import Operation function.
Output Dataset Path	Select the OBS path to the output dataset. NOTE: The output dataset path cannot be the same as the input dataset path or cannot be the subdirectory of the input dataset path. Select an empty directory as the Output Dataset Path.
Label Set	Label name: Enter a label name. The label name can contain only letters, digits, underscores (_), and hyphens (-). The name contains 1 to 32 characters. Add Label: Click Add Label to add more labels. Setting a label color: This function is available only for datasets of the object detection type. Select a color from the color palette on the right of a label, or enter the hexadecimal color code to set the color. Setting label attributes: For an object detection dataset, you can click the plus sign (+) on the right to add label attributes after setting a label color. Label attributes are used to distinguish different attributes of the objects with the same label. For example, yellow kittens and black kittens have the same label cat and their label attribute is color.
Team Labeling	Enable or disable team labeling. Image segmentation does not support team labeling. Therefore, this parameter is unavailable when you use image segmentation. To enable the team labeling function, you need to enter the name and type of the team labeling task, and select the labeling team and team members. For details about the parameter settings, see Creating Team Labeling Tasks. Before enabling the team labeling function, ensure that you have added a team and members on the Labeling Teams page. If no labeling team is available, click the link on the page to go to the Labeling Teams page, and add your team and members. For details, see Introduction to Team Labeling. After a dataset is created with team labeling enabled, you can view the Team Labeling mark in Labeling Type.

Audio (Sound Classification, Speech Labeling, and Speech Paragraph Labeling)

Figure 4 Parameters of datasets for sound classification, speech labeling, and speech paragraph labeling

Parameter	Description
Input Dataset Path	Select the OBS path to the input dataset.
Output Dataset Path	Select the OBS path to the output dataset. NOTE: The output dataset path cannot be the same as the input dataset path or cannot be the subdirectory of the input dataset path. Select an empty directory as the Output Dataset Path.
Label Set (Sound Classification)	You need to set labels only for datasets of the sound classification type. Label name: Enter a label name. The label name can contain only letters, digits, underscores (_), and hyphens (-). The name contains 1 to 32 characters. Add Label: Click Add Label to add more labels.
Label Management (Speech Paragraph Labeling)	Only datasets for speech paragraph labeling support multiple labels. Single Label A single label is used to label a piece of audio that has only one class. Label Name: Enter a label name. The label name can contain contains 1 to 32 characters. Only letters, digits, underscores (_), and hyphens (-) are allowed. Label Color: Set the label color in the Label Color column. You can select a color from the color palette or enter a hexadecimal color code to set the color. Multiple Labels Multiple labels are suitable for multi-dimensional labeling. For example, you can label a piece of audio as both noise and speech. For speech, you can label the audio with different speakers. You can click Add Label Class to add multiple label classes. A label class can contain multiple labels. The label class and name can contain contains 1 to 32 characters. Only letters, digits, underscores (_), and hyphens (-) are allowed. Label Class: Set a label class. Label Name: Enter a label name. Add Label: Click Add Label to add more labels.
Speech Labeling (Speech Paragraph Labeling)	Only datasets for speech paragraph labeling support speech labeling. By default, speech labeling is disabled. If this function is enabled, you can label speech content.
Team Labeling	Only datasets of speech paragraph labeling support team labeling. After enabling team labeling, you need to set the name and type of the team labeling task, and select the team and team members. For details about the parameter settings, see Creating Team Labeling Tasks. Before enabling the team labeling function, ensure that you have added a team and members on the Labeling Teams page. If no labeling team is available, click the link on the page to go to the Labeling Teams page, and add your team and members. For details, see Introduction to Team Labeling. After a dataset is created with team labeling enabled, you can view the Team Labeling mark in Labeling Type.

Text (Text Classification, Named Entity Recognition, and Text Triplet)

Figure 5 Parameters of datasets for text classification, named entity recognition, and text triplet

**Table 2** Dataset parameters
Parameter	Description
Input Dataset Path	Select the OBS path to the input dataset. NOTE: Labeled text classification data can be identified only when you import data. When creating a dataset, set an empty OBS directory. After the dataset is created, import the labeled data into it. For details about the format of the data to be imported, see Specifications for Importing Data from an OBS Directory.
Output Dataset Path	Select the OBS path to the output dataset. NOTE: The output dataset path cannot be the same as the input dataset path or cannot be the subdirectory of the input dataset path. Select an empty directory as the Output Dataset Path.
Label Set (for text classification and named entity recognition)	Label name: Enter a label name. The label name can contain only letters, digits, underscores (_), and hyphens (-). The name contains 1 to 32 characters. Add Label: Click Add Label to add more labels. Setting a label color: Select a color from the color palette or enter the hexadecimal color code to set the color.
Label Set (for text triplet)	For datasets of the text triplet type, you need to set entity labels and relationship labels. Entity Label: You need to set the label name and label color. You can click the plus sign (+) on the right of the color area to add multiple labels. Relationship Label: A relationship label is a relationship between two entities. You need to set the source entity and target entity. You need to add at least two entity labels before adding a relationship label.
Team Labeling	Enable or disable team labeling. To enable the team labeling function, you need to enter the name and type of the team labeling task, and select the labeling team and team members. For details about the parameter settings, see Creating Team Labeling Tasks. Before enabling the team labeling function, ensure that you have added a team and members on the Labeling Teams page. If no labeling team is available, click the link on the page to go to the Labeling Teams page, and add your team and members. For details, see Introduction to Team Labeling. After a dataset is created with team labeling enabled, you can view the Team Labeling mark in Labeling Type.

Table

Figure 6 Parameters of datasets of the table type

When using a CSV file, pay attention to the following:

When the data type is set to String, the data in the double quotation marks is regarded as one record by default. Ensure that the double quotation marks in the same row are closed. Otherwise, the data will be too large to display.
If the number of columns in a row of the CSV file is different from that defined in the schema, the row will be ignored.

**Table 3** Dataset parameters
Parameter	Description
Storage Path	Select the OBS path for storing table data. The data imported from the data source is stored in this path. The path cannot be the same as or a subdirectory of the file path in the OBS data source. After a table dataset is created, the following four directories are automatically generated in the storage path: annotation: version publishing directory. Each time a version is published, a subdirectory with the same name as the version is generated in this directory. data: data storage directory. Imported data is stored in this directory. logs: directory for storing logs temp: temporary working directory
Import	If you have stored table data on other cloud services, you can enable this function to import data stored on OBS, DLI, or MRS.
Data Source (OBS)	File Path: Browse all OBS buckets of the account and select the directory where the data file to be imported is located. Contain Table Header: If this parameter is enabled, the imported file contains table headers. In this case, the first row of the imported file is used as the column name. Otherwise, the default column name is added and automatically filled in the schema information. For details about OBS functions, see Object Storage Service Console Operation Guide.
Data Source (DWS)	Cluster Name: The system automatically displays the DWS clusters of the account. You can select a DWS cluster from the drop-down list. Database Name: Enter the name of the database where the data is located based on the selected DWS cluster. Table Name: Enter the name of the table where the data is located based on the selected database. User Name: Enter the username of the DWS cluster administrator. Password: Enter the password of the DWS cluster administrator. For details about DWS functions, see Data Warehouse Service User Guide. NOTE: To import data from DWS, you need to use DLI functions. If you do not have the permission to access DLI, create a DLI agency as prompted.
Data Source (DLI)	Queue Name: The system automatically displays the DLI queues of the account. You can select a queue from the drop-down list. Database Name: All databases are displayed based on the selected queue. Select the required database from the drop-down list. Table Name: All tables in the selected database are displayed. Select the required table from the drop-down list. For details about DLI functions, see Data Lake Insight User Guide.
Data Source (MRS)	Cluster Name: The system automatically displays the MRS clusters of the account in the list. However, streaming clusters do not support data import. Select the required cluster from the drop-down list. File Path: Enter the file path based on the selected cluster. The file path is an HDFS path. Contain Table Header: If this parameter is enabled, the imported file contains table headers. For details about MRS functions, see MapReduce Service User Guide.
Schema	Names and types of table columns, which must be the same as those of the imported data. Set the column name based on the imported data and select the column type. For details about the supported types, see Table 4. Click Add Schema to add a new record. When creating a dataset, you must specify a schema. Once created, the schema cannot be modified. When data is imported from OBS, the schema of the CSV file in the file path is automatically obtained. If the schemas of multiple CSV files are inconsistent, an error is reported.

**Table 4** Migration data types
Type	Description	Storage Space	Range
String	String	-	-
Short	Signed integer	2 bytes	-32768-32767
Int	Signed integer	4 bytes	–2147483648 to 2147483647
Long	Signed integer	8 bytes	–9223372036854775808 to 9223372036854775807
Double	Double-precision floating point	8 bytes	-
Float	Single-precision floating point	4 bytes	-
Byte	Signed integer	1 byte	-128-127
Date	Date type in the format of yyyy-MM-dd, for example, 2014-05-29	-	-
Timestamp	Timestamp that represents date and time. Format: yyyy-MM-dd HH:mm:ss	-	-
Boolean	Boolean	1 byte	TRUE/FALSE

Video

Figure 7 Parameters of datasets of the video type

**Table 5** Dataset parameters
Parameter	Description
Input Dataset Path	Select the OBS path to the input dataset.
Output Dataset Path	Select the OBS path to the output dataset. NOTE: The output dataset path cannot be the same as the input dataset path or cannot be the subdirectory of the input dataset path. It is a good practice to select an empty directory for Output Dataset Path.
Label Set	Label name: Enter a label name. The label name can contain only letters, digits, underscores (_), and hyphens (-). The name contains 1 to 32 characters. Add Label: Click Add Label to add more labels. Setting a label color: Select a color from the color palette or enter the hexadecimal color code to set the color.

Other (Free Format)

Figure 8 Parameters of datasets of the free format type

**Table 6** Dataset parameters
Parameter	Description
Input Dataset Path	Select the OBS path to the input dataset.
Output Dataset Path	Select the OBS path to the output dataset. NOTE: The output dataset path cannot be the same as the input dataset path or cannot be the subdirectory of the input dataset path. It is a good practice to select an empty directory for Output Dataset Path.