Sample Dataset Storage Path

When the sample dataset is stored, two formats are available: compressed and decompressed files. The data of both formats is the same.

• Compressed file: After downloading the compressed file, upload it to your OBS bucket. You need to decompress the file before using it. However, it is convenient to download the file.
• Decompressed file: You can directly copy the decompressed file to your OBS bucket. That is, you can copy an OBS bucket to another OBS bucket. The prerequisite is that your OBS bucket and the OBS bucket of the sample dataset belong to the same region.

Method 1: Download and Upload Files

There is no specific restriction on the region. You can select any region to download the dataset. To improve operation efficiency, you are advised to download the compressed dataset. However, the download and upload speeds depend on your local network conditions.

Figure 2 Operations of method 1

1. Select the storage path of the target sample dataset. You can select a dataset in an OBS bucket in any region. You are advised to download the compressed dataset file. Click the link to download the sample dataset to the local PC.

   For example, if you click the link for downloading the Yunbao-Data-Custom.zip dataset for Yunbao Detection in the CN North-Beijing1 region, the Yunbao-Data-Custom.zip file is downloaded to the local PC.

2. Decompress the obtained file and upload all folders of the dataset to the OBS directory.
   1. First, create an OBS bucket and a folder for storing the sample dataset.

      For example, create an OBS bucket named test-modelarts and a folder named dataset-yunbao.

   2. Decompress the Yunbao-Data-Custom.zip file to the Yunbao-Data-Custom directory on the local PC.
   3. Upload all files in the Yunbao-Data-Custom directory to the test-modelarts/dataset-yunbao directory on OBS. For details about how to upload files, see Uploading a File.

Method 2: Use the MoXing API to Copy a Dataset from the Public Bucket to Your OBS Bucket

The sample dataset must be in the same region as your OBS bucket and you are familiar with notebook operations and ModelArts MoXing. You can copy the sample dataset from the public bucket to your OBS bucket.

You are advised to obtain the OBS path (in OBS format) of the desired decompressed dataset listed in Table 1, create a notebook instance in ModelArts, and copy the dataset to your OBS bucket.

1. Access the ModelArts management console, create a notebook instance, and create a file on the Jupyter page.
2. Click the new file to access the development environment.
3. Check whether the public bucket where the sample dataset resides is accessible.

   For example, obtain the sample dataset of Yunbao Detection in the CN North-Beijing1 region from Table 1. The OBS path is obs://modelarts-cnnorth1-market-dataset/dataset-market/Yunbao-Data-Custom/unarchiver. Run the following command to check whether the public bucket is accessible:

   import moxing as mox
   mox.file.exists('obs://modelarts-cnnorth1-market-dataset/dataset-market/Yunbao-Data-Custom/unarchiver')

   If True is returned, the OBS bucket is normal.

4. Check whether your OBS bucket can be accessed.

   For example, create an OBS bucket named test-modelarts and a folder named dataset-yunbao. Run the following command to check whether your bucket is accessible:

   import moxing as mox
   mox.file.exists('obs://test-modelarts/dataset-yunbao')

   If True is returned, the OBS bucket is normal.

5. Check whether you have the write permission on the OBS bucket.

   For example, the path of the target OBS bucket is obs://test-modelarts/dataset-yunbao. Run the following command to check the permission:

   import moxing as mox
   mox.file.write('obs://test-modelarts/dataset-yunbao/obs_file.txt', 'Hello, OBS Bucket!')
   mox.file.remove('obs://test-modelarts/dataset-yunbao/obs_file.txt', recursive=False)

6. Run the following command to copy the sample dataset from the public bucket to your OBS bucket:

   import moxing as mox
   mox.file.copy_parallel('obs://modelarts-cnnorth1-market-dataset/dataset-market/Yunbao-Data-Custom/unarchiver', 'obs://test-modelarts/dataset-yunbao
   ')
   print ('Copy procedure is completed')

   When Copy procedure is completed and the execution time are returned, the dataset is copied. Information similar to the following is displayed:

   Copy procedure is completed
   CPU times: user 117 ms, sys: 92.3 ms, total: 209 ms
   Wall time: 58.3 s

Method 3: Use the obsutill Tool of OBS to Copy Files

The sample dataset must be in the same region as your OBS bucket. You can use the obsutil tool provided by OBS to copy the sample dataset. You are advised to obtain the OBS path (in OBS format) of the decompressed dataset file in Table 1 and copy the file to your OBS bucket by running the object copy command in Copying an Object.

For details about how to use obsutil, see obsutil in the Object Storage Service Tools Guide.