Help Center/ Scalable File Service/ Best Practices/ Using OBS+SFS Turbo for Storage Acceleration in AI Scenarios/ Implementation Procedure/ Training/ Uploading Data to OBS and Preloading the Data to SFS Turbo
Updated on 2025-07-25 GMT+08:00
View PDF
Share

Uploading Data to OBS and Preloading the Data to SFS Turbo

Uploading Data to OBS

An OBS bucket has been created by referring to Creating a Bucket.

obsutil has been installed by referring to Downloading and Installing obsutil.

  1. Visit the ImageNet official website at http://image-net.org/ and download the ImageNet-21K dataset.
  2. Convert the format and then download the following annotation files: ILSVRC2021winner21k_whole_map_train.txt and ILSVRC2021winner21k_whole_map_val.txt.
  3. Upload the preceding files to the imagenet21k_whole folder in the OBS bucket. For details, see obsutil Quick Start.

OBS provides multiple data migration solutions. You can select an appropriate solution based on your data volume, required time, and cost. For more information, see Migrating Local Data to OBS.

Preloading Data from OBS to SFS Turbo

After an OBS bucket is added as a storage backend of the SFS Turbo HPC file system, you can preload data from OBS to SFS Turbo to reduce the time required when training data is accessed for the first time.

Before starting training jobs, you can preload data from OBS to SFS Turbo by importing the metadata and data. For details about how to preload data, see Managing SFS Turbo+OBS Storage Interworking.

  1. You can call the data import or export query API to query the status of import tasks.
  2. If your dataset size is small or the dataset does not change frequently, you can use an external tool, instead of auto import and export, to migrate data from OBS to SFS Turbo. obsutil is recommended. For details, see How Can I Migrate Data Between SFS and OBS?
Parent topic: Training