Storage Types

EVS

It is suitable for data and algorithm exploration in the development environment and offers good performance.

Block storage SSDs feature better overall I/O performance than NFS. The storage capacity can be dynamically expanded to up to 4,096 GB.

As persistent storage, EVS disks are mounted to /home/ma-user/work. The data in this directory is retained after the instance is stopped. The storage capacity can be expanded online based on demand.

This type of storage can only be used in a single development environment.

Object Storage Service – Parallel File System

Parallel file systems mounted as persistent storage for AI development and exploration.

1. Dataset storage. Datasets stored in PFS buckets are directly mounted to notebook instances for browsing and data processing and can be directly used during training. Select PFS when creating a notebook instance.

   After the instance is running, the parallel file system that carries the datasets is dynamically mounted to the notebook instances. For details, see Dynamically Mounting an OBS Parallel File System.

2. Code storage. After debugging on a notebook instance, specify the OBS path as the code path for starting training, facilitating temporary modification.
3. Training observation. Mount storage to the training output path such as the path to training logs. In this way, view and check training on the notebook instance in real time. This is especially suitable for analyzing the output of jobs trained using Using TensorBoard Visualization Jobs in JupyterLab.

PFS is an optimized high-performance object storage file system with low storage costs and large throughput. It can quickly process high-performance computing (HPC) workloads. PFS mounting is recommended if OBS is used.

The performance of frequent read and write operations on small files is poor, which may cause notebook instance freezing. Exercise caution when using this storage type for heavy model training and large file decompression.

Object Storage Service – Bucket

When uploading or downloading a large amount of data in the development environment, you can use OBS buckets to transfer data.

Low storage cost and high throughput, but average performance in reading and writing small files. It is good practice to package or split the file by 128 MB or 64 MB. In this way, you can download the packages, decompress them, and use them locally.

The object storage semantics is different from the Posix semantics and needs to be further understood.

The performance of frequent read and write operations on small files is poor, which may cause notebook instance freezing. Exercise caution when using this storage type in scenarios such as model training and large file decompression.

Scalable File Service

Available only in dedicated resource pools. Use SFS storage in informal production scenarios such as exploration and experiments. Development and training environments can mount the same SFS storage simultaneously, eliminating the need to download data for every training job. Generally, this setup is not suitable for large-scale training exceeding 32 PUs or for models with heavy I/O read/write requirements.

SFS uses NFS and can be shared among multiple development and training environments. It is ideal for light-duty distributed training jobs that do not need extra data downloads at the start.

The performance is lower than that of EVS block storage.

Local storage

First choice for heavy-duty training jobs.

High-performance SSDs for the used VM or BMS, featuring high file I/O throughput. For heavy-duty training jobs, store data in the target directory and then start training.

By default, the storage is mounted to the /cache directory. For details about the available space of the /cache directory, see What Are Sizes of the /cache Directories for Different Notebook Specifications in DevEnviron?

The storage lifecycle is associated with the container lifecycle. Data needs to be downloaded each time the training job starts.