Help Center/ ModelArts/ ModelArts User Guide (Lite Cluster)/ Configuring Lite Cluster Resources/ Configuring Lite Cluster Storage
Updated on 2024-11-11 GMT+08:00
View PDF
Share

Configuring Lite Cluster Storage

The available storage space is determined by dockerBaseSize when no external storage is mounted. However, the accessible storage space is limited. It is recommended that you mount external storage to overcome this limitation.

You can mount storage to a container in various methods. The recommended method depends on the scenario. For details, see Table 1. Storage Basics helps you understand this section. Data Disk Space Allocation helps you understand how to configure data disk size based on service needs.

Table 1 Different methods of mounting storage to a container

Method

Scenario

Description

Operation Reference

EmptyDir

Training cache

Kubernetes ephemeral volumes, which are created and deleted together with Pods following the Pod lifecycle.

Using a Temporary Path

HostPath

This method is suitable for:

  1. Containerized workload logs that need to be saved permanently
  2. Containerized workloads that need to access internal data structure of the Docker engine in the host

Node storage. Multiple containers may share the storage, causing write conflicts.

Deleting a Pod does not clear its storage.

hostPath

OBS

Training dataset storage

Object storage. The OBS SDKs are used to download sample data. Due to the large storage capacity being far from nodes, direct training speed is slow. To improve this, data is typically pulled to a local cache before training.

SFS Turbo

Massive amounts of small files
  • POSIX file system
  • Shared or interconnected VPC between the file system and resource pool
  • High costs

SFS

Persistent storage for frequent reads and writes

This method applies to cost-sensitive workloads which require large-capacity scalability, such as media processing, content management, big data analytics, and workload analysis.

SFS Capacity-Oriented file systems are not suitable for services with massive amounts of small files.

EVS

Data persistence for notebook-based development

Each volume can be mounted to only one node.

The storage size depends on the size of the EVS disk.