Cette page n'est pas encore disponible dans votre langue. Nous nous efforçons d'ajouter d'autres langues. Nous vous remercions de votre compréhension.

On this page
Help Center/ ModelArts/ Troubleshooting/ Training Jobs/ Memory Limit Issues/ Insufficient Container Space for Copying Data

Insufficient Container Space for Copying Data

Updated on 2024-06-11 GMT+08:00

Symptom

When a ModelArts training job is running, the following error is reported in the log. As a result, data cannot be copied to the container.

OSError:[Errno 28] No space left on device

Possible Causes

The container space is insufficient for downloading data.

Solution

  1. Check whether data is downloaded to the /cache directory. Each GPU node has a /cache directory with 4 TB of storage.
  2. Check whether GPU resources are used. If CPU resources are used, /cache and the code directory share 10 GB of memory. As a result, the memory is insufficient. In this case, use GPU resources instead.
  3. Add the following environment variable to the code:
    import os
    os.system('export TMPDIR=/cache')
Feedback

Feedback

Feedback

0/500

Selected Content

Submit selected content with the feedback