Updated on 2025-08-22 GMT+08:00

System Container Exits Unexpectedly

Symptom

After a training job is created, the system container exits unexpectedly.

[ModelArts Service Log]2022-10-11 19:18:23,267 - file_io.py[1ine:748] - ERROR:
	stat:404
	errorCode:NoSuchKey
	errorMessage:The specifiedkey does not exist.
	reason:Not Found
	request-id:00000183C6C4010C66D399E000COE3xx
	retry:0
[ModelArts Service Log]2022-10-11 19:18:23,267 - modelarts-downloader.py[line:90] - ERROR: modelarts-downloader. py: Download directory failed: [Errno
{'status': 404, ......}] file or directoryor bucket not found.

Possible Causes

The possible causes are as follows:

  1. An error occurred in OBS.
    1. Unavailable file: The specified key does not exist.
    2. Insufficient OBS permissions
    3. OBS traffic limiting
    4. Others
  2. The disk space is insufficient.

Solution

  1. For an OBS error:
    1. Unavailable file: The specified key does not exist.

      For details, see Error Message "errorMessage:The specified key does not exist" Displayed in Logs.

    2. Insufficient OBS permissions

      For details, see Error "stat:403 reason:Forbidden" Is Displayed in Logs When a Training Job Accesses OBS.

    3. OBS traffic limiting

      For details, see Error Message "BrokenPipeError: Broken pipe" Is Displayed When OBS Data Is Copied.

    4. Others

      For details, see OBS Server-Side Error Codes. Alternatively, obtain the request ID and contact OBS customer service.

  2. For insufficient disk space:

    For details, see Insufficient Disk Space.