Help Center/ ModelArts/ Troubleshooting/ Training Jobs/ In-Cloud Migration Adaptation Issues/ Error Message "retCode=0x91, [the model stream execute failed]" Displayed in MindSpore Logs
Updated on 2022-12-08 GMT+08:00

Error Message "retCode=0x91, [the model stream execute failed]" Displayed in MindSpore Logs

Symptom

When MindSpore is used for training, the following error message is displayed:
[ERROR] RUNTIME(3002)model execute error, retCode=0x91, [the model stream execute failed]

Possible Causes

The speed of reading data cannot keep up with the model iteration speed.

Solution

  1. Reduce shuffle operations during preprocessing.
    dataset = dataset.shuffle(buffer_size=x)
  2. Disable data preprocessing, which may affect system performance.
    NPURunConfig(enable_data_pre_proc=Flase)

Summary and Suggestions

Before creating a training job, use the ModelArts development environment to debug the training code to maximally eliminate errors in code migration.