Help Center/ ModelArts/ Troubleshooting/ Training Jobs/ In-Cloud Migration Adaptation Issues/ Error Message "retCode=0x91, [the model stream execute failed]" Displayed in MindSpore Logs
Updated on 2024-06-11 GMT+08:00
Error Message "retCode=0x91, [the model stream execute failed]" Displayed in MindSpore Logs
Symptom
When MindSpore is used for training, the following error message is displayed:
[ERROR] RUNTIME(3002)model execute error, retCode=0x91, [the model stream execute failed]
Possible Causes
The speed of reading data cannot keep up with the model iteration speed.
Solution
- Reduce shuffle operations during preprocessing.
dataset = dataset.shuffle(buffer_size=x)
- Disable data preprocessing, which may affect system performance.
NPURunConfig(enable_data_pre_proc=Flase)
Summary and Suggestions
Before creating a training job, use the ModelArts development environment to debug the training code to maximally eliminate errors in code migration.
Parent topic: In-Cloud Migration Adaptation Issues
What is your overall rating for this page?
0
1
2
3
4
5
6
7
8
9
10
Very dissatisfiedVery satisfied
Thank you very much for your feedback. We will continue working to improve the documentation.
The system is busy. Please try again later.
