Help Center/
ModelArts/
Troubleshooting/
Training Jobs/
In-Cloud Migration Adaptation Issues/
Error Message "retCode=0x91, [the model stream execute failed]" Displayed in MindSpore Logs
Updated on 2022-12-08 GMT+08:00
Error Message "retCode=0x91, [the model stream execute failed]" Displayed in MindSpore Logs
Symptom
When MindSpore is used for training, the following error message is displayed:
[ERROR] RUNTIME(3002)model execute error, retCode=0x91, [the model stream execute failed]
Possible Causes
The speed of reading data cannot keep up with the model iteration speed.
Solution
- Reduce shuffle operations during preprocessing.
dataset = dataset.shuffle(buffer_size=x)
- Disable data preprocessing, which may affect system performance.
NPURunConfig(enable_data_pre_proc=Flase)
Summary and Suggestions
Before creating a training job, use the ModelArts development environment to debug the training code to maximally eliminate errors in code migration.
- Use the online notebook environment for debugging. For details, see Using JupyterLab to Develop a Model.
- Use the local IDE (PyCharm or VS Code) to access the cloud environment for debugging. For details, see Using the Local IDE to Develop a Model.
Parent topic: In-Cloud Migration Adaptation Issues
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
The system is busy. Please try again later.
For any further questions, feel free to contact us through the chatbot.
Chatbot