Training Job Failed with Error Code 139

The training job failed, and error code 139 is returned.

[Modelarts Service Log]Training end with reeturn code: 139
INFO:root:Using MoXing-v1.17.2-c806a92f
INFO;root:Using OBS-Python-SDK-3.1.2

The possible causes are as follows:

Certain pip packages in the pip source have been updated, leading to data incompatibility. For example, an error occurs when the transformers package is imported after the package update.
The user code has a bug, leading to memory overwriting or unauthorized memory access.
An unknown system error occurs. In this case, copy the training job again. If the fault persists, submit a service ticket.

If the training job succeeded before and no modification has been made, compare the logs in the two cases and check whether any dependency package has been updated in the pip source.
Figure 1 Log comparison
Use the local PyCharm to remotely access notebook for debugging.
If the fault persists, contact technical support engineers.

Before creating a training job, use the ModelArts development environment to debug your training code and minimize migration errors.

Use the notebook environment for online debugging. For details, see Using JupyterLab to Develop Models.
Use a local IDE (PyCharm or VS Code) to access the cloud environment for debugging. For details, see Using a Local IDE to Develop Models.

Parent topic: Service Code Issues