Training Job Failed Due to OOM

If a training job failed due to out of memory (OOM), possible symptoms as as follows:

Error code 137 is returned.
The log file contains error information with keyword killed.
Figure 1 Error log
Error message "RuntimeError: CUDA out of memory." is displayed in logs.
Figure 2 Error log
Error message "Dst tensor is not initialized" is displayed in TensorFlow logs.

The possible causes are as follows:

GPU memory is insufficient.
OOM occurred on certain nodes. This issue is typically caused by the node fault.

Modify hyperparameter settings to release unnecessary tensors.
1. Modify network parameters, such as batch_size, hide_layer, and cell_nums.
2. Release unnecessary tensors.
```
del tmp_tensor 
torch.cuda.empty_cache()
```
Use the local PyCharm to remotely access notebook for debugging.
If the fault persists, submit a service ticket to locate the fault or even isolate the affected node.

Before creating a training job, use the ModelArts development environment to debug the training code to maximally eliminate errors in code migration.

Use the online notebook environment for debugging. For details, see Using JupyterLab to Develop a Model.
Use the local IDE (PyCharm or VS Code) to access the cloud environment for debugging. For details, see Using the Local IDE to Develop a Model.

Parent topic: Hard Faults Due to Space Limit