Help Center/ ModelArts/ Troubleshooting/ Training Jobs/ GPU Issues/ Error Message "RuntimeError: connect() timed out" Displayed in Logs
Updated on 2024-12-30 GMT+08:00
View PDF
Share

Error Message "RuntimeError: connect() timed out" Displayed in Logs

Symptom

When PyTorch is used for distributed training, the following error occurs.

Figure 1 Error log

Possible Causes

The possible causes are as follows:

If data had been copied before this issue occurred, the data copy process for all nodes was not completed at the same time. If you executed torch.distributed.init_process_group() when data copy was still in progress on certain nodes, the execution timed out.

Solution

If the issue is caused by asynchronous data copy between nodes and no barrier occurs, perform torch.distributed.init_process_group() before copying data, copy data based on local_rank()==0, call torch.distributed.barrier(), and wait until data copy is complete on all nodes. For details, see the following code: 
import moxing as mox
import torch

torch.distributed.init_process_group()
if local_rank == 0: 
    mox.file.copy_parallel(src,dst) 

torch.distributed.barrier()

Summary and Suggestions

Debug your training code in the ModelArts development environment before creating a job.
Parent topic: GPU Issues