日志提示“no socket interface found”

问题现象

import os
os.environ["NCCL_DEBUG"] = "INFO"

会出现如下错误：

job0879f61e-job-base-pda-2-0:712:71 2 [0] bootstrap.cc:37 NCCL WARN Bootstrap : no socket interface found
job0879f61e-job-base-pda-2-0:712:712 [0] NCCL INFO init.cC:128 -> 3
job0879f61e-job-base-pda-2-0:712:712 [0] NCCL INFO bootstrap.cc:76 -> 3
job0879f61e-job-base-pda-2-0:712:712 [0] NCCL INFO bootstrap.cc:245 -> 3	job0879f61e-job-base-pda-2-0:712:712 [0] NCCL INFO bootstrap.cc:266 -> 3
Traceback (most recent call last):
	File "train_net.py", line 1923, in <module>
		main_worker(args)
	File "train net.py", line 355, in main_ worker
		network = torch.nn.parallel.DistributedDataParallel(network, device_ids=device_ids, find_unused _parameters=True)
	File "/home/work/anaconda/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 298, in init_self.broadcast bucket_size)
	File "/home/work/anaconda/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 480, in _distributed broacIcast coalesced	dist. broadcast coalesced(seIf.process group, tensors, buffer size)
RuntimeError: NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:374, internal error

原因分析

可能原因如下：

• 原因1：未设置环境变量NCCL_IB_TC、NCCL_IB_GID_INDEX、NCCL_IB_TIMEOUT，因此会导致通信速度慢且不稳定，最后造成IB通信断连，偶发上述现象。
• 原因2：NCCL_SOCKET_IFNAME设置错误。当用户的NCCL版本低于2.14时，则需要手动设置NCCL_SOCKET_IFNAME环境变量。

处理方法

• 针对原因1，需要在代码中补充如下环境变量。

  import os
  os.environ["NCCL_IB_TC"] = "128"
  os.environ["NCCL_IB_GID_INDEX"] = "3"
  os.environ["NCCL_IB_TIMEOUT"] = "22"

• 针对原因2，需要在代码中设置环境变量NCCL_SOCKET_IFNAME。

  import os
  os.environ["NCCL_SOCKET_IFNAME"] = "eth0"

  

  只有当用户的NCCL版本低于2.14时，才需要进行以上设置。