Creating a Multiple-Node Multi-PU Distributed Training Job (DistributedDataParallel)

Training Process

Compared with DataParallel, DistributedDataParallel can start multiple processes for computing, greatly improving compute resource usage. Based on torch.distributed, DistributedDataParallel has obvious advantages over DataParallel in the distributed computing case. The process is as follows:

1. Initializes the process group.
2. Creates a distributed parallel model. Each process has the same model and parameters.
3. Creates a distributed sampler for data distribution to enable each process to load a unique subset of the original dataset in a mini batch.
4. Parameters are organized into buckets based on their shapes or sizes, which are generally determined by each layer of the network that requires parameter update in a neural network model.
5. Each process does its own forward propagation and computes its gradient.
6. After all parameter gradients at a bucket are obtained, communication is performed for gradient averaging.
7. Each GPU updates model parameters.

The detailed flowchart is as follows.

Figure 1 Multi-node multi-PU parallel training

Code Modifications

• Multi-process startup
• New variables such as rank ID and world_size are used along with the TCP protocol.
• Sampler for data distribution to avoid duplicate data between different processes
• Model distribution: DistributedDataParallel(model)
• Model saved in GPU 0

import torch
class Net(torch.nn.Module):
	pass

model = Net().cuda()

### DistributedDataParallel Begin ###
model = torch.nn.parallel.DistributedDataParallel(Net().cuda())
### DistributedDataParallel End ###

Multi-Node Distributed Debugging Adaptation and Code Example

• Example: Creating a DDP Distributed Training Job (PyTorch + GPU): describes the procedure and code example of distributed debugging adaptation.
• Example: Creating a DDP Distributed Training Job (PyTorch + NPU): provides a complete code sample of distributed parallel training for the classification task of ResNet18 on the CIFAR-10 dataset.

FAQs

1. How Do I Use Different Datasets in the Sample Code?
   • To use the CIFAR-10 dataset in the preceding code, download and decompress the dataset and upload it to the OBS bucket. The file directory structure is as follows:

     DDP
     |--- main.py
     |--- input_dir
     |------ cifar-10-batches-py
     |-------- data_batch_1
     |-------- data_batch_2
     |-------- ...

     DDP is the code directory specified during training job creation, main.py is the preceding code example (the boot file specified during training job creation), and cifar-10-batches-py is the unzipped dataset folder (stored in input_dir).

   • To use user-defined random data, change the value of custom_data in the code example to true.

     parser.add_argument('--custom_data', default='true')

     Then, run main.py. The parameters for creating a training job are the same as those shown in the preceding figure.

2. Why Can I Leave the IP Address of the Master Node Blank for DDP?

   The init method parameter in parser.add_argument('--init_method', default=None, help='tcp_port') contains the IP address and port number of the master node, which are automatically input by the platform.