Horovod/MPI/MindSpore-GPU

Horovod/MPI/MindSpore-GPU Startup Principle

Specifications and number of nodes

In this case, GPU: 8 × GP-Vnt1 | CPU: 72 cores | Memory: 512 GB is used as an example to describe how to allocate ModelArts resources for single-node and distributed jobs.

For a single-node job (running on only one node), ModelArts starts a training container that exclusively uses the resources on the node.

For a distributed job (running on more than one node), there are as many workers as the nodes that are selected during job creation. Each worker is allocated with the compute resources of the selected specification. For example, if there are 2 compute nodes, two workers will be started, and each worker owns the compute resources of GPU: 8 × GP-Vnt1 | CPU: 72 cores | Memory: 512 GB.

Network communication

• For a single-node job, no network communication is required.
• For a distributed job, network communications are required in nodes and between nodes.

In nodes

NVLink and shared memory are used for communication.

Between nodes

If there is more than one compute node, PyTorch distributed training will be started. The following figure shows the network communications between workers in PyTorch distributed training. Workers can communicate with each other using the container network and a 100-Gbit/s InfiniBand or RoCE NIC. RoCE NICs are described specifically for certain specifications. The containers can communicate through DNS domain names, which is suitable for small-scale point-to-point communication that requires average network performance. The InfiniBand and RoCE NICs are suitable for distributed training jobs using collective communication that require high-performance network.

Figure 1 Network communications for distributed training

Boot Commands

By default, the training service uses the python interpreter in the job image to start up the training script. To obtain the python interpreter, run the which python command. The working directory during startup is /home/ma-user/user-job-dir/<The code directory name>, which is the directory returned by running pwd or os.getcwd() in python.

Boot commands

mpirun \        
-np ${OPENMPI_NP} \       
-hostfile ${OPENMPI_HOST_FILE_PATH} \        
-mca plm_rsh_args "-p ${SSHD_PORT}" \        
-tune ${TUNE_ENV_FILE} \        
${OPENMPI_BIND_ARGS} \        
${OPENMPI_X_ARGS} \        
${OPENMPI_MCA_ARGS} \        
${OPENMPI_EXTRA_ARGS} \        
python <Relative path of the startup file> <Job parameters>

• OPENMPI_NP: number of processes started by mpirun. The default value is the number of GPUs multiplied by the number of nodes. Do not modify this value.
• OPENMPI_HOST_FILE_PATH: value of hostfile. Do not modify this value.
• SSHD_PORT: Port for SSH login. Do not modify this value.
• TUNE_ENV_FILE: environment variables of worker-0. Broadcast the following environment variables to other worker nodes of the current training job.
  • env with the MA_ prefix
  • env with the SHARED_ prefix
  • env with the S3_ prefix
  • env of PATH
  • env with the VC_WORKER_ prefix
  • env with the SCC prefix
  • env with the CRED prefix

    env|grep -E '^MA_|^SHARED_|^S3_|^PATH|^VC_WORKER_|^SCC|^CRED'|grep -v '=$'> ${TUNE_ENV_FILE}

• OPENMPI_BIND_ARGS: process pinning with the mpirun cpu command. The default setting is as follows:

  OPENMPI_BIND_ARGS="-bind-to none -map-by slot"

• OPENMPI_X_ARGS: -x parameters of the mpirun command. The default settings are as follows:

  OPENMPI_X_ARGS="-x LD_LIBRARY_PATH -x HOROVOD_MPI_THREADS_DISABLE=1 -x NCCL_DEBUG=INFO -x NCCL_SOCKET_IFNAME=ib0,bond0,eth0 -x NCCL_SOCKET_FAMILY=AF_INET -x NCCL_IB_DISABLE=0"

• OPENMPI_X_ARGS: -mca parameters of the mpirun command. The default setting is as follows:

  OPENMPI_MCA_ARGS="-mca pml ob1 -mca btl ^openib -mca plm_rsh_no_tree_spawn true"

• OPENMPI_EXTRA_ARGS: parameters passed to mpirun. The default value is empty.
• Relative path of the startup file: path of the startup file relative to /home/ma-user/user-job-dir/<The code directory name>
• Job parameters: running parameters configured for a training job

Figure 2 Creating a training job

Configure the parameters by referring to the above figure. Then, run the following command on the console background:

mpirun \        
-np ${np} \        
-hostfile ${OPENMPI_HOST_FILE_PATH} \        
-mca plm_rsh_args "-p ${SSHD_PORT}" \        
-tune ${TUNE_ENV_FILE} \        
${OPENMPI_BIND_ARGS} \        
${OPENMPI_X_ARGS} \        
${OPENMPI_MCA_ARGS} \        
${OPENMPI_EXTRA_ARGS} \        
python /home/ma-user/user-job-dir/gpu-train/train.py --datasets=obs://modelarts-train-test/gpu-train/data_url_0