Help Center/ High Performance Computing/ User Guide/ Typical Applications in the BMS Scenario/ Installing and Using MPIs (Kunpeng BMS)/ Open MPI Delivered with the IB Driver
Updated on 2022-05-09 GMT+08:00
View PDF
Share

Open MPI Delivered with the IB Driver

Scenarios

This section describes how to install and use Open MPI (version 4.0.2a1 is used as an example) delivered with the IB driver on a BMS.

Perform the operations on each BMS in a cluster.

Prerequisites

Password-free login has been configured between BMSs in the cluster.

Procedure

  1. Check whether the IB driver has been installed.

    1. Run the following commands to check whether the IB driver has been installed:

      $ ls /usr/mpi/gcc/openmpi-4.0.2a1/bin/mpirun

      $ rpm -qa | grep mlnx-ofa

      Figure 1 Checking the IB driver
    2. Check the command output.
      • If information shown in Figure 1 is displayed, the IB driver has been installed. Then, go to 3.
      • If the IB driver has not been installed, go to 2.

  2. Install the IB driver.

    1. Download the installation package MLNX_OFED_LINUX-4.6-1.0.1.1-rhel7.6alternate-aarch64.tgz.

      Download path: https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/

      Figure 2 IB driver download center
    2. Run the following commands to install the software package:

      # yum install tk tcl -y

      # tar -xvf MLNX_OFED_LINUX-4.6-1.0.1.1-rhel7.6alternate-aarch64.tgz

      # cd MLNX_OFED_LINUX-4.6-1.0.1.1-rhel7.6-x86_64/

      # ./mlnxofedinstall

  3. Install and configure UCX.

    1. Download the UCX installation package.

      # cd /opt && wget https://github.com/openucx/ucx/releases/download/v1.6.0/ucx-1.6.0.tar.gz

    1. Decompress the package.

      # tar -xvf ucx-1.6.0.tar.gz

    2. Compile and install UCX.

    # cd /opt/ucx-1.6.0

    # yum install autoconf automake libtool numactl-devel -y

    # ./contrib/configure-release --prefix=/opt/ucx160 --enable-optimizations

    # make && make install

  4. Configure UCX.

    1. Create non-root user rhel.

      # useradd rhel; su - rhel

    2. Obtain PKEY, delete the third digit of PKEY, and use this value to replace {pkey} in 4.d.

      # cat /sys/class/infiniband/mlx5_0/ports/1/pkeys/* | grep -v 0000 | head -n1

      For example, if the obtained PKEY is 0x8f05, delete the third digit to obtain 0xf05.

      Figure 3 Obtaining PKEY
    3. Obtain UCX PKEY.

      # ucx_info -c | grep -i pkey > ucx.env

    4. Replace PKEY in UCX.

      # sed -i 's/0x[a-f0-9]*/{pkey}/g' ucx.env

      In this example, run sed -i 's/0x[a-f0-9]*/0xf05/g' ucx.env.

      Figure 4 Replacing PKEY in UCX
    5. Set UCX PKEY as an environment variable.

      # sed -i 's/^UCX/export UCX/g' ucx.env

      # cat ucx.env >> ~/.bashrc

  5. Configure MPI environment variables.

    1. Use VIM to open the ~/.bashrc file and add the following data to the file:

      export PATH=$PATH:/usr/mpi/gcc/openmpi-4.0.2a1/bin

      export LD_LIBRARY_PATH=/usr/mpi/gcc/openmpi-4.0.2a1/lib64

    2. Run the following command to check whether the MPI environment variables are correct:

      $ which mpirun

      Figure 5 Viewing the Open MPI environment variables

      If information shown in Figure 5 is displayed, environment variables have been configured.

    3. Run the following command to run Open MPI delivered with the IB driver on a BMS:

      #mpirun -np 2 -mca btl_openib_if_include "mlx5_0:1" -x MXM_IB_USE_GRH=y /usr/mpi/gcc/openmpi-3.1.0rc2/tests/imb/IMB-MPI1 PingPong

      Figure 6 Running Open MPI delivered with the IB driver