High Performance ComputingHigh Performance Computing

Compute
Elastic Cloud Server
Bare Metal Server
Auto Scaling
Image Management Service
Dedicated Host
FunctionGraph
Networking
Virtual Private Cloud
Elastic IP
Elastic Load Balance
NAT Gateway
Direct Connect
Virtual Private Network
Domain Name Service
VPC Endpoint
Cloud Connect
Enterprise Switch
Security & Compliance
Anti-DDoS
Web Application Firewall
Host Security Service
Data Encryption Workshop
Database Security Service
Advanced Anti-DDoS
Data Security Center
Container Guard Service
Situation Awareness
Managed Threat Detection
Compass
Cloud Certificate Manager
Anti-DDoS Service
Databases
Relational Database Service
Document Database Service
Data Admin Service
Data Replication Service
GaussDB NoSQL
GaussDB(for MySQL)
Distributed Database Middleware
GaussDB(for openGauss)
Developer Services
ServiceStage
Distributed Cache Service
Simple Message Notification
Application Performance Management
Application Operations Management
Blockchain
API Gateway
Cloud Performance Test Service
Distributed Message Service for Kafka
Distributed Message Service for RabbitMQ
Distributed Message Service for RocketMQ
Cloud Service Engine
DevCloud
ProjectMan
CodeHub
CloudRelease
CloudPipeline
CloudBuild
CloudDeploy
Cloud Communications
Message & SMS
Cloud Ecosystem
Marketplace
Partner Center
User Support
My Account
Billing Center
Cost Center
Resource Center
Enterprise Management
Service Tickets
HUAWEI CLOUD (International) FAQs
ICP License Service
Support Plans
Customer Operation Capabilities
Partner Support Plans
Professional Services
enterprise-collaboration
Meeting
IoT
IoT
Intelligent EdgeFabric
DeveloperTools
SDK Developer Guide
API Request Signing Guide
Terraform
Koo Command Line Interface
Updated at: May 09, 2022 GMT+08:00

Open MPI Delivered with the IB Driver

Scenarios

This section describes how to install and use Open MPI (version 4.0.2a1 is used as an example) delivered with the IB driver on a BMS.

Perform the operations on each BMS in a cluster.

Prerequisites

Password-free login has been configured between BMSs in the cluster.

Procedure

  1. Check whether the IB driver has been installed.

    1. Run the following commands to check whether the IB driver has been installed:

      $ ls /usr/mpi/gcc/openmpi-4.0.2a1/bin/mpirun

      $ rpm -qa | grep mlnx-ofa

      Figure 1 Checking the IB driver
    2. Check the command output.
      • If information shown in Figure 1 is displayed, the IB driver has been installed. Then, go to 3.
      • If the IB driver has not been installed, go to 2.

  2. Install the IB driver.

    1. Download the installation package MLNX_OFED_LINUX-4.6-1.0.1.1-rhel7.6alternate-aarch64.tgz.

      Download path: https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/

      Figure 2 IB driver download center
    2. Run the following commands to install the software package:

      # yum install tk tcl -y

      # tar -xvf MLNX_OFED_LINUX-4.6-1.0.1.1-rhel7.6alternate-aarch64.tgz

      # cd MLNX_OFED_LINUX-4.6-1.0.1.1-rhel7.6-x86_64/

      # ./mlnxofedinstall

  3. Install and configure UCX.

    1. Download the UCX installation package.

      # cd /opt && wget https://github.com/openucx/ucx/releases/download/v1.6.0/ucx-1.6.0.tar.gz

    1. Decompress the package.

      # tar -xvf ucx-1.6.0.tar.gz

    2. Compile and install UCX.

    # cd /opt/ucx-1.6.0

    # yum install autoconf automake libtool numactl-devel -y

    # ./contrib/configure-release --prefix=/opt/ucx160 --enable-optimizations

    # make && make install

  4. Configure UCX.

    1. Create non-root user rhel.

      # useradd rhel; su - rhel

    2. Obtain PKEY, delete the third digit of PKEY, and use this value to replace {pkey} in 4.d.

      # cat /sys/class/infiniband/mlx5_0/ports/1/pkeys/* | grep -v 0000 | head -n1

      For example, if the obtained PKEY is 0x8f05, delete the third digit to obtain 0xf05.

      Figure 3 Obtaining PKEY
    3. Obtain UCX PKEY.

      # ucx_info -c | grep -i pkey > ucx.env

    4. Replace PKEY in UCX.

      # sed -i 's/0x[a-f0-9]*/{pkey}/g' ucx.env

      In this example, run sed -i 's/0x[a-f0-9]*/0xf05/g' ucx.env.

      Figure 4 Replacing PKEY in UCX
    5. Set UCX PKEY as an environment variable.

      # sed -i 's/^UCX/export UCX/g' ucx.env

      # cat ucx.env >> ~/.bashrc

  5. Configure MPI environment variables.

    1. Use VIM to open the ~/.bashrc file and add the following data to the file:

      export PATH=$PATH:/usr/mpi/gcc/openmpi-4.0.2a1/bin

      export LD_LIBRARY_PATH=/usr/mpi/gcc/openmpi-4.0.2a1/lib64

    2. Run the following command to check whether the MPI environment variables are correct:

      $ which mpirun

      Figure 5 Viewing the Open MPI environment variables

      If information shown in Figure 5 is displayed, environment variables have been configured.

    3. Run the following command to run Open MPI delivered with the IB driver on a BMS:

      #mpirun -np 2 -mca btl_openib_if_include "mlx5_0:1" -x MXM_IB_USE_GRH=y /usr/mpi/gcc/openmpi-3.1.0rc2/tests/imb/IMB-MPI1 PingPong

      Figure 6 Running Open MPI delivered with the IB driver

Did you find this page helpful?

Failed to submit the feedback. Please try again later.

Which of the following issues have you encountered?







Please complete at least one feedback item.

Content most length 200 character

Content is empty.

OK Cancel