Help Center/ GaussDB(DWS)/ Standard Data Warehouse (9.1.0.x)/ RDMA Configuration
Updated on 2024-09-30 GMT+08:00
View PDF

RDMA Configuration

The current TCP/IP stack is inadequate for high-concurrency and low-latency I/O applications, such as high-performance computing and big data analysis. This is because conventional TCP/IP networking relies on the kernel to transmit messages. This communication mode incurs high costs for data transfer and duplication. Remote Direct Memory Access (RDMA) is developed to reduce the data processing delay on the server side during network transmission. GaussDB(DWS) supports the RDMA/RoCE protocol, which significantly improves the query performance in high-concurrency and large-bandwidth scenarios.

RDMA causes severe performance deterioration when packet loss occurs on the network. The RoCE protocol is not supported by the IB network. Therefore, priority flow control (PFC) needs to be configured on the Ethernet to achieve losslessness.

Procedure

  1. Install the official driver.

    1. Download the official driver and install it.

      Take Mellanox as an example. Obtain the OS version 5.4 from the following link:

      https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/

    2. Install the driver.

      After decompressing the driver package, run the ./mlnxofedinstall command to install the driver. If the system displays a message indicating that the OS version does not match, run the following command to repack the installation package and use the new installation package to install the driver:

      1
      		 
      ./mlnxofedinstall --force --without-fw-update --add-kernel-support

  2. Configure PFC on the host.

    Run the following commands or scripts as the root user to upgrade the CCE components so that they can adapt to the new kernel: Replace enp1s0f0 with the actual RDMA NIC name.

    1
    		 
    mlnx_qos -i enp1s0f0 --pfc 1,0,0,0,0,0,0,0 --trus dscp

    If in the output of PFC configuraiton command, the configuration items of priority 0 are all enabled, the configuration is successful.

    • If the bond network is used, you need to configure each RDMA network adapter. You can run the cat /proc/net/bonding/bond0 command to query the physical network adapter corresponding to the bond network adapter.
    • /etc/rc.d/rc.local needs to be added to the mlnx_qos command for persistence configuration.

  3. Configure PFC on the switch.

    To avoid damaging the network environment, only qualified personnel should configure switches.

    To ensure proper configuration, replace the example port name with the actual name of the port used by the RoCE network. You can either perform the same operations on each port individually or group the ports that use the RoCE network.
    1. Configure PFC. 
      1
2
3
4
      		 
      sys
interface Port name
dcb pfc enable mode manual
commit
    2. Set the priority. 
      1
2
3
      		 
      dcb pfc
priority 0
commit
    3. Configure ECN. 
      1
2
3
      		 
      interface Port name
qos queue 0 ecn
commit
    4. Persistence configuration.

      After the configuration is complete, run the dis cur command to check whether the configuration takes effect.

      1
      		 
      dis cur | in pfc

      After confirming that the parameters are correct, run the save command to save the parameters for persistence configuration.