IPoIB Functions

Updated on 2022-06-28 GMT+08:00

View PDF

IPoIB

Internet Protocol over InfiniBand (IPoIB) allows connection and data transmission through physical IB networks (IB card, cable, and switch on the server) over the IP protocol.

It provides a RDMA-based IP network simulation layer and allows you to run applications on the InfiniBand network without making any modification. However, IPoIB is inferior to RDMA in performance. Most applications use RDMA that provides high bandwidth and low latency, and some key applications use IPoIB.

NOTE:

Applications can run on the InfiniBand network over the IP protocol without any modification required.

Communication Modes

Two modes datagram and connected can be configured for IPoIB devices. The former provides unreliable and connectionless links, and the latter provides reliable links with connections.

In the datagram mode, the queue pair does not allow the packet size to exceed the MTU value of the IB link layer. The IPoIB header contains four bytes, and the IPoIB MTU value is less than that of the IB link layer.
In the connected mode, the queue pair allows packets larger than those of the IB link layer. In theory, the packet can contain a maximum of 65535 bytes. The connected mode has better performance, but consumes more memory. Most systems give higher priority to performance, and the connected mode is configured for the IB network port.

The NIC driver of the current version does not support the connected mode.

NOTE:

Due to NIC driver performance issues, the connected mode is disabled. Therefore, the NIC driver in the HPC solution does not support the connected mode.

IP Address Assignment Methods

Two methods are available, static configuration and DHCP.

Static configuration
IPoIB devices have a hardware address containing 20 bytes. The first four bytes is the queue pair number, the middle eight bytes is the subnet prefix, and the last eight bytes is the GUID.

You can query the hardware address of an IPoIB device only by running the ip command. If you run the ifconfig command, you cannot obtain a complete address. The following figure shows an example of a static IP address.

Figure 1 Static IP address
DHCP
A standard DHCP frame contains fields, such as hardware type (htype), MAC address length (hlen), and MAC address (chaddr). The MAC address is not long enough to contain the IPoIB hardware address. Therefore, field client-identifier is defined to identify DHCP sessions on the client. The client-identifier field is used to associate the IP address with the client. DHCP Server uses the field to distinguish IP addresses assigned to different clients.

The HPC solution uses the DHCP method to ensure that IP addresses are automatically assigned and configured.

Constraints

One IB NIC can be managed.
Similar to constraints of the InfiniBand NICs of BMSs, H2, HL1, and HI3 ECSs, ECSs using IPoIB cannot be migrated.
Similar to constraints of the InfiniBand NICs of H2, HL1, and HI3 ECSs, InfiniBand networks do not support security groups, QoS, and layer-3 and higher-layer network functions.
ECSs using IPoIB do not support ARP anti-spoofing and DHCP anti-spoofing due to NIC driver limitations.

Parent topic: Overview

Previous topic: HPC and Public Cloud

Next topic: Quota Adjustment