User Manual

Driver FeaturesRev 2.1-1.0.6

Mellanox Technologies

4.3.6 Bonding IPoIB

To create an interface configuration script for the ibX and bondX interfaces, you should use the

standard syntax (depending on your OS).

Bonding of IPoIB interfaces is accomplished in the same manner as would bonding of Ethernet

interfaces: via the Linux Bonding Driver.

• Network Script files for IPoIB slaves are named after the IPoIB interfaces (e.g: ifcfg-

ib0)

• The only meaningful bonding policy in IPoIB is High-Availability (bonding mode num-

ber 1, or active-backup)

• Bonding parameter "fail_over_mac" is meaningless in IPoIB interfaces, hence, the only

supported value is the default: 0 (or "none" in SLES11)

For a persistent bonding IPoIB Network configuration, use the same Linux Network Scripts

semantics, with the following exceptions/ additions:

• In the bonding master configuration file (e.g: ifcfg-bond0), in addition to Linux bond-

ing semantics, use the following parameter: MTU=65520

• In the bonding slave configuration file (e.g: ifcfg-ib0), use the same Linux Network

Scripts semantics. In particular: DEVICE=ib0

• In the bonding slave configuration file (e.g: ifcfg-ib0.8003), the line TYPE=InfiniBand

is necessary when using bonding over devices configured with partitions (p_key)

• For RHEL users:

In /etc/modprobe.b/bond.conf add the following lines:

• For SLES users:

It is necessary to update the MANDATORY_DEVICES environment variable in /etc/sysconfig/net-

work/config with the names of the IPoIB slave devices (e.g. ib0, ib1, etc.). Otherwise, bonding mas-

ter may be created before IPoIB slave interfaces at boot time.

It is possible to have multiple IPoIB bonding masters and a mix of IPoIB bonding master and Ethernet

bonding master. However, It is NOT possible to mix Ethernet and IPoIB slaves under the same bond-

ing master

5 packets transmitted, 5 received, 0% packet loss, time 3999ms rtt min/avg/max/mdev =

0.044/0.058/0.079/0.014 ms, pipe 2

65520 is a valid MTU value only if all IPoIB slaves operate in Connected mode (See

Section 4.3.2, “IPoIB Mode Setting”, on page 56) and are configured with the

same value. For IPoIB slaves that work in datagram mode, use MTU=2044. If you do

not set the correct MTU or do not set MTU at all, performance of the interface might

decrease.

alias bond0 bonding

Restarting openibd does no keep the bonding configuration via Network Scripts. You

have to restart the network service in order to bring up the bonding master. After the

configuration is saved, restart the network service by running: /etc/init.d/network

restart.