User Manual

Driver FeaturesRev 2.2-1.0.1

Mellanox Technologies

4.4.6 Bonding IPoIB

To create an interface configuration script for the ibX and bondX interfaces, you should use the

standard syntax (depending on your OS).

Bonding of IPoIB interfaces is accomplished in the same manner as would bonding of Ethernet

interfaces: via the Linux Bonding Driver

• Network Script files for IPoIB slaves are named after the IPoIB interfaces (e.g: ifcfg-

ib0)

• The only meaningful bonding policy in IPoIB is High-Availability (bonding mode num-

ber 1, or active-backup)

• Bonding parameter "fail_over_mac" is meaningless in IPoIB interfaces, hence, the only

supported value is the default: 0 (or "none" in SLES1

For a persistent bonding IPoIB Network configuration, use the same Linux Network Scripts

semantics, with the following exceptions/ additions:

• In the bonding master configuration file (e.g: ifcfg-bond0), in addition to Linux bond-

ing semantics, use the following parameter: MTU=65520

65520 is a valid MTU value only if all IPoIB slaves operate in Connected mode (See

Section 4.4.2, “IPoIB Mode Setting”, on page 76) and are configured with the

same value. For IPoIB slaves that work in datagram mode, use MTU=2044. If you do

not set the correct MTU or do not set MTU at all, performance of the interface might

decrease.

• In the bonding slave configuration file (e.g: ifcfg-ib0), use the same Linux Network

Scripts semantics. In particular: DEVICE=ib0

• In the bonding slave configuration file (e.g: ifcfg-ib0.8003), the line TYPE=InfiniBand

is necessary when using bonding over devices configured with partitions (p_key)

• For RHEL users:

In /etc/modprobe.b/bond.conf add the following lines:

alias bond0 bonding

• For SLES users:

It is necessary to update the MANDATORY_DEVICES environment variable in /etc/syscon-

fig/network/config with the names of the IPoIB slave devices (e.g. ib0, ib1, etc.). Otherwise,

bonding master may be created before IPoIB slave interfaces at boot time.

It is possible to have multiple IPoIB bonding masters and a mix of IPoIB bonding master and

Ethernet bonding master

. However, It is NOT possible to mix Ethernet and IPoIB slaves under

the same bonding master

Restarting openibd does no keep the bonding configuration via Network Scripts. You

have to restart the network service in order to bring up the bonding master. After the

configuration is saved, restart the network service by running: /etc/init.d/network

restart.