HP StorageWorks Scalable File Share Release Notes - Version 2.3

Documentation updates and corrections 1–39
1.8.3.13 Section 5a Configuring multiple InfiniBand interfaces
SFS V2.3 adds a new configuration option that allows to distribute the Lustre LNET traffic across multiple IB
interfaces (rails). Refer to sections 7.11.1a and 7.11.1a.3 of the SFS System User Guide for more detailed
information.
• Lustre sends LNET traffic through a given IB interface depending on the destination node.
• All network traffic to a same SFS server will be directed through a single IB interface.
• Every IB interface to be used must be configured with an IPoIB address.
• The LNET network option in /etc/modprobe.conf.lustre is automatically configured by the
sfsconfig.
• sfsconfig does not order multiple IB LNETs optimally; therefore, the order must be set manually. (It
does not change the ordering it finds in the config file, so running sfsconfig again will not break
the order that has been manually set.)
• The typical LNET setting for a client node with n+1 IB interfaces named ib0 to ibn would be options
lnet network=o2ibn(ibn),...,o2ib1(ib1),o2ib0(ib0).
Notice that the LNETs are ordered by decreasing index!
• Enabling multiple IB interfaces requires to setup multiple LNETs on the SFS server. See section 7.11.1a
of the SFS System User Guide for more detailed information
WARNING! The use of multiple IB interfaces will not be effective unless the target SFS server is configured
appropriately.
In summary, after the server is configured, configure the clients:
1. Configure the InfiniBand IPoIB interfaces using the standard Linux network management commands.
2. Run sfsconfig -s SERVER all to query the server config, and set the client LNET configuration
accordingly.
3. Edit /etc/modprobe.conf.lustre, and make sure that the LNET networks are ordered in
decreasing sequence.
/etc/modprobe.conf.lustre install osc /sbin/modprobe
lquota ;
/sbin/modprobe --ignore-
install osc
Required for quota support, if the
server supports it.
/etc/modprobe.conf.lustre remove mdc /sbin/modprobe -r -
-ignore-remove mdc ;
/sbin/modprobe -r lquota
Required for quota support, if the
server supports it.
/etc/sfstab All lines Updates server and filesystem LNET
addresses and options, based on the
up-to-date server capabilities fetched
on the server web site.
/etc/sfstab.proto All lines Updates server and filesystem LNET
addresses and options, based on the
up-to-date server capabilities fetched
on the server web site.
Configuration File Setting Description