HP StorageWorks Scalable File Share System Installation and Upgrade Guide Version 2.2

Preparing for the installation2–18
Recommendations:
• If possible, do not use routers in the network between client nodes and servers; using routers in the
network may impact performance adversely.
• If your switch allows it, give your client nodes sequential MAC addresses for Gigabit Ethernet
interconnect interfaces. For many switch types, this results in the most even balance between server
slave devices and reduces the possibility of one server link being significantly more heavily loaded
than others.
2.4.3 Configuring the Voltaire InfiniBand interconnects
NOTE: Use the worksheet in Table B-3 in Appendix B to record the information you gather in this section.
If you are using a Voltaire InfiniBand interconnect to connect the servers in the HP SFS system to the client
nodes, you must configure at least one InfiniBand IP address (IPoIB address) to the Host Channel Adapter
(HCA) in each server in the HP SFS system.
The Lustre NID (Network ID) for an InfiniBand interconnect is the address of the first IPoIB interface (usually
ipoib0). The IPoIB addresses are used only to establish connections (through the ARP protocol) between
the client nodes and the servers in the HP SFS system. The IPoIB addresses are not used by the LNET network
layer for file data traffic; all file data traffic uses the dedicated InfiniBand native protocol.
You can configure partitions on your InfiniBand interconnect; for more information, see Section 2.4.3.1. If
you do not intend to use partitioning, you can proceed directly to Section 2.4.3.2.
2.4.3.1 InfiniBand interconnect partitioning
Partitioning is an optional configuration that enforces isolation among systems sharing an InfiniBand
interconnect. One partition key (Pkey) is assigned to each partition. Please see Appendix D of this Guide
for an example of how to configure InfiniBand partitions at the switch; also, refer to the Voltaire HCA 4XX
User Manual and the Voltaire Switch Manual to obtain more information about InfiniBand partitioning.
Pkeys are configured at the InfiniBand switch, on a per port (host) basis. Up to 16 keys (Pkey0 through
Pkey15) can be configured per port. A collection of nodes with the same Pkey are referred to as being
members of a partition. There are two types of membership: Limited and Full. The high-order bit of the
partition key is used to record the type of membership; the values are 0 for Limited membership and 1 for
Full membership. Limited members cannot accept information from other Limited members, but
communication is allowed between every other combination of membership types. Pkey0 must always be
configured with a value of either 0xffff or 0x7fff.
At the host side, the ib-setup command is used to configure the IPoIB interfaces. A host can join one or
multiple partitions, and a Pkey must be provided for each interface. The host is authorized to join the
corresponding partition only if the same Pkey has been configured for the host's port at the switch. The
/proc/voltaire/ipoib-ud/control text file provides information about the membership status of
each IPoIB interface.
If the InfiniBand interconnect is not partitioned, you must configure a single IP interface (ipoib0) with the
default Pkey value 0xffff; this means the servers will accept traffic from any other system. When the switch
is partitioned, one interface (ipoib0, ipoib1, and so on) must be configured for each partition where the
HP SFS system will be used. Each partition must have a different Pkey.
IB partitioning for HP SFS systems can be configured in three ways:
• Using the default partition for the HP SFS traffic.
In such a configuration:
• The HP SFS servers only join the default partition (in the same way as in a non-partitioned
interconnect).