HP-MPI Version 2.2.7 for Linux Release Note

1.2.7.10 InfiniBand support with MPI_Comm_connect() and MPI_Comm_accept()
HP–MPI Version 2.2.7 supports MPI_Comm_connect() and MPI_Comm_accept() over
InfiniBand processes using the IBV protocol. Both sides must have InfiniBand support enabled
and use the same InfiniBand parameter settings.
MPI_Comm_connect() and MPI_Comm_accept() need a port name, which is the IP and port
at the root process of the accept side. First, a TCP connection is established between the root
process of both sides. Next, TCP connections are set up among all the processes. Finally, IBV
InfiniBand connections are established among all process-pairs and the TCP connections are
closed.
1.2.7.11 -ha permitted with IBV
HP-MPI's -ha option for high availability has been expanded substantially in this release. In
prior releases, the -ha option forced the use of TCP for communication. Now, both TCP and IBV
are possible network selections when using -ha. If no forced selection criteria (i.e. -TCP, -IBV,
or equivalent MPI_IC_ORDER setting) is specified by the user, then IBV is selected where available,
and TCP is used otherwise.
1.2.7.12 InfiniBand multiple fabric support
If a cluster has multiple InfiniBand cards in each node, connected to physically separated fabrics,
HP-MPI requires that each fabric has its own subnet ID. When the subnet IDs are the same,
HP-MPI cannot identify which ports are on the same fabric, and the connection setup is likely
to be less than desirable.
If all the fabrics have a unique subnet ID, by default HP-MPI assumes that the ports are connected
based on the ibv_devinfo output port order on each node. All the port 1s are assumed to be
connected to fabric 1, and all the port 2s are assumed to be connected to fabric 2. If all the nodes
in the cluster have the first InfiniBand port connected to the same fabric with the same subnet
ID, HP-MPI can run without any additional fabric topology hints.
If the physical fabric connections do not follow the convention described above, then the fabric
topology information must be supplied to HP-MPI. The ibv_devinfo -v utility can be used
on each node within the cluster to get the port GID. If all the nodes in the cluster are connected
in the same way and each fabric has a unique subnet ID, the ibv_devinfo command only needs
to be done on one node.
The MPI_IB_PORT_GID environment variable is used to specify which InfiniBand fabric subnet
should be used by HP-MPI to make the initial InfiniBand connectin between the nodes.
For example, if the user runs HP-MPI on two nodes with the following ibv_devinfo -v output,
on the first node:
$ ibv_devinfo -v
hca_id: mthca0
fw_ver: 4.7.0
node_guid: 0008:f104:0396:62b4
....
max_pkeys: 64
local_ca_ack_delay: 15
port: 1
state: PORT_ACTIVE (4)
max_mtu: 2048 (4)
....
phys_state: LINK_UP (5)
GID[ 0]: fe80:0000:0000:0000:0008:f104:0396:62b5
port: 2
state: PORT_ACTIVE (4)
max_mtu: 2048 (4)
....
phys_state: LINK_UP (5)
GID[ 0]: fe80:0000:0000:0001:0008:f104:0396:62b6
The following is the second node configuration:
$ ibv_devinfo -v
hca_id: mthca0
1.2 What’s in This Version 15