HP-MPI User's Guide (11th Edition)

Understanding HP-MPI
Running applications on HP-UX and Linux
Chapter 3 89
The example above uses the max locked-in-memory address space in KB
units. The recommendation is to set the value to half of the physical
memory.
Machines can have multiple InfiniBand cards. By default each HP-MPI
rank selects one card for its communication, and the ranks cycle through
the available cards on the system, so the first rank uses the first card,
the second rank uses the second card, etc.
The environment variable MPI_IB_CARD_ORDER can be used to control
which card the different ranks select. Or, for increased potential
bandwidth and greater traffic balance between cards, each rank can be
instructed to use multiple cards by using the variable
MPI_IB_MULTIRAIL.
Lazy deregistration is a performance enhancement used by HP-MPI on
several of the high speed interconnects on Linux. This option is turned
on by default, and requires the application to be linked in such a way
that HP-MPI is able to intercept calls to malloc, munmap, etc. Most
applications are linked that way, but if one is not then HP-MPI's lazy
deregistration can be turned off with the command line -ndd.
Some applications decline to directly link against libmpi and instead
link against a wrapper library which is in turn linked against libmpi. In
this case it is still possible for HP-MPI's malloc etc. interception to be
used by supplying the --auxiliary option to the linker when creating
the wrapper library, by using a compiler flag such as -Wl, --auxiliary,
libmpi.so.
Note that dynamic linking is required with all InfiniBand use on Linux.
HP-MPI does not use the Connection Manager (CM) library with OFED.
VAPI The MPI_IB_CARD_ORDER card selection option and the -ndd
option described above for IBV applies to VAPI.
uDAPL The -ndd option described above for IBV applies to uDAPL.
GM The -ndd option described above for IBV applies to GM.
Elan HP-MPI supports the Elan3 and Elan4 protocols for Quadrics.
By default HP-MPI uses Elan collectives for broadcast and barrier.
If messages are outstanding at the time the Elan collective is entered
and the other side of the message enters a completion routine on the
outstanding message before entering the collective call, it is possible for