HP-MPI User's Guide (11th Edition)

Understanding HP-MPI
MPI options
Chapter 3 151
% setenv MPI_IB_CARD_ORDER <card#>[:port#]
Where:
card# ranges from 0 to N-1
port# ranges from 0 to 1
Card:port can be a comma separated list which drives the assignment of
ranks to cards and ports within the cards.
Note that HP-MPI numbers the ports on a card from 0 to N-1, whereas
utilities such as vstat display ports numbered 1 to N.
Examples:
To use the 2nd IB card:
% mpirun -e MPI_IB_CARD_ORDER=1 ...
To use the 2nd port of the 2nd card:
% mpirun -e MPI_IB_CARD_ORDER=1:1 ...
To use the 1st IB card:
% mpirun -e MPI_IB_CARD_ORDER=0 ...
To assign ranks to multiple cards:
% mpirun -e MPI_IB_CARD_ORDER=0,1,2
will assign the local ranks per node in order to each card.
% mpirun -hostlist "host0 4 host1 4"
creates ranks 0-3 on host0 and ranks 4-7 on host1. Will assign rank 0 to
card 0, rank 1 to card 1, rank 2 to card 2, rank 3 to card 0 all on host0.
And will assign rank 4 to card 0, rank 5 to card 1, rank 6 to card 2, rank
7 to card 0 all on host1.
% mpirun -hostlist -np 8 "host0 host1"
creates ranks 0 through 7 alternatingly on host0, host1, host0, host1, etc.
Will assign rank 0 to card 0, rank 2 to card 1, rank 4 to card 2, rank 6 to
card 0 all on host0. And will assign rank 1 to card 0, rank 3 to card 1,
rank 5 to card 2, rank 7 to card 0 all on host1.
MPI_IB_PKEY HP-MPI supports IB partitioning via Mellanox VAPI and
OFED Verbs API.