HP-MPI User's Guide (11th Edition)

Understanding HP-MPI
MPI options
Chapter 3154
from MPI_GLOBMEMSIZE, which is the total shared memory across all the
ranks on the host. MPI_RANKMEMSIZE takes precedence over
MPI_GLOBMEMSIZE if both are set. Both MPI_RANKMEMSIZE and
MPI_GLOBMEMSIZE are mutually exclusive to MPI_SMEMCNTL. If
MPI_SHMEMCNTL is set, then the user cannot set the other two, and vice
versa.
MPI_PIN_PERCENTAGE MPI_PIN_PERCENTAGE communicates the
maximum percentage of physical memory (see MPI_PHYSICAL_MEMORY)
that can be pinned at any time. The default is 20%.
% export MPI_PIN_PERCENTAGE=30
The above example permits the HP-MPI library to pin (lock in memory)
up to 30% of physical memory. The pinned memory is shared between
ranks of the host that were started as part of the same mpirun
invocation. Running multiple MPI applications on the same host can
cumulatively cause more than one application’s MPI_PIN_PERCENTAGE to
be pinned. Increasing MPI_PIN_PERCENTAGE can improve communication
performance for communication intensive applications in which nodes
send and receive multiple large messages at a time, such as is common
with collective operations. Increasing MPI_PIN_PERCENTAGE allows more
large messages to be progressed in parallel using RDMA transfers,
however pinning too much of physical memory may negatively impact
computation performance. MPI_PIN_PERCENTAGE and
MPI_PHYSICAL_MEMORY are ignored unless InfiniBand or Myrinet GM is
in use.
MPI_SHMEMCNTL MPI_SHMEMCNTL controls the subdivision of each
process’s shared memory for the purposes of point-to-point and collective
communications. It cannot be used in conjunction with
MPI_GLOBMEMSIZE. The MPI_SHMEMCNTL syntax is a comma separated list
as follows:
nenv, frag, generic
where
nenv Specifies the number of envelopes per process pair. The
default is 8.
frag Denotes the size in bytes of the message-passing
fragments region. The default is 87.5 percent of shared
memory after mailbox and envelope allocation.