HP-MPI User's Guide (11th Edition)

Understanding HP-MPI
Improved deregistration via ptmalloc (Linux only)
Chapter 3166
Improved deregistration via ptmalloc (Linux
only)
To achieve the best performance on RDMA enabled interconnects like
InfiniBand and Myrinet, the MPI library must be aware when memory is
returned to the system in malloc() and free() calls. To enable more robust
handling of that information, HP-MPI contains a copy of the ptmalloc
implementation and uses it by default.
For applications with particular needs, there are a number of available
modifications to this default configuration. To avoid using HP-MPI's
ptmalloc implementation and instead use the standard libc
implementation (or perhaps a malloc implementation contained in the
application), set the environment variable MPI_NO_MALLOCLIB at
runtime.
If the above option is applied so that the ptmalloc contained in HP-MPI
is not used, then there is a risk of MPI not being informed of when
memory is returned to the system. This can be alleviated with the
settings MPI_USE_MALLOPT_SBRK_PROTECTION and
MPI_USE_MALLOPT_AVOID_MMAP at runtime, which essentially results in
the libc malloc implementation not returning memory to the system.
There are cases where these two settings cannot keep libc from returning
memory to the system, in particular when multiple threads call
malloc/free at the same time. In these cases, the only remaining option is
to disable HP-MPI's lazy deregistration by giving the -ndd flag to
mpirun.
Note that in the default case where the ptmalloc contained within
HP-MPI is used, the above cases are all avoided and lazy deregistration
works correctly as is. So the above tunables are only recommended for
applications with special requirements concerning their malloc/free
usage.