HP-MPI Version 2.2 for Linux Release Note

HP-MPI V2.2 for Linux Release Note
Known Problems and Workarounds
30
while RDMA operations are being performed on the same page, silent data corruption can
occur as RDMA operations continue to stream data to a page that has moved. To avoid
this, the Mellanox driver does not use copy-on-write behavior during a fork for pinned
pages. Instead, access to these pages by the child process will result in a segmentation
violation.
If you believe that your application is experiencing a problem due to this limitation,
HP-MPI provides a feature that may avoid the scenario from occurring. By setting the
environment variable MPI_PAGE_ALIGN_MEM, HP-MPI will page-align and page-pad libc
memory allocation requests which are large enough to be pinned during MPI message
transfer. This can result in slightly more memory being allocated, but reduces the
likelihood that a forked process will write to a page of memory that was also being used
for message transfer when a fork call occurred.
It is still possible to experience problems if you allocate a large buffer, start a message
transfer operation on part of the buffer, fork, and then access another part of the buffer.
Nevertheless, the use of MPI_PAGE_ALIGN_MEM will alleviate the most common problems
related to calling fork( ) while memory is pinned.
• 32-bit applications wishing to use InfiniBand on x86_64 systems should consider using
Sockets Direct Protocol (SDP). Set the LD_PRELOAD environment variable before running
an executable that uses sockets:
% setenv LD_PRELOAD /usr/voltaire/lib/sock-redirect.so
This will automatically and transparently convert to the use of SDP.
• The HP-MPI 2.2 library for Linux contains a dependency on pthread_self, which
requires libpthread to be included on the link line if the application is linked -static.
The mpicc, mpif90, etc. compiler wrapper scripts automatically add the necessary
-lpthread, but a user who links by hand will have to explicitly add -lpthread.
• On XC clusters using Fortran applications, a call to MPI_Abort results in a SIGTERM
being raised by default. The version of srun included with XC V2.1 would not see the
signal if the app was Fortran, and teardown would not occur.
The version of srun included with XC V3.0 will teardown correctly by default. This fix was
included in SLURM 0.5.0-pre10. If you are using an older version of srun and you observe
this teardown issue, a workaround is provided with the MPI_ABORT_SRUN_SIGNUM
environment variable. Set this to any signal (for example SIGABRT) and when MPI_Abort
is called, that is the signal it will raise instead of SIGTERM, and srun will correctly
teardown in that case.
• The way to add to the paths searched by man is to add an entry in /etc/man.config. To
add MPI man pages, add the following line to /etc/man.config:
MANPATH /opt/hpmpi/share/man