HP-MPI Version 2.2.7 for Linux Release Note

• The MPI_FLUSH_FCACHE environment variable is silently ignored if the -cpu_bind option
is not specified. This limitation will be removed in a future release. See the mpienv(1)
manpage for more information.
• HP-MPI uses the $MPI_REMSH command to launch on remote machines. The commands
constructed are of the general form "$MPI_REMSH <host> -n <command>". If a remote
shell command is desired for which the -n in the above syntax is not appropriate, a shell
script such as the following can be constructed and used for the MPI_REMSH command:
#!/bin/sh
host="$1"
shift
shift
ssh $host "$@"
• InfiniBand requires memory to be pinned (locked in memory) for message passing. This
can become a problem when a child process is forked and a pinned page exists in both the
parent's and child's address spaces. Normally a copy-on-write would occur when one of the
processes touches memory on a shared page, and the virtual to physical mapping would
change for that process. In the context of InfiniBand, such a change in the mapping results
in data corruption when an RDMA sends data to the original physical address.
OFED 1.2 and later (with a fork safety mode enabled) avoids this problem by not using
copy-on-write behavior during a fork for pinned pages. Instead, any access to these pages
by the child process will result in a segmentation violation of the child, and the parent's
mapping remains unchanged so the parent is able to continue running normally with no
data corruption.
HP-MPI turns this option on by default when the IBV or uDAPL protocols are being used.
If the fork safety mode is not desired, it can be turned off with the MPI environment variable
MPI_IBV_NO_FORK_SAFE=1.
By setting the environment variable MPI_PAGE_ALIGN_MEM=1, HP-MPI page-aligns and
page-pads libc memory allocation requests which are large enough to be pinned during MPI
message transfer. This can result in slightly more memory being allocated, but reduces the
likelihood that a forked process will write to a page of memory that was also being used for
message transfer when a fork call occurred.
• Applications running on Linux systems with kernels older than 2.6.12 might display the
following warning message:
libibverbs: Warning: fork()-safety requested but init failed
This warning message appears because the HP-MPI library enabling the OFED 1.2 fork
safety feature is not supported by Linux kernels older than 2.6.12. It does not impact the
application run. To disable HP-MPI fork safety, set the environment variable
MPI_IBV_NO_FORK_SAFE, as in the following example:
% /opt/hpmpi/bin/mpirun -np 4 -prot -e MPI_IBV_NO_FORK_SAFE=1 \
-hostlist nodea,nodeb,nodec,noded /my/dir/hello_world
• The initial release of OFED 1.2 contains a bug that causes the memory pinning function to
fail after certain patterns of malloc and free. The symptom, which is visible from HP-MPI,
might be any of several error messages such as:
> prog.x: Rank 0:1: MPI_Get: Unable to pin memory for put/get
This bug has already been fixed in OFED 1.3, but if you are running with the initial release
of OFED 1.2, the only workaround is to set MPI_IBV_NO_FORK_SAFE=1.
• When upgrading to OFED 1.2 from older versions, the installation script might not stop the
previous OFED version before uninstalling it. Therefore, HP recommends stopping the old
OFED stack before upgrading to OFED 1.2. For example:
/etc/init.d/openibd stop
20 Known Problems and Workarounds