HP-MPI V2.3 for Linux Release Note

• InfiniBand requires pages to be pinned (locked in memory) for message passing.

This can become a problem when a child process is forked and a pinned page

exists in both the parent's and child's address spaces. Normally a copy-on-write

would occur when one of the processes touches memory on a shared page, and

the virtual to physical mapping would change for that process. In the context of

InfiniBand, such a change in the mapping results in data corruption when an

RDMA sends data to the original physical address.

OFED 1.2 and later (with a fork safety mode enabled) avoids this problem by not

using copy-on-write behavior during a fork for pinned pages. Instead, any access

to these pages by the child process will result in a segmentation violation of the

child, and the parent's mapping remains unchanged so that the parent can continue

running normally with no data corruption.

HP-MPI turns on this option by default when the IBV or uDAPL protocols are

being used. If the fork safety mode is not desired, you can turn it off with the MPI

environment variable MPI_IBV_NO_FORK_SAFE=1.

By setting the environment variable MPI_PAGE_ALIGN_MEM=1, HP-MPI

page-aligns and page-pads libc memory allocation requests that are large enough

to be pinned during MPI message transfer. This results in slightly more memory

being allocated, but reduces the likelihood that a forked process writes to a page

of memory that was also being used for message transfer when a fork call occurred.

• Applications running on Linux systems with kernels older than 2.6.12 might display

the following warning message:

libibverbs: Warning: fork()-safety requested but init failed

This warning message appears because the HP-MPI library enabling the OFED

1.2 fork safety feature is not supported by Linux kernels older than 2.6.12. It does

not affect the application run. To disable HP-MPI fork safety, set the environment

variable MPI_IBV_NO_FORK_SAFE, as in the following example:

% /opt/hpmpi/bin/mpirun -np 4 -prot -e MPI_IBV_NO_FORK_SAFE=1 \

-hostlist nodea,nodeb,nodec,noded /my/dir/hello_world

• Applications using fork() might crash on configurations with InfiniBand using

OFED on kernels older than v2.6.18. You can avoid known problems with fork()

and OFED in any of the following ways:

— Run on XC V3.2.1 or greater, where all known OFED fork() fixes have been

made.

— Run on a system with kernel 2.6.18 or greater.

— Run InfiniBand with non-OFED drivers. (This option is not available on

configurations with ConnectX InfiniBand Host Channel Adapters where OFED

is required.)