HP-MPI Version 2.2.7 for Linux Release Note

2 Known Problems and Workarounds

• OFED 1.3 documents minimum supported firmware revisions for a variety of InfiniBand

adapters. If unsupported firmware is used, HP-MPI might experience issues with abnormal

application teardown.

• Codes which call either MPI_Comm_spawn() or MPI_Comm_spawn_multiple() may not

be able to locate the commands to spawn on remote nodes. To work around this issue, users

can either specify an absolute path to the commands to spawn, or users can set the

MPI_WORKDIR environment variable to the path of the command to spawn.

• When the -ha flag is supplied to mpirun, teardown will be eliminated when a rank exits

abnormally. In these cases, when the application completes, some of the mpid daemons

might continue running after all the remaining ranks have exited. When run interactively,

this can appear as a hang on the tty. This is a known defect that will be addressed in an

upcoming release.

To work around this issue in 2.2.7, consider setting a run time limit through the job launching

mechanism such as the LSF bsub -W argument or SLURMs -t timelimit argument to cleanup

all job processes (including mpids) after a certain period of time. Alternatively, a SIGINT

(Control-C) can be sent to the mpirun process after all remaining ranks have finished. This

will propagate to all remaining mpids, essentially tearing down the remaining infrastructure.

• The –ha option in previous HP-MPI releases forced the use of TCP for communication. As

of this release, both IBV and TCP are possible network selections when using –ha. If no

forced selection criteria (for example, –TCP, –IBV, or equivalent MPI_IC_ORDER setting)

is specified by the user, then IBV is selected where it is available. Otherwise, TCP is used.

• In order to support the -dd (deferred deregistration) option, HP-MPI must intercept calls

to glibc routines that allocate and free memory. The compiler wrapper scripts included with

HP-MPI attempts to link MPI applications in such a way as to make this possible. If you

choose not to link your application with the provided compiler wrappers, you must either

ensure that libmpi.so precedes libc.so on the linker command line or specify "-e

LD_PRELOAD=%LD_PRELOAD:libmpi.so" on the mpirun command line.

• Whether mpirun is invoked on a host where at least one MPI process is running or on a

host remote from all MPI processes, HP-MPI writes the instrumentation output file

prefix.instr to the working directory on the host that is running rank 0 (when

instrumentation for multi-host runs is enabled). When using -ha, the output file is located

on the host that is running the lowest existing rank number at the time the instrumentation

data is gathered during MPI_Finalize().

• Applications that use the ScaLAPACK library must use the HP-MPI mpich compatibility

mode. When the application is built, mpicc.mpich or mpif77.mpich or mpif90.mpich

must be used. At runtime, mpirun.mpich must be used to launch the application.

• HP-MPI uses shared memory for communications between processes on the same node and,

under some circumstances, might allocate a shared memory segment that is larger than the

operating system allows. The most common issue a user might experience when this happens

is an error message like:

Cannot create shared memory segment of <size> bytes.

To increase the maximum allowed shared memory segment size, use the command:

# /sbin/sysctl -w kernel.shmmax=<size in bytes>

To make changes to the kernel.shmmax setting persist across a reboot, add the line:

kernel.shmmax=<size in bytes>

to the /etc/sysctl.conf file.