HP-MPI Version 2.3.1 for Linux Release Note
Table Of Contents
- HP-MPI V2.3.1 for Linux Release Note
- Table of Contents
- 1 Information About This Release
- 2 New or Changed Features in V2.3.1
- 3 New or Changed Features in V2.3
- 3.1 Options Supported Only on HP Hardware
- 3.2 System Check
- 3.3 Default Message Size Changed For -ndd
- 3.4 MPICH2 Compatibility
- 3.5 Support for Large Messages
- 3.6 Redundant License Servers
- 3.7 License Release/Regain on Suspend/Resume
- 3.8 Expanded Functionality for -ha
- 3.8.1 Support for High Availability on InfiniBand Verbs
- 3.8.2 Highly Available Infrastructure (-ha:infra)
- 3.8.3 Using MPI_Comm_connect and MPI_Comm_accept
- 3.8.4 Using MPI_Comm_disconnect
- 3.8.5 Instrumentation and High Availability Mode
- 3.8.6 Failure Recover (-ha:recover)
- 3.8.7 Network High Availability (-ha:net)
- 3.8.8 Failure Detection (-ha:detect)
- 3.8.9 Clarification of the Functionality of Completion Routines in High Availability Mode
- 3.9 Enhanced InfiniBand Support for Dynamic Processes
- 3.10 Singleton Launching
- 3.11 Using the -stdio=files Option
- 3.12 Using the -stdio=none Option
- 3.13 Expanded Lightweight Instrumentation
- 3.14 The api option to MPI_INSTR
- 3.15 New mpirun option -xrc
- 4 Known Issues and Workarounds
- 4.1 Running on iWarp Hardware
- 4.2 Running with Chelsio uDAPL
- 4.3 Mapping Ranks to a CPU
- 4.4 OFED Firmware
- 4.5 Spawn on Remote Nodes
- 4.6 Default Interconnect for -ha Option
- 4.7 Linking Without Compiler Wrappers
- 4.8 Locating the Instrumentation Output File
- 4.9 Using the ScaLAPACK Library
- 4.10 Increasing Shared Memory Segment Size
- 4.11 Using MPI_FLUSH_FCACHE
- 4.12 Using MPI_REMSH
- 4.13 Increasing Pinned Memory
- 4.14 Disabling Fork Safety
- 4.15 Using Fork with OFED
- 4.16 Memory Pinning with OFED 1.2
- 4.17 Upgrading to OFED 1.2
- 4.18 Increasing the nofile Limit
- 4.19 Using appfiles on HP XC Quadrics
- 4.20 Using MPI_Bcast on Quadrics
- 4.21 MPI_Issend Call Limitation on Myrinet MX
- 4.22 Terminating Shells
- 4.23 Disabling Interval Timer Conflicts
- 4.24 libpthread Dependency
- 4.25 Fortran Calls Wrappers
- 4.26 Bindings for C++ and Fortran 90
- 4.27 Using HP Caliper
- 4.28 Using -tv
- 4.29 Extended Collectives with Lightweight Instrumentation
- 4.30 Using -ha with Diagnostic Library
- 4.31 Using MPICH with Diagnostic Library
- 4.32 Using -ha with MPICH
- 4.33 Using MPI-2 with Diagnostic Library
- 4.34 Quadrics Memory Leak
- 5 Installation Information
- 6 Licensing Information
- 7 Additional Product Information
4 Known Issues and Workarounds
The following items are known issues and workarounds.
4.1 Running on iWarp Hardware
• When running on iWARP hardware, you might see messages similar the following
when applications exit:
disconnect: ID 0x2b65962b2b10 ret 22
This is a debugging message that prints erroneously from the uDAPL library and
can be ignored. The message can be completely suppressed by passing the -e
DAPL_DBG_TYPE=0 option to mpirun. Alternatively, you can set
DAPL_DBG_TYPE=0 in the $MPI_ROOT/etc/hpmpi.conf file to avoid having
to pass the option on the mpirun command line.
• Users might see the following error during launch of HP-MPI applications on
Chelsio iWARP hardware:
Rank 0:0: MPI_Init: dat_evd_wait()1 unexpected event number 16392
Rank 0:0: MPI_Init: MPI BUG: Processes cannot connect to rdma device
MPI Application rank 0 exited before MPI_Finalize() with status 1
To prevent these errors, Chelsio recommends passing the peer2peer=1 parameter
to the iw_cxgb3 kernel module. This is accomplished by running the following
commands as root on all nodes:
# echo "1" > /sys/module/iw_cxgb3/parameters/peer2peer
# echo "options iw_cxgb3 peer2peer=1" >> /etc/modprobe.conf
The second command is optional and makes the setting persist across a system
reboot.
• Users of iWARP hardware might see errors similar to the following:
dapl async_event QP (0x2b27fdc10d30) ERR 1 dapl_evd_qp_async_error_callback() IB async QP err
- ctx=0x2b27fdc10d30
Previous versions of HP-MPI required passing -e MPI_UDAPL_MSG1=1 on some
iWARP hardware. As of HP-MPI V2.3, no iWARP implementations are known to
require this setting, and you must remove it from all scripts unless otherwise
instructed.
4.2 Running with Chelsio uDAPL
At the time of this release, Chelsio uDAPL has a limitation that one-sided operations
are only implemented for off-host transfers, not data transfers, between ranks on the
same host. On a Chelsio system, the symptom for this problem resembles the following:
dapl_cma_connect: rdma_connect ERR -1 Function not implemented
4.1 Running on iWarp Hardware 25