HP-MPI V2.3.
© Copyright 1979-2009 © Hewlett-Packard Development Company, L.P Legal Notices The information in this document is subject to change without notice. Hewlett-Packard makes no warranty of any kind with regard to this manual, including, but not limited to, the implied warranties of merchantability and fitness for a particular purpose.
Table of Contents 1 Information About This Release....................................................................................................7 1.1 Announcement................................................................................................................7 1.2 HP-MPI Product Information.........................................................................................7 1.2.1 Platforms Supported...................................................................................
3.13 Expanded Lightweight Instrumentation.....................................................................23 3.14 The api option to MPI_INSTR...................................................................................23 3.15 New mpirun option -xrc.............................................................................................24 4 Known Issues and Workarounds................................................................................................25 4.
6 Licensing Information................................................................................................................37 6.1 Licensing Policy.............................................................................................................37 6.2 License Installation and Testing....................................................................................37 6.2.1 Installing License Files........................................................................................
List of Tables 1-1 1-2 1-3 3-1 3-2 7-1 6 Systems Supported.......................................................................................................8 Interconnect Support for HP-MPI V2.3.1......................................................................8 Directory Structure.......................................................................................................9 MPICH Wrappers...................................................................................................
1 Information About This Release 1.1 Announcement HP-MPI V2.3.1 for Linux is the March 2009 release of HP-MPI, the HP implementation of the Message Passing Interface standard for Linux. HP-MPI V2.3.1 for Linux is supported on HP ProLiant and HP Integrity servers running CentOS 5, Red Hat Enterprise Linux AS 4 and 5, SuSE Linux Enterprise Server 9 and 10 operating systems, and HP XC3000, HP XC4000, and HP XC6000 Clusters. 1.
Table 1-1 Systems Supported Platform Interconnects Itanium 32-bit Myrinet InfiniBand Ethernet1 Itanium 64-bit Myrinet InfiniBand Quadrics Ethernet1 AMD Opteron Myrinet InfiniBand Quadrics Ethernet1 EM64T Myrinet InfiniBand Quadrics Ethernet1 Interconnect information for XC systems is available at http://docs.hp.com/en/linuxhpc.html in the release notes, installation guides, and hardware preparation guides. 1 Ethernet includes 10baseT, 100baseT, GigE, 10GbE, and 10GbE with RNIC. 1.2.
Table 1-2 Interconnect Support for HP-MPI V2.3.1 (continued) Protocol Option Supported Architectures NIC Version Driver Version Quadrics Elan -ELAN Itanium, i386, x86_64 Rev 01 Elan4 TCP/IP -TCP Itanium, i386, x86_64 All cards that support Ethernet Driver, IP IP Contact your interconnect provider to verify driver support. NOTE: Support for the Mellanox VAPI protocol has been discontinued in HP-MPI. However, VAPI functionality has not been removed. VAPI support for HP-MPI releases V2.2.5.
Table 1-3 Directory Structure (continued) Subdirectory Contents lib/linux_amd64 HP-MPI Linux 64-bit libraries for Opteron and EM64T newconfig Configuration files and release notes share/man/man1* manpages for the HP-MPI utilities share/man/man3* manpages for the HP-MPI library doc Release Notes licenses License files 1.2.5 Documentation Changes This release note might be updated periodically. To make sure that you have the latest version of this release note, see the HP website: www.docs.hp.
2 New or Changed Features in V2.3.1 HP-MPI V2.3.1 for Linux includes the following new or changed features. 2.1 LD_PRELOAD Message Using -cpu_bind on HP-MPI V2.3 produced the following message before running the application normally: ERROR: ld.so: object 'libmpi_cpu.so' from LD_PRELOAD cannot be preloaded: ignored. This message does not appear with this release. 2.2 -prot Option In HP-MPI V2.3, the output of the -prot diagnostic option might be incorrect. Using -prot on HP-MPI V2.
3 New or Changed Features in V2.3 The following new or changed features were included in the HP-MPI V2.3 release. 3.1 Options Supported Only on HP Hardware HP-MPI V2.3 for Linux incorporates a number of new features that are supported only on HP hardware. These features are described in detail in the following sections. If these options are used on third-party hardware, an error message appears.
application runs the system check tests. When the system checks are completed, the application exits. This allows the normal application launch procedure to be used during the test, including any job schedulers, wrapper scripts, and local environment settings. By default, the HPMPI_SYSTEM_CHECK API cannot be used if MPI_Init has already been called, and the API will call MPI_Finalize before returning. 3.3 Default Message Size Changed For -ndd When deferred deregistration is used, pinning memory is fast.
NOTE: Do not use MPICH compatibility mode to produce a single executable to run under both MPICH and HP-MPI. For more information on MPICH compatibility, see the HP-MPI User's Guide. 3.5 Support for Large Messages The current MPI standard allows the data transferred using standard API calls to be greater than 2 GB.
3.6 Redundant License Servers This release supports redundant license servers using the FLEXnet Publisher licensing software. Three servers can create a redundant license server network. For a license checkout request to be granted, at least two servers must be running and able to communicate with each other. This avoids a single-license server failure which would prevent new HP-MPI jobs from starting. With three-server redundant licensing, the full number of HP-MPI licenses can be used by a single job.
amount of physical memory that is detected by HP-MPI. If the detection is wrong, the correct amount of physical memory should be set with MPI_PHYSICAL_MEMORY in bytes. This memory is only returned to the operating system for use by other processes after the job resumes and exits. 3.8 Expanded Functionality for -ha HP-MPI high availability mode eliminates teardown when ranks exit abnormally.
NOTE: If a process uses -ha:detect, then all processes it communicates with must also use -ha:detect. Likewise, if a process uses -ha:recover then all processes it communicates with must also use -ha:recover. 3.8.1 Support for High Availability on InfiniBand Verbs You can use the -ha option with the -IBV option. When using -ha, automatic network selection is restricted to TCP and IBV. Be aware that -ha no longer forces the use of TCP.
3.8.4 Using MPI_Comm_disconnect In high availability mode, MPI_Comm_disconnect is collective only across the local group of the calling process. This enables a process group to independently break a connection to the remote group in an intercommunicator without synchronizing with those processes. Unreceived messages on the remote side are buffered and might be received until the remote side calls MPI_Comm_disconnect.
than the largest communicator the rank can join, it returns MPI_COMM_NULL. However, extensive communication failures, such as a failed switch, can make such knowledge unattainable to a rank and result in splitting the communicator. If the communicator returned by rank A contains rank B, then either the communicator return by ranks A and B will be identical or rank B will return MPI_COMM_NULL and any attempt by rank A to communicate with rank B immediately returns MPI_ERR_EXITED.
return from the MPI_Sendrecv_replace() call on commB if their partners are also members of commA and are in the call to MPI_Comm_dup() call on commA. This demonstrates the importance of using care when dealing with multiple communicators. In this example, if the intersection of commA and commB is MPI_COMM_SELF, it is simpler to write an application that does not deadlock during failure. The use of the -ha:recover option is available only on HP hardware.
IMPORTANT: When waiting on a receive request that uses MPI_ANY_SOURCE on an intracommunicator, the request is never considered complete due to rank or interconnect failures because the rank that created the receive request can legally match it. For intercommunicators, after all processes in the remote group are unavailable, the request is considered complete and, the MPI_ERROR field of the MPI_Status() structure indicates MPI_ERR_EXITED.
MPI_STDIO_OUTFILE and MPI_STDIO_ERRFILE. If these environment variables are not set, /dev/null or NUL is used. In addition, these file specifications can include the substrings %%, %h, %p, and %r, which are expanded to %, hostname, process id, and rank number in MPI_COMM_WORLD. The files option causes the stdio options p, r, and I to be ignored. 3.
######################################################## api ## ## api ## Detailed MPI_Reduce routine information ## api ## api ######################################################## api ## api ---------------------------------------------------------------------------------------------- api Rank MPI_Op MPI_Datatype Num Calls Contig Non-Contig Message Sizes Total Bytes api ---------------------------------------------------------------------------------------------- api R: 0 sum
4 Known Issues and Workarounds The following items are known issues and workarounds. 4.1 Running on iWarp Hardware • When running on iWARP hardware, you might see messages similar the following when applications exit: disconnect: ID 0x2b65962b2b10 ret 22 This is a debugging message that prints erroneously from the uDAPL library and can be ignored. The message can be completely suppressed by passing the -e DAPL_DBG_TYPE=0 option to mpirun.
If this happens, add the setting -e MPI_UDAPL_READ=0 to the mpirun command line. This results in HP-MPI only using uDAPL for the regular non-one-sided data transfers, and using a slower TCP-based communication method for one-sided operations. 4.3 Mapping Ranks to a CPU When mapping ranks to a CPU, the ordering of the CPUs relative to the locality domain (ldom)/socket can vary depending on the architecture and operating system.
on host MPI_CPU_AFFINITY on host MPI_CPU_AFFINITY on host MPI_CPU_AFFINITY on host MPI_CPU_AFFINITY on host MPI_CPU_AFFINITY on host MPI_CPU_AFFINITY on host Hello world! I'm ....
4.5 Spawn on Remote Nodes Codes which call either MPI_Comm_spawn() or MPI_Comm_spawn_multiple() might not be able to locate the commands to spawn on remote nodes. To work around this issue, you can either specify an absolute path to the commands to spawn, or users can set the MPI_WORKDIR environment variable to the path of the command to spawn. 4.6 Default Interconnect for -ha Option The –ha option in previous HP-MPI releases forced the use of TCP for communication.
To increase the maximum allowed shared memory segment size, enter the following command as root: # /sbin/sysctl -w kernel.shmmax= To make changes to the kernel.shmmax setting persist across a reboot, add the following line to the /etc/sysctl.conf file: kernel.shmmax= 4.11 Using MPI_FLUSH_FCACHE The MPI_FLUSH_FCACHE environment variable is silently ignored if the -cpu_bind option is not specified. This limitation will be removed in a future release.
during MPI message transfer. This results in slightly more memory being allocated, but reduces the likelihood that a forked process writes to a page of memory that was also being used for message transfer when a fork call occurred. 4.14 Disabling Fork Safety Applications running on Linux systems with kernels earlier than 2.6.
4.18 Increasing the nofile Limit The nofile limit on large Linux clusters needs to be increased in /etc/security/ limits.conf * soft nofile 1024 For larger clusters, HP recommends a setting of at least: • 2048 for clusters of 1900 cores or fewer • 4096 for clusters of 3800 cores or fewer • 8192 for clusters of 7600 cores or fewer 4.19 Using appfiles on HP XC Quadrics To use appfiles on HP XC Quadrics clusters, set MPI_USESRUN=1. The appfile can only differ in host name and rank count. 4.
4.23 Disabling Interval Timer Conflicts Interval timer functionality used by HP-MPI on HP XC systems can conflict with gprof data collection phase requirements. Set the following two environment variables to workaround this issue: % export MPI_FLAGS=s0 % export GMON_OUT_PREFIX=/tmp/app_name In the above example, setting MPI_FLAGS disables the HP-MPI conflicting use of interval timers. See the mpienv(1) manpage for descriptions of MPI_FLAG settings.
4.28 Using -tv To use the -tv option to mpirun, the TotalView binary must be in the user’s PATH, or the TOTALVIEW environment variable must be set to the full path of the TotalView binary. % export TOTALVIEW=/usr/toolworks/totalview/bin/totalview 4.29 Extended Collectives with Lightweight Instrumentation Extended collectives with intercommunicators are not profiled by the HP-MPI lightweight instrumentation mode. 4.
5 Installation Information 5.1 Installation Requirements HP-MPI V2.3.1 for Linux is supported on HP ProLiant and HP Integrity servers running CentOS 5, Red Hat Enterprise Linux AS 4 and 5, SuSE Linux Enterprise Server 9 and 10 operating systems, and HP XC3000, HP XC4000 and HP XC6000 Clusters. You must install the correct HP-MPI product for your system. HP-MPI requires a minimum of 90 MB of disk space in /opt. 5.2 Installation Instructions 1. 2. 3. Run the su command and enter the superuser password.
6 Licensing Information 6.1 Licensing Policy A license is required to use HP-MPI for Linux. You can purchase licenses from the HP software depot at http://www.hp.com/go/softwaredepot, or by contacting your HP representative. Demo licenses for HP-MPI are also available from the HP software depot. No separate HP-MPI license is required on an HP XC system. 6.2 License Installation and Testing HP-MPI V2.3.1 for Linux uses FLEXnet Publisher licensing technology. A license file can be named either as license.
If the license needs to be placed in another location, which is not found by the previous search, you can set the environment variable LM_LICENSE_FILE to explicitly specify the location of the license file. For more information, see http://licensing.hp.com. 6.2.1 Installing License Files A valid license file contains the system hostid and the associated license key. License files can be named either as license.dat or any name with extension of *.lic (for example mpi.lic).
7 Additional Product Information 7.1 Product Documentation The HP-MPI Documentation Kit is an optional product (product number B6281AA) consisting of the following: • • MPI: The Complete Reference (2 volume set) product number B6011-96012 HP-MPI User’s Guide (Eleventh Edition) product number B6060-96024 The HP-MPI User’s Guide and HP-MPI Release Notes are available from the following resources: • • • $MPI_ROOT/doc (After you install the product.) http://docs.hp.com http://www.hp.
7.2 Product Packaging HP-MPI is packaged as an optional software product installed in /opt/hpmpi by default. 7.3 Software Availability in Native Languages There is no information on non-English languages for HP-MPI for Linux systems.
*T1919-90018* Printed in the US
