HP-MPI Version 2.2.5 for Linux Release Note Manufacturing Part Number: T1919-90011B E0906 Release U.S.A. © Copyright 1979-2006 © Hewlett-Packard Development Company, L.P.
Legal Notices The information in this document is subject to change without notice. Hewlett-Packard makes no warranty of any kind with regard to this manual, including, but not limited to, the implied warranties of merchantability and fitness for a particular purpose. Hewlett-Packard shall not be held liable for errors contained herein or direct, indirect, special, incidental, or consequential damages in connection with the furnishing, performance, or use of this material. Warranty.
Intel® and Itanium® are trademarks or registered trademarks of the Intel Corporation or its subsidiaries in the United States and other countries. AMD Opteron and combinations thereof are trademarks of Advanced Micro Devices. FLEXlm® is a registered trademark of Globetrotter Software and Macrovision Corporation. SilverStorm is a trademark of SilverStorm Technologies, Inc. Mellanox is a registered trademark of Myricom, Inc. InfiniBand is a trademark and service mark of the InfiniBand Trade Association.
HP-MPI V2.2.5 for Linux Release Note Announcement HP-MPI V2.2.5 for Linux Release Note Announcement This release note describes the release of HP-MPI Version 2.2.5 for Linux. HP-MPI 2.2.5 is the September 2006 release of HP-MPI, Hewlett-Packard’s fully functional implementation of the Message Passing Interface standard for Linux. HP-MPI 2.2.
HP-MPI V2.2.5 for Linux Release Note What’s in This Version What’s in This Version HP-MPI is a high-performance implementation of the Message Passing Interface standard. HP-MPI complies fully with the MPI-1.2 standard and provides full MPI-2 functionality. HP-MPI provides an application programming interface and software libraries to support parallel, message-passing applications that are efficient, portable, and flexible. HP-MPI is available as both archive and shared libraries.
HP-MPI V2.2.5 for Linux Release Note What’s in This Version Table 1 Systems Supported (Continued) Platform Interconnects HP XC3000 Clusters Myrinet InfiniBand Ethernet* HP XC4000 Clusters Myrinet InfiniBand Elan4 Ethernet* HP XC6000 Clusters InfiniBand Elan4 Ethernet* * Ethernet includes 10baseT, 100baseT, GigE, and 10GbE. NOTE 32-bit apps are not supported on InfiniBand when running on Opteron AMD64 or Intel EM64T machines. InfiniBand Platforms Supported HP-MPI 2.2.
HP-MPI V2.2.5 for Linux Release Note What’s in This Version HP XC3000 HP XC4000 HP XC6000 Contact your InfiniBand provider to verify driver support on any systems not specified above. Compilers HP-MPI strives to be compiler neutral. HP-MPI 2.2.5 for Linux systems was tested with the following compilers: • GNU 3.2, 3.4, 4.1 • glibc 2.2, 2.3, 2.4 • Intel Fortran and C 8.1, 9.0, 9.1 • PathScale 2.0, 2.1, 2.2, 2.3, 2.4, 2.5 • Portland Group 5.2, 6.05, 6.1, 6.
HP-MPI V2.2.5 for Linux Release Note What’s in This Version Table 2 Directory Structure (Continued) Subdirectory Contents share/man/man1* Man pages for the HP-MPI utilities share/man/man3* Man pages for the HP-MPI library doc Release Notes NOTE 32-bit apps are not supported on InfiniBand when running on Opteron AMD64 or Intel EM64T machines.
HP-MPI V2.2.
HP-MPI V2.2.
HP-MPI V2.2.5 for Linux Release Note What’s in This Version hpmpi.conf HP-MPI now supports setting environment variables in an hpmpi.conf file. These variables are read by mpirun and exported globally, as if they had been included on the mpirun command line as "-e VAR=VAL" settings. The hpmpi.conf file search is performed in three places and each one is parsed, which allows the last one parsed to overwrite values set by the previous files. The three locations are: • $MPI_ROOT/etc/hpmpi.conf • /etc/hpmpi.
HP-MPI V2.2.5 for Linux Release Note What’s in This Version An abbreviated example would be: MPI_IC_SUFFIXES="VAPI_MAIN:VAPI_CISCO" MPI_ICLIB_VAPI__VAPI_MAIN="libmtl_common.so:libmpga.so:libmosal.so:libvapi.so "MPI_ICMOD_VAPI__VAPI_MAIN ="^mod_vapi :||:^mod_vip " MPI_ICLIB_VAPI__VAPI_CISCO="libpthread.so:libmosal.so:libvapi.
HP-MPI V2.2.5 for Linux Release Note What’s in This Version OpenFabrics support HP-MPI supports OpenFabrics using Verbs API. HP-MPI selects the fast interconnect by default, but the user can force OpenFabrics use with the mpirun option -IBV, or just recommend OpenFabrics use with -ibv. HP-MPI does not use the Connection Manager (CM) library. In order to use OpenFabrics on Linux, the memory size for locking must be specified. It is controlled by the /etc/security/limits.conf file.
HP-MPI V2.2.5 for Linux Release Note What’s in This Version New mpirun option -netaddr -netaddr is similar to -subnet, but allows finer control of the selection process for TCP/IP connections. MPI has two main sets of connections: those between ranks and/or daemons where all the real message traffic occurs, and connections between mpirun and the daemons, which are necessary, but little traffic occurs here.
HP-MPI V2.2.5 for Linux Release Note What’s in This Version setenv MPI_REMSH rsh export MPI_REMSH=rsh C++ bindings HP-MPI 2.2.5 supports C++ bindings as described in the MPI-2 Standard. If compiling and linking with the mpiCC command, no additional work is needed to include and use the bindings. You can include either mpi.h or mpiCC.h in your C++ source files. The bindings provided by HP-MPI are an interface class, calling the equivalent C bindings.
HP-MPI V2.2.5 for Linux Release Note What’s in This Version % mkdir test ; cd test % cp $MPI_ROOT/help/sort.C . % $MPI_ROOT/bin/mpiCC HPMPI_CC=pgCC sortC -mpiCClib ../libmpiCC.a sort.C: % $MPI_ROOT/bin/mpirun -np 2 ./a.out Rank 0 -980 -980 . . . 965 965 % Myrinet MX support HP-MPI supports the Myrinet Express (MX) protocol. The selection is done via mpirun command line options and environment variables. The mpirun command line options are: • -mx • -MX -mx is a request to use MX on Myrinet.
HP-MPI V2.2.5 for Linux Release Note What’s in This Version If the above option is applied so that the ptmalloc contained in HP-MPI is not used, then there is a risk of MPI not being informed of when memory is returned to the system. This can be alleviated with the setting MPI_USE_MALLOPT_SBRK_PROTECTION at runtime, which essentially results in the libc malloc implementation not returning memory to the system.
HP-MPI V2.2.5 for Linux Release Note What’s in This Version Provides an easy way to modify the arguments contained in an appfile by supplying a list of space-separated arguments that mpirun should ignore. % setenv MPI_USEPRUN_IGNORE_ARGS
HP-MPI V2.2.5 for Linux Release Note What’s in This Version Each of the above arguments may be combined with an optional rank count '-np N' which, if present, will select exactly N ranks. The ranks are selected by traversing the list|file|env-vars of hosts in order with wraparound if the requested number of ranks exceeds the number of hosts. If the number of desired ranks is not specified, then the list|file|env-vars are traversed once to produce a complete list of hosts.
HP-MPI V2.2.5 for Linux Release Note What’s in This Version -hostfile hfile # 2 ranks on n1 and 3 ranks on n3. -lsb_hosts uses the environment variable established by the LSF bsub command to launch jobs across the list of hosts, using the list of hosts within $LSB_HOSTS. -lsb_mcpu_hosts uses the environment variable established by the LSF bsub command to launch jobs across the list of hosts, using the list of hosts within $LSB_MCPU_HOSTS.
HP-MPI V2.2.5 for Linux Release Note What’s in This Version For NUMA-based systems, the following options are also available: ldom Schedule ranks on ldoms according to packed rank id. cyclic Cyclic dist on each ldom according to packed rank id. block Block dist on each ldom according to packed rank id. rr Same as cyclic, but consider ldom load average. fill Same as block, but consider ldom load average. packed Bind all ranks to same ldom as lowest rank. slurm slurm binding.
HP-MPI V2.2.5 for Linux Release Note What’s in This Version • MPI_CPU_AFFINITY is an alternative method to using -cpu_bind on the command line for specifying binding strategy. The possible settings are LL, RANK, MAP_CPU, MASK_CPU, LDOM, CYCLIC, BLOCK, RR, FILL, PACKED, SLURM, AND MAP_LDOM. • MPI_BIND_MAP allows specification of the integer CPU numbers, ldom numbers, or CPU masks. These are a list of integers separated by commas (,).
HP-MPI V2.2.5 for Linux Release Note What’s in This Version If prun/srun is used for launching the application, then mpirun sends the signal to the responsible launcher and relies on the signal propagation capabilities of the launcher to ensure that the signal is propagated to the ranks. When using prun, SIGTTIN is also intercepted by mpirun, but is not propagated. When using an appfile, HP-MPI propagates these signals to remote HP-MPI daemons (mpid) and local ranks.
HP-MPI V2.2.5 for Linux Release Note What’s in This Version Table 3 Feature Scalability Affected Interconnect/Protocol Scalability Impact All Forces use of pairwise socket connections between all mpid’s (typically one mpid per machine) one-sided shared lock/unlock All except VAPI and IBV Only VAPI and IBV provide low-level calls to efficiently implement shared lock/unlock. All other interconnects require mpid’s to satisfy this feature.
HP-MPI V2.2.5 for Linux Release Note What’s in This Version Examples: To use the 2nd IB card: % setenv MPI_IB_CARD_ORDER 1 To use the 2nd port of the 2nd card: % setenv MPI_IB_CARD_ORDER 1:1 To use the 1st IB card: % setenv MPI_IB_CARD_ORDER 0 To assign ranks to multiple cards: % setenv MPI_IB_CARD_ORDER 0,1,2 will assign the local ranks per node in order to each card. % mpirun -hostlist "host0 4 host1 4" creates ranks 0-3 on host0 and ranks 4-7 on host1.
HP-MPI V2.2.5 for Linux Release Note What’s in This Version a Time-out value for IBV retry if no response from target. Minimum is 1. Maximum is 31. Default is 18. b The retry count after time-out before error is issued. Minimum is 0. Maximum is 7. Default is 7. c The minimum Receiver Not Ready (RNR) NAK timer. After this time, an RNR NAK is sent back to the sender. Values: 1(0.01ms) - 31(491.52ms); 0(655.36ms). The default is 24(40.96ms). d RNR retry count before error is issued. Minimum is 0.
HP-MPI V2.2.5 for Linux Release Note What’s in This Version b Middle message protocol threshold. If the message length is less than or equal to b, consecutive short messages are used to send the whole message. By default, we set b to 16384 bytes, the same as a, to effectively turn off middle message protocol. c Long message fragment size.
HP-MPI V2.2.5 for Linux Release Note What’s in This Version Note that these settings are not available when running under srun or prun. However, similar information can be gathered from the variables set by those systems; such as SLURM_NPROCS and SLURM_PROCID. Shared Memory Sizing Environment Variables MPI_RANKMEMSIZE=d Where d is the total bytes of shared memory of the rank. Specifies the shared memory for each rank. 12.5% is used as generic. 87.5% is used as fragments.
HP-MPI V2.2.5 for Linux Release Note What’s in This Version % $MPI_ROOT/bin/mpicc -o hello_world $MPI_ROOT/help/hello_world.c • Create a file "appfile" as: -h n01 -np 2 /path/to/hello_world -h n02 -np 2 /path/to/hello_world • Run the hello_world executable file: % $MPI_ROOT/bin/mpirun -f appfile By default, mpirun will ssh to the remote machines n01 and n02. • Analyze hello_world output. HP-MPI prints the output from running the hello_world executable in non-deterministic order.
HP-MPI V2.2.5 for Linux Release Note What’s in This Version n00 n01 n02 n03 rank1 rank2 rank3 rank4 % $MPI_ROOT/bin/mpirun [-prun|-srun] -n4 ./a.out will run a.out with 4 ranks, 2 ranks per node, ranks are block allocated. Two nodes used. n00 n00 n01 n01 rank1 rank2 rank3 rank4 % $MPI_ROOT/bin/mpirun [-prun|-srun] -n6 -O -N2 -m block ./a.out will run a.out with 6 ranks (oversubscribed), 3 ranks per node, ranks are block allocated. Two nodes used.
HP-MPI V2.2.5 for Linux Release Note What’s in This Version 5. Interconnect Selection Examples Example 1 Interconnect Selection % echo MPI_IC_ORDER elan:TCP % export MPIRUN_SYSTEM_OPTIONS="-netaddr 192.168.1.1" % export MPIRUN_OPTIONS="-prot" % $MPI_ROOT/bin/mpirun -prun -n4 ./a.out The command line for the above will appear to mpirun as $MPI_ROOT/bin/mpirun -netaddr 192.168.1.1 -prot -prun -n4 ./a.out and the interconnect decision will look for the presence of Elan and use it if found.
HP-MPI V2.2.5 for Linux Release Note What’s in This Version [user@opte10 user]$ bsub -I -n3 -ext "SLURM[nodes=3]" $MPI_ROOT/bin/mpirun -prot -srun ./a.out Job <59304> is submitted to default queue . <> <
HP-MPI V2.2.5 for Linux Release Note What’s in This Version host | 0 1 2 ======|================ 0 : SHM TCP TCP 1 : TCP SHM TCP 2 : TCP TCP SHM Hello world! I'm 0 of 3 on opte2 Hello world! I'm 1 of 3 on opte3 Hello world! I'm 2 of 3 on opte4 • Elan interface [user@opte10 user]$ /sbin/ifconfig eip0 eip0 Link encap:Ethernet HWaddr 00:00:00:00:00:0F inet addr:172.22.0.10 Bcast:172.22.255.255 Mask:255.255.0.
HP-MPI V2.2.5 for Linux Release Note What’s in This Version % bsub -I -n3 -ext "SLURM[nodes=3]" $MPI_ROOT/bin/mpirun -prot -TCP -netaddr 172.22.0.10 -srun ./a.out The -ext option can also be used to specifically request a node. The command line would look something like the following: % bsub -I -n2 -ext "SLURM[nodelist=n10]" mpirun -srun ./hello_world Job <1883> is submitted to default queue . <> <
HP-MPI V2.2.5 for Linux Release Note Known Problems and Workarounds Known Problems and Workarounds • The nofile limit on large linux clusters needs to be increased in /etc/security/limits.conf % soft nofile 1024 For larger clusters, we recommend — 2048 for clusters of 1900 cores or less — 4096 for clusters of 3800 cores or less — 8192 for clusters of 7600 cores or less — Etc.
HP-MPI V2.2.5 for Linux Release Note Known Problems and Workarounds When a background HP-MPI job is run and the shell is terminated, the job may or may not continue depending on the actual shell used. For /bin/bash, the job is killed. For bin/sh and /bin/ksh, the job continues. If 'nohup' is used when launching the job, only background ksh jobs can continue. This behavior may vary depending on your system. • MPI_IC_ORDER must provide the same definition on every node in a cluster to be effective.
HP-MPI V2.2.5 for Linux Release Note Known Problems and Workarounds this, the Mellanox driver does not use copy-on-write behavior during a fork for pinned pages. Instead, access to these pages by the child process will result in a segmentation violation. If you believe that your application is experiencing a problem due to this limitation, HP-MPI provides a feature that may avoid the scenario from occurring.
HP-MPI V2.2.5 for Linux Release Note Known Problems and Workarounds • Calling MPI from Fortran 90 or C++ programs HP-MPI complies with the 1.2 version of the MPI standard, which defines bindings for Fortran 77 and C, but not Fortran 90 or C++. Some features of Fortran 90 may interact with MPI non-blocking semantics to produce unexpected results. Consult the HP-MPI User’s Guide for details. C++ applications should be able to use the existing C binding for MPI with no problems.
HP-MPI V2.2.5 for Linux Release Note Installation Requirements Installation Requirements HP-MPI 2.2.5 for Linux is supported on HP ProLiant and HP Integrity servers running Red Hat Enterprise Linux AS 3 and 4, or SuSE Linux Enterprise Server 9 and 10 operating systems; as well as HP XC3000, HP XC4000 and HP XC6000 Clusters. You must install the correct HP-MPI product for your system. HP-MPI requires a minimum of 90 MB of disk space in /opt.
HP-MPI V2.2.5 for Linux Release Note Installation Instructions Installation Instructions 1. Become root by running the su command and entering the superuser password. 2. Place the downloaded file into the /tmp directory (or another suitable directory). 3. Run the rpm command to install the package. % rpm -Uvh /tmp/hpmpi-2.02.05-??.[i386|ia64|x86_64].rpm Or use the following command to overwrite a previous version: % rpm -ivh --force /tmp/hpmpi-2.02.05-??.[i386|ia64|x86_64].
HP-MPI V2.2.5 for Linux Release Note Licensing Policy Licensing Policy HP-MPI 2.2.5 for Linux uses FLEXlm® licensing technology. A license is required to use HP-MPI for Linux. Licenses can be purchased from HP’s software depot at http://www.hp.com/go/softwaredepot, or contact your HP representative. Demo licenses for HP-MPI are also available from HP’s software depot. If you’re running on an HP XC system, no license is required at runtime. Licensing HP-MPI 2.2.5 for Linux uses FLEXlm licensing technology.
HP-MPI V2.2.5 for Linux Release Note Licensing Policy If the license needs to be placed in another location which would not be found by the above search, the user may set the environment variable LM_LICENSE_FILE to explicitly specify the location of the license file. For more information, see http://licensing.hp.com. Installing License Files A valid license file contains the system hostid and the associated license key. License files can be named either as license.dat or any name with extension of *.
HP-MPI V2.2.5 for Linux Release Note Licensing Policy SERVER myserver 0014c2c1f34a DAEMON HPQ INCREMENT HP-MPI HPQ 1.0 permanent 16 BE468B74B592 \ NOTICE="License Number = AAAABBBB2222" SIGN=9AB4034C6CB2 The result is a valid license for 24 ranks.
HP-MPI V2.2.5 for Linux Release Note Product Documentation Product Documentation The HP-MPI Documentation Kit is an optional product (product number B6281AA) consisting of the following hardcopy books: • MPI: The Complete Reference (2 volume set) Product number B6011-96012 • HP-MPI User’s Guide (Ninth Edition) Product number B6060-96018 The HP-MPI User’s Guide and HP-MPI Release Notices are available online • in /opt/hpmpi/doc after you install the product. • at http://docs.hp.com.
HP-MPI V2.2.5 for Linux Release Note Product Documentation Table 4 Category Man Page Categories Man Pages Description General MPI.1 Describes the general features of HP-MPI Compilation mpicc.1 mpiCC.1 mpif77.1 mpif90.1 Describes the available compilation utilities Runtime mpiclean.1 mpidebug.1 mpienv.1 mpiexec.1 mpijob.1 mpimtsafe.1 mpirun.1 mpirun.all.1 mpistdio.1 autodbl.1 Describes runtime utilities, environment variables, debugging, thread-safe and diagnostic libraries.
HP-MPI V2.2.5 for Linux Release Note Product Packaging Product Packaging HP-MPI is packaged as an optional software product installed in /opt/hpmpi.
HP-MPI V2.2.5 for Linux Release Note Software Availability in Native Languages Software Availability in Native Languages There is no information on non-English languages for HP-MPI for Linux systems.
