HP-MPI Version 2.2 for Linux Release Note Manufacturing Part Number: T1919-90009 E1205 Release U.S.A. © Copyright 1979-2005 © Hewlett-Packard Development Company, L.P.
Legal Notices The information in this document is subject to change without notice. Hewlett-Packard makes no warranty of any kind with regard to this manual, including, but not limited to, the implied warranties of merchantability and fitness for a particular purpose. Hewlett-Packard shall not be held liable for errors contained herein or direct, indirect, special, incidental, or consequential damages in connection with the furnishing, performance, or use of this material. Warranty.
Intel® and Itanium® are trademarks or registered trademarks of the Intel Corporation or its subsidiaries in the United States and other countries. AMD Opteron and combinations thereof are trademarks of Advanced Micro Devices. FLEXlm® is a registered trademark of Globetrotter Software and Macrovision Corporation. SilverStorm is a trademark of SilverStorm Technologies, Inc. Mellanox is a trademark of Mellanox Technologies, Inc. Myrinet® is a registered trademark of Myricom, Inc.
HP-MPI V2.2 for Linux Release Note Announcement HP-MPI V2.2 for Linux Release Note Announcement This release note describes the release of HP-MPI Version 2.2 for Linux. HP-MPI 2.2 is the December 2005 release of HP-MPI, Hewlett-Packard’s fully functional implementation of the Message Passing Interface standard for Linux. HP-MPI 2.2 for Linux is supported on servers and workstations running Red Hat Enterprise Linux AS 3 and 4, or SuSE Linux Enterprise Server 9.
HP-MPI V2.2 for Linux Release Note What’s in This Version What’s in This Version HP-MPI is a high-performance implementation of the Message Passing Interface standard. HP-MPI complies fully with the MPI-1.2 standard and provides full MPI-2 functionality. HP-MPI provides an application programming interface and software libraries to support parallel, message-passing applications that are efficient, portable, and flexible. HP-MPI is available as both archive and shared libraries.
HP-MPI V2.2 for Linux Release Note What’s in This Version Table 1 Systems Supported (Continued) Platform Interconnects XC3000 Clusters Myrinet InfiniBand Ethernet* XC4000 Clusters Myrinet InfiniBand Elan4 Ethernet* XC6000 Clusters InfiniBand Elan3/Elan4 Ethernet* * Ethernet includes 10baseT, 100baseT, GigE, and 10GbE. NOTE This is the last HP-MPI release that will support the IT-API protocol on Linux. NOTE HP-MPI strives to be compiler neutral. HP-MPI 2.
HP-MPI V2.2 for Linux Release Note What’s in This Version rx4640 rx7620 rx8620 XC3000 XC4000 XC6000 Contact your InfiniBand provider to verify support on any systems not specified above. Directory Structure All HP-MPI for Linux files are stored in the /opt/hpmpi directory. The directory structure is organized as described in Table 2. If you choose to move the HP-MPI installation directory from its default location in /opt/hpmpi, set the MPI_ROOT environment variable to point to the new location.
HP-MPI V2.2 for Linux Release Note What’s in This Version Benefits and Features MPI has become the industry standard for distributed technical applications and is widely supported on most technical computing platforms. HP-MPI offers several benefits. • Applications developed using HP-MPI port easily to other platforms, even those with a different architecture.
HP-MPI V2.2 for Linux Release Note What’s in This Version HP-MPI 2.2 for Linux includes the following new environment variables: • MPI_RDMA_MSGSIZE • MPI_RDMA_NENVELOPE • MPI_RDMA_NSRQRECV • MPI_RDMA_NFRAGMENT • MPI_RDMA_NONESIDED • MPI_MAX_REMSH • MPI_VAPI_QPPARAMS • MPI_RANKMEMSIZE • MPI_ICLIB_MX Description of Features The following section provides brief descriptions of the new features included in this release.
HP-MPI V2.2 for Linux Release Note What’s in This Version To use rsh instead, the following script needs to be run as root on each node in the cluster: % /opt/hpmpi/etc/mpi.remsh.default Or, to use rsh, use the alternative method of manually populating the files /etc/profile.d/hpmpi.csh and /etc/profile.d/hpmpi.sh with the following settings respectively: setenv MPI_REMSH rsh export MPI_REMSH=rsh C++ bindings HP-MPI 2.2 now supports C++ bindings as described in the MPI-2 Standard.
HP-MPI V2.2 for Linux Release Note What’s in This Version % make g++ -c intercepts.cc -I/opt/hpmpi/include -DHPMP_BUILD_CXXBINDING g++ -c mpicxx.cc -I/opt/hpmpi/include -DHPMP_BUILD_CXXBINDING ar rcs libmpiCC.a intercepts.o mpicxx.o % 4. Link your application with the -mpiCClib option, using the library you created. % mpiCC sort.C -mpiCClib /scratch/libmpicc/libmpiCC.a Myrinet MX support HP-MPI supports the Myrinet Express (MX) protocol.
HP-MPI V2.2 for Linux Release Note What’s in This Version If the above option is applied so that the ptmalloc contained in HP-MPI is not used, then there is a risk of MPI not being informed of when memory is returned to the system. This can be alleviated with the setting MPI_USE_MALLOPT_SBRK_PROTECTION at runtime, which essentially results in the libc malloc implementation not returning memory to the system.
HP-MPI V2.2 for Linux Release Note What’s in This Version % setenv MPI_PRUNOPTIONS
HP-MPI V2.2 for Linux Release Note What’s in This Version Each of the above arguments may be combined with an optional rank count '-np N' which, if present, will select exactly N ranks. The ranks are selected by traversing the list|file|env-vars of hosts in order with wraparound if the requested number of ranks exceeds the number of hosts. If the number of desired ranks is not specified, then the list|file|env-vars are traversed once to produce a complete list of hosts.
HP-MPI V2.2 for Linux Release Note What’s in This Version New mpirun options The default setting varies depending on the interconnect and rank count. See “New Environment Variables” on page 20 for more details on default selections. -srq specifies use of the shared receiving queue protocol when Mellanox VAPI or uDAPL V1.1 or V1.2 interfaces are used. This protocol uses less pre-pinned memory for short message transfer. -rdma specifies use of envelope pairs for short message transfer.
HP-MPI V2.2 for Linux Release Note What’s in This Version fill Same as block, but consider ldom load average. packed Bind all ranks to same ldom as lowest rank. slurm slurm binding. ll Bind each rank to ldoms each is currently running on. map_ldom Schedule ranks on ldoms in cycle through MAP variable. Three -cpu_bind options require the specification of a map/mask description. This allows for very explicit binding of ranks to processors. The three options are map_ldom, map_cpu, and mask_cpu.
HP-MPI V2.2 for Linux Release Note What’s in This Version Signal Propagation HP-MPI 2.2 supports the propagation of signals from mpirun to application ranks.
HP-MPI V2.2 for Linux Release Note What’s in This Version In general, HP-MPI relies on applications terminating when they are sent SIGTERM. Applications which intercept SIGTERM may not terminate properly. Fast 1sided lock/unlock under VAPI When using the VAPI protocol, HP-MPI 2.2 is able to use low-level hardware atomic operations to provide a high performance and scalable 1sided lock/unlock implementation.
HP-MPI V2.2 for Linux Release Note What’s in This Version New Environment Variables The following section provides brief descriptions of the new environment variables included in this release. When using InfiniBand, a certain amount of memory is pinned. This means it is locked to physical memory and cannot be paged out. The amount of pre-pinned memory HP-MPI uses can be adjusted using several tunables. These are described below.
HP-MPI V2.2 for Linux Release Note What’s in This Version In the above example, the number of receiving buffers are calculated as 8 times the number of offhost connections. If this number is greater than 2048, then 2048 is used as the maximum number. MPI_RDMA_NFRAGMENT=N Specifies the number of long message fragments that can be concurrently pinned down for each connection. This is only for one-direction traffic. The default is 3.
HP-MPI V2.2 for Linux Release Note What’s in This Version c The minimum Receiver Not Ready (RNR) NAK timer. After this time, an RNR NAK is sent back to the sender. Values: 1(0.01ms) - 31(491.52ms); 0(655.36ms). The default is 24(40.96ms). d RNR retry count before error is issued. Minimum is 0. Maximum is 7. Default is 7 (infinite). MPI_RANKMEMSIZE=d Specifies the shared memory for each rank. 12.5% is used as generic. 87.5% is used as fragments. The only way to change this ratio is to use MPI_SHMEMCNTL.
HP-MPI V2.2 for Linux Release Note What’s in This Version • Add $MPI_ROOT/share/man to MANPATH. NOTE HP-MPI must be installed in the same directory on every execution host. 2. The following is an example of basic compilation and run steps to execute hello_world.c on a cluster with 4-way parallelism given the ability to rsh/remsh/ssh between hosts. To build and run hello_world.c on a cluster using an appfile: • Change to a writable directory that is visible from all hosts on which the job will run.
HP-MPI V2.2 for Linux Release Note What’s in This Version For more information on prun usage: % man prun For more information on srun usage: % man srun 4. The following is an example that assumes the system has the Quadrics Elan4 interconnect, and the system is a collection of 2-CPU nodes. (For srun, SLURM is configured to use the Elan4): % $MPI_ROOT/bin/mpirun [-prun|-srun] -N4 ./a.out will run a.out with 4 ranks, one per node, ranks are cyclically allocated.
HP-MPI V2.2 for Linux Release Note What’s in This Version Other forms of usage include allocating the nodes you wish to use, which creates a subshell. Then jobsteps can be launched within that subshell until the subshell is exited. % $MPI_ROOT/bin/mpirun [-prun|-srun] -A -N6 This allocates 6 nodes and creates a subshell. % $MPI_ROOT/bin/mpirun [-prun|-srun] -n4 -m block ./a.out This allocates 4 ranks on 4 nodes cyclically. Note that we asked for block. n00 n01 n02 n03 rank1 rank2 rank3 rank4 5.
HP-MPI V2.2 for Linux Release Note What’s in This Version % export MPI_IC_ORDER="elan:TCP" % export MPIRUN_SYSTEM_OPTIONS=" " % $MPI_ROOT/bin/mpirun -prot -TCP -prun -n4 ./a.out Example 4 • Protocol Maps This runs on ELAN [user@opte10 user]$ bsub -I -n3 -ext "SLURM[nodes=3]" $MPI_ROOT/bin/mpirun -prot -srun ./a.out Job <59304> is submitted to default queue . <> <
HP-MPI V2.2 for Linux Release Note What’s in This Version [user@opte10 user]$ bsub -I -n3 -ext "SLURM[nodes=3]" $MPI_ROOT/bin/mpirun -prot -TCP -subnet 172.22.0.10 -srun ./a.out Job <59307> is submitted to default queue . <> <> Host 0 -- ip 172.22.0.2 -- ranks 0 Host 1 -- ip 172.22.0.3 -- ranks 1 Host 2 -- ip 172.22.0.
HP-MPI V2.2 for Linux Release Note What’s in This Version LSF creates an allocation of 12 processors and srun uses 1 CPU per node (6 nodes). Here, we assume 2 CPUs per node. % bsub -I -n12 $MPI_ROOT/bin/mpirun -srun -n6 -N6./a.out LSF jobs can be submitted without the -I (interactive) option. An alternative mechanism for achieving the one rank per node which uses the -ext option to LSF: % bsub -I -n3 -ext "SLURM[nodes=3]" $MPI_ROOT/bin/mpirun -prot -TCP -subnet 172.22.0.10 -srun ./a.
HP-MPI V2.2 for Linux Release Note Known Problems and Workarounds Known Problems and Workarounds • In order to use appfiles on XC ELAN clusters, MPI_USESRUN must be set to 1 and the appfile must be homogenous. • The SilverStorm uDAPL driver has an accumulating issue. If the system has been running for more than 24 hours, and a large enough number of applications have been run, new applications could have problems establishing new uDAPL connections. The error occurs depending on the usage of the system.
HP-MPI V2.2 for Linux Release Note Known Problems and Workarounds while RDMA operations are being performed on the same page, silent data corruption can occur as RDMA operations continue to stream data to a page that has moved. To avoid this, the Mellanox driver does not use copy-on-write behavior during a fork for pinned pages. Instead, access to these pages by the child process will result in a segmentation violation.
HP-MPI V2.2 for Linux Release Note Known Problems and Workarounds • Profiling routines built for C calls will no longer cause the corresponding Fortran calls to be wrapped automatically. In order to profile Fortran routines, separate wrappers need to be written for the Fortran calls. • Calling MPI from Fortran 90 or C++ programs HP-MPI complies with the 1.2 version of the MPI standard, which defines bindings for Fortran 77 and C, but not Fortran 90 or C++.
HP-MPI V2.2 for Linux Release Note Known Problems and Workarounds 2. If the application receives on MPI_ANY_SOURCE. Try setting the environment variable LIBELAN_TPORT_BIGMSG to an appropriate message size to resolve the resource issue. If this doesn’t eliminate the error, contact Quadrics for the fix (which is unavailable at the time of this release).
HP-MPI V2.2 for Linux Release Note Installation Requirements Installation Requirements HP-MPI 2.2 for Linux is supported on Red Hat Enterprise Linux AS 3 and 4, or SuSE Linux Enterprise Server 9.0 or later operating systems on Intel IA-32, Itanium2, EM64T, and AMD Opteron systems; as well as HP XC3000, HP XC4000 and HP XC6000 Clusters. You must install the correct HP-MPI product for your system. HP-MPI requires a minimum of 50 MB of disk space in /opt.
HP-MPI V2.2 for Linux Release Note Installation Instructions Installation Instructions 1. Become root by running the su command and entering the superuser password. 2. Place the downloaded file into the /tmp directory (or another suitable directory). 3. Run the rpm command to install the package. % rpm -Uvh /tmp/hpmpi-2.02.00-??.[i386|ia64|x86_64].rpm Or use the following command to overwrite a previous version: % rpm -ivh --force /tmp/hpmpi-2.02.00-??.[i386|ia64|x86_64].
HP-MPI V2.2 for Linux Release Note Licensing Policy Licensing Policy HP-MPI 2.2 for Linux uses FLEXlm licensing technology. A license is required to use HP-MPI for Linux. Licenses can be purchased from HP’s software depot at http://www.hp.com/go/software, or contact your HP representative. Demo licenses for HP-MPI are also available from HP’s software depot. If you’re running on an HP XC system, no HP-MPI license is required at runtime because we use the XC system license. Licensing HP-MPI 2.
HP-MPI V2.2 for Linux Release Note Licensing Policy If the license needs to be placed in another location which would not be found by the above search, the user may set the environment variable LM_LICENSE_FILE to explicitly specify the location of the license file. For more information, see http://licensing.hp.com. Installing License Files A valid license file contains the system hostid and the associated license key. License files can be named either as license.dat or any name with extension of *.
HP-MPI V2.2 for Linux Release Note Product Documentation Product Documentation The HP-MPI Documentation Kit is an optional product (product number B6281AA) consisting of the following hardcopy books: • MPI: The Complete Reference (2 volume set) Product number B6011-96012 • HP-MPI User’s Guide (Ninth Edition) Product number B6060-96018 The HP-MPI User’s Guide and HP-MPI Release Notices are available online • in /opt/hpmpi/doc after you install the product. • at http://docs.hp.com. • at http://www.
HP-MPI V2.2 for Linux Release Note Product Documentation Table 4 Category Man Page Categories (Continued) Man Pages Description Compilation mpicc.1 mpiCC.1 mpif77.1 mpif90.1 Describes the available compilation utilities Runtime mpiclean.1 mpidebug.1 mpienv.1 mpiexec.1 mpijob.1 mpimtsafe.1 mpirun.1 mpirun.all.1 mpistdio.1 autodbl.1 Describes runtime utilities, environment variables, debugging, thread-safe and diagnostic libraries.
HP-MPI V2.2 for Linux Release Note Product Packaging Product Packaging HP-MPI is packaged as an optional software product installed by default in /opt/hpmpi.
HP-MPI V2.2 for Linux Release Note Software Availability in Native Languages Software Availability in Native Languages There is no information on non-English languages for HP-MPI for Linux systems.
