QLogic Host Channel Adapter and QLogic OFED Software Users Guide QLogic OFED Version 1.
S QLogic Host Channel Adapter and QLogic OFED Software Users Guide QLogic OFED Version 1.4 Information furnished in this manual is believed to be accurate and reliable. However, QLogic Corporation assumes no responsibility for its use, nor for any infringements of patents or other rights of third parties which may result from its use. QLogic Corporation reserves the right to change product specifications at any time without notice.
S QLogic Host Channel Adapter and QLogic OFED Software Users Guide QLogic OFED Version 1.4 In the second bullet, removed references to Ethernet emulation stacks and TCP (related to now-obsolete ipath_ether). Added that Infinipath uses a protocol that is optimized for MPI over Verbs. “Interoperability” on page 1-3 Changed the definition of #. Table 1-1 on page 1-4 Updated list of product documentation. “Documentation” on page 1-5 Updated contact information.
QLogic Host Channel Adapter and QLogic OFED Software Users Guide QLogic OFED Version 1.4 Deleted reference to MTRR BIOS setting. S “BIOS Settings” on page 4-5 Deleted paragraph about BIOS Setup utility. Added a NOTE about the Page Attribute Table (PAT) mechanism. Rearranged driver configuration information to match QLogic HCA and QLogic OFED Software Install Guide.
S QLogic Host Channel Adapter and QLogic OFED Software Users Guide QLogic OFED Version 1.4 Added ibv_devinfo command to see the current MTU size. “Other Configuration: Changing the MTU Size” on page 4-18 Added more details about setting the switch MTU default to 4K. Renamed Configuring the ib_ipath driver to Managing the Infinipath driver. Updated section: this driver now runs as a system service.
S QLogic Host Channel Adapter and QLogic OFED Software Users Guide QLogic OFED Version 1.4 Changed Intel® processor name. “Hyper-Threading” on page 4-24 New section “Host Environment Setup for MPI” on page 4-25 Moved Configuring ssh and sshd Using shosts.equiv, Configuring for ssh Using ssh-agent, and Process Limitation with ssh sections here. New section “Configuring for ssh” on page 4-25 Changed first paragraph. “Configuring ssh and sshd Using shosts.equiv” on page 4-25 Changed first paragraph.
S QLogic Host Channel Adapter and QLogic OFED Software Users Guide QLogic OFED Version 1.4 Renamed Compiling and Linking section. Added language information to MPI wrapper script names (Table 5-1). “Use Wrapper Scripts for Compiling and Linking” on page 5-6 Removed the NOTE about GNU environment. Replaced PathScale compiler with GNU compiler. Updated supported compilers (first paragraph). “To Use Another Compiler” on page 5-8 Added compiler options (Table 5-3, Table 5-4, Table 5-5).
S QLogic Host Channel Adapter and QLogic OFED Software Users Guide QLogic OFED Version 1.4 sysctl parameters do not help TCP/SDP performance in all systems. Dropped reference to TCP/SDP. “Use sysctl to Configure Kernel Parameters” on page 5-21 Added NOTE that sysctl tuning is not necessary for kernel versions 2.6.18 or higher.
S QLogic Host Channel Adapter and QLogic OFED Software Users Guide QLogic OFED Version 1.4 Moved mpirun Options Summary to new Appendix A. “mpirun Options Summary” on page A-1 was on page 4-20 Added these options: -H, -i, -job-info, -no-syslog, -statsmode Updated these options: -np, -rcfile, psc-debug-level (removed -d option), -debug, -v Updated performance numbers.
S QLogic Host Channel Adapter and QLogic OFED Software Users Guide QLogic OFED Version 1.4 In the first and last paragraphs, added a reference to write combining and the PAT mechanism. “Unexpected Low Bandwidth or Poor Latency” on page D-9 In the fifth paragraph, changed the bandwidth range for the QLE7240. Removed the sentence “This issue will be fixed in the next release”.
S QLogic Host Channel Adapter and QLogic OFED Software Users Guide QLogic OFED Version 1.4 Added program example. “ibv_devinfo” on page F-5 Updated command for QLogic RPMs. “ident” on page F-6 In the second paragraph, changed the RPM name to kernel-ib. Removed ident string for non-QLogic built kernel modules. Added a NOTE about using ibhosts to create a nodefile. “ipath_checkout” on page F-7 Updated output for ipath_control.
QLogic Host Channel Adapter and QLogic OFED Software Users Guide QLogic OFED Version 1.4 Updated output file name and example contents. S “boardversion” on page F-15 Removed NOTE information about QHT7040, as it is obsolete. Corrected definition: boardversion reports version of the chip architecture, not installed software. Change location/name of file. Remove sentence about subdirectories. “status_str” on page F-15 Changed location/name of the file. Updated file contents examples.
S Added Glossary entry for Verbs. IB6054601-00 H QLogic Host Channel Adapter and QLogic OFED Software Users Guide QLogic OFED Version 1.
QLogic Host Channel Adapter and QLogic OFED Software Users Guide QLogic OFED Version 1.
Table of Contents 1 Introduction Who Should Read this Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . How this Guide is Organized . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Interoperability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Conventions Used in this Guide . . . . . . . . . . . . .
QLogic Host Channel Adapter and QLogic OFED Software Users Guide QLogic OFED Version 1.4 S Other Configuration: Changing the MTU Size . . . . . . . . . . . . . . . . . . . . . . . Managing the InfiniPath Driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Configure the Infinipath Driver State . . . . . . . . . . . . . . . . . . . . . . . . . . Start, Stop, or Restart InfiniPath . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Unload the Driver/Modules Manually. . . . . .
A QLogic Host Channel Adapter and QLogic OFED Software Users Guide QLogic OFED Version 1.4 To Use Another Compiler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Compiler and Linker Variables . . . . . . . . . . . . . . . . . . . . . . . . . . Process Allocation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . InfiniPath Hardware Contexts on the QLE7240 and QLE7280. . Enabling and Disabling Software Context Sharing . . . . . . . . . . .
QLogic Host Channel Adapter and QLogic OFED Software Users Guide QLogic OFED Version 1.4 S Compiling MVAPICH Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . Running MVAPICH Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Further Information on MVAPICH . . . . . . . . . . . . . . . . . . . . . . . . . . . . Managing Open MPI, MVAPICH, and QLogic MPI with the mpi-selector Utility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A B QLogic Host Channel Adapter and QLogic OFED Software Users Guide QLogic OFED Version 1.4 Benchmark Programs Benchmark 1: Measuring MPI Latency Between Two Nodes . . . . . . . . . . . Benchmark 2: Measuring MPI Bandwidth Between Two Nodes . . . . . . . . . Benchmark 3: Messaging Rate Microbenchmarks. . . . . . . . . . . . . . . . . . . . Benchmark 4: Measuring MPI Latency in Host Rings . . . . . . . . . . . . . . . . . C Integration with a Batch Queuing System Using mpiexec with PBS . . . . . . . . .
QLogic Host Channel Adapter and QLogic OFED Software Users Guide QLogic OFED Version 1.4 S Performance Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Unexpected Low Bandwidth or Poor Latency . . . . . . . . . . . . . . . . . . . Large Message Receive Side Bandwidth Varies with Socket Affinity on Opteron Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . MVAPICH Performance Issues . . . . . . . . . . . . . . . . . . . . . . .
A QLogic Host Channel Adapter and QLogic OFED Software Users Guide QLogic OFED Version 1.4 MTRR Mapping and Write Combining . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Edit BIOS Settings to Fix MTRR Issues . . . . . . . . . . . . . . . . . . . . . . . Use the ipath_mtrr Script to Fix MTRR Issues. . . . . . . . . . . . . . . . F Useful Programs and Files Check Cluster Homogeneity with ipath_checkout . . . . . . . . . . . . . . . . . Restarting InfiniPath . . . . . . . . . . . . . . . . . .
QLogic Host Channel Adapter and QLogic OFED Software Users Guide QLogic OFED Version 1.4 S Networking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Rocks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Other Software Packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . G-2 G-2 G-2 Glossary Index List of Figures Figure Page 4-1 InfiniPath Software Structure . .
1 Introduction This section describes the objectives, intended audience, and organization of the QLogic Host Channel Adapter and QLogic OFED Software User Guide. The QLogic Host Channel Adapter and QLogic OFED Software User Guide shows end users how to use their InifiniPath cluster. End users include both the cluster administrator and the Message-Passing Interface (MPI) application programmers, who have different but overlapping interests in the details of the technology.
S 1–Introduction Overview Section 4, InfiniPath Cluster Setup and Administration, describes the lower levels of the supplied InfiniPath software. This section is of interest to an InfiniPath cluster administrator. Section 5, Using QLogic MPI, helps the MPI programmer make the best use of the QLogic MPI implementation. Examples are provided for compiling and running MPI programs. Section 6, Using Other MPIs, gives examples for compiling and running MPI programs with other MPI implementations.
A 1–Introduction Interoperability The QLogic host channel adapters are InfiniBand 4X. The Double Data Rate (DDR) QLE7240 and QLE7280 adapters have a raw data rate of 20Gbps (data rate of 16Gbps). For the Single Data Rate (SDR) adapters, the QLE7140 and QHT7140, the raw data rate is 10Gbps (data rate of 8Gbps). The QLE7240 and QLE7280 can also run in SDR mode. The QLogic adapters utilize standard, off-the-shelf InfiniBand 4X switches and cabling.
S 1–Introduction Conventions Used in this Guide Conventions Used in this Guide This guide uses the typographical conventions listed in Table 1-1. Table 1-1. Typographical Conventions Convention command Fixed-space font is used for literal items such as commands, functions, programs, files and pathnames, and program output. variable Italic fixed-space font is used for variable names in programs and command lines.
A 1–Introduction Documentation Documentation The product documentation includes: The QLogic HCA and QLogic OFED Software Install Guide The QLogic Host Channel Adapter and QLogic OFED Software User Guide The InfiniBand Software Installation Guide (for information on QLogic InfiniBand Fabric Suite) The OFED+ Users Guide (for information on QLogic VNIC and QLogic SRP) Release Notes Quick Start Guide Readme file Contact Information Support Headquarters QLogic Corporation 4601 Dean
1–Introduction Contact Information 1-6 S IB6054601-00 H
2 Feature Overview This section contains the features for this release, the supported QLogic adapter models, supported distributions and kernels, and a list of the software components. Features The QLogic OFED 1.4 software release contains the complete OFED 1.4, plus additional QLogic improvements, including an enhanced QLogic Host Channel Adapter driver. The InfiniPath 2.3 components (libraries, QLogic Message-Passing Interface/Performance Scaled Messaging (MPI/PSM), and utilities) are also included.
S 2–Feature Overview Features The QLogic InfiniBand Fabric Suite (IFS) is available separately for purchase. It includes FastFabric, the QLogic Fabric Manager, Fabric Viewer, and the InfiniServ Host Software. The QLogic OFED 1.4 software is supported by IFS. Support for new compiler versions (PathScale 3.x, PGI 7.x, PGI 8.x, Intel 10.x, Intel 11.
A 2–Feature Overview Features This version of the QLogic OFED software provides support for all of the QLogic adapters in Table 2-1. Table 2-1. QLogic Adapter Model Numbers QLogic Model Number Description QHT7040 Single port 10Gbps SDR 4X InfiniBand to HTX adapter. For systems with HTX expansion slots. QHT7140 a Single port 10Gbps SDR 4X InfiniBand to HTX adapter. For systems with HTX expansion slots. QLE7140 Single port 10Gbps SDR 4X InfiniBand to PCI Express x8 adapter.
S 2–Feature Overview Supported Distributions and Kernels Supported Distributions and Kernels The QLogic interconnect runs on AMD™ Opteron™ and 64-bit Intel Xeon systems running Linux®. The currently supported distributions and associated Linux kernel versions for InfiniPath and OpenFabrics are listed in Table 2-2. The kernels are the ones that shipped with the distributions. All are for the x86_64 architecture. Table 2-2.
A 2–Feature Overview Compiler Support Compiler Support QLogic MPI supports the following compilers: GNU gcc 3.3.x, 3.4.x, 4.0, 4.1, 4.2.x, and 4.3.x compiler suites PathScale Compiler Suite 3.0, 3.1 and 3.2 PGI 5.2, 6.0. 7.1, 7.2-4, and 8.0-3 Intel 9.x, 10.1, and 11.0 gfortran 4.1.x PathScale Compiler Suite Version 3.x is now supported on systems that have the GNU 4.0 and 4.1 compilers and compiler environment (header files and libraries).
S 2–Feature Overview Software Components Reliable Datagram Sockets (RDS) iSCSI Extensions for RDMA (iSER) This release supports two versions of SCSI RDMA Protocol (SRP): OFED SRP QLogic SRP No support is provided for Reliable Datagram (RD).
3 Step-by-Step Cluster Setup and MPI Usage Checklists This section describes how to set up your cluster to run high-performance MPI jobs. Cluster Setup Perform the following tasks when setting up the cluster. These include BIOS, adapter, and system settings. 1. Make sure that hardware and software installation and driver configuration has been completed according to the instructions in the QLogic HCA and QLogic OFED Software Install Guide.
3–Step-by-Step Cluster Setup and MPI Usage Checklists Using MPI S Using MPI 3-2 1. Verify that the QLogic hardware and software has been installed on all the nodes you will be using, and that ssh is set up on your cluster (see all the steps in the Cluster Setup checklist). 2. Copy the examples to your working directory. See “Copy Examples” on page 5-3. 3. Make an mpihosts file that lists the nodes on which your programs will run. See “Create the mpihosts File” on page 5-3. 4.
4 InfiniPath Cluster Setup and Administration This section describes what the cluster administrator needs to know about the InfiniPath software and system administration. Introduction The InfiniPath driver ib_ipath, Open source Subnet Manager (OpenSM), the protocol and Message-Passing Interface (MPI) support libraries, and other modules are components of the InfiniPath software. This software provides the foundation that supports the MPI implementation. Figure 4-1 illustrates these relationships.
4–InfiniPath Cluster Setup and Administration Installed Layout S Installed Layout This section describes the default installed layout for the InfiniPath software and QLogic-supplied MPIs.
A 4–InfiniPath Cluster Setup and Administration Memory Footprint The InfiniPath driver modules in this release are installed in: /lib/modules/$(uname -r)/updates/kernel/drivers/infiniband/hw/ipath Most of the other OFED modules are installed under the infiniband subdirectory.
S 4–InfiniPath Cluster Setup and Administration Memory Footprint Table 4-1. Memory Footprint of the QLogic Adapter on Linux x86_64 Systems (Continued) Adapter Required/ Component Optional OpenFabrics Optional Memory Footprint 1~6 MB Comment This component has not been fully characterized at the time of publication. + ~500 bytes per QP + TBD bytes per MR + ~500 bytes per EE + OpenFabrics stack from openfabrics.
A 4–InfiniPath Cluster Setup and Administration BIOS Settings BIOS Settings To achieve the best performance with QLogic adapters, you need to configure your BIOS with specific settings. The BIOS settings, which are stored in non-volatile memory, contain certain parameters characterizing the system. These parameters may include date and time, configuration settings, and information about the installed hardware. The Advanced Configuration and Power Interface (ACPI) setting must be enabled.
4–InfiniPath Cluster Setup and Administration OpenFabrics Drivers and Services Configuration and Startup S SRP (OFED and QLogic modules). SRP is not running until the module is loaded and the SRP devices on the fabric have been discovered. MPI over uDAPL (can be used by Intel MPI or HP®-MPI). IPoIB must be configured before MPI over uDAPL can be set up.
A 4–InfiniPath Cluster Setup and Administration OpenFabrics Drivers and Services Configuration and Startup The output from this command will be similar to: ib0 Link encap:InfiniBand HWaddr 00:00:00:02:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 inet addr:10.1.17.3 Bcast:10.1.17.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:4096 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:128 RX bytes:0 (0.
4–InfiniPath Cluster Setup and Administration OpenFabrics Drivers and Services Configuration and Startup S If you are using the rpm install method, note that after installing the opensm package, OpenSM is configured to be off after the next machine reboot.
A 4–InfiniPath Cluster Setup and Administration OpenFabrics Drivers and Services Configuration and Startup Using QLogic SRP If you installed QLogic SRP as part of the QLogicIB-Basic download, configure it according to the steps shown in the QLogic ULP and Tools Reference Guide (OFED+ Users Guide). Using OFED SRP To use OFED SRP, follow these steps: 1. Add the line SRP_LOAD=yes to the module list in /etc/infiniband/openib.conf to have it automatically loaded. 2.
4–InfiniPath Cluster Setup and Administration OpenFabrics Drivers and Services Configuration and Startup 3.
A 4–InfiniPath Cluster Setup and Administration OpenFabrics Drivers and Services Configuration and Startup Configuring and Administering the VNIC Interface The VirtualNIC (VNIC) Upper Layer Protocol (ULP) works in conjunction with firmware running on Virtual Input/Output (VIO) hardware such as the SilverStorm® Ethernet Virtual I/O Controller (EVIC™) or the InfiniBand/Ethernet Bridge Module for IBM® BladeCenter®, providing virtual Ethernet connectivity.
4–InfiniPath Cluster Setup and Administration OpenFabrics Drivers and Services Configuration and Startup S controller[ 1] GUID: 00066a01de000070 vendor ID: 00066a device ID: 000030 IO class : 2000 ID: EVIC in Chassis 0x00066a00db00001e, Slot 1, Ioc 1 service entries: 2 service[ 0]: 1000066a00000001 / InfiniNIC.InfiniConSys.Control:01 service[ 1]: 1000066a00000101 / InfiniNIC.InfiniConSys.
A 4–InfiniPath Cluster Setup and Administration OpenFabrics Drivers and Services Configuration and Startup controller[ 1] GUID: 00066a01de000070 vendor ID: 00066a device ID: 000030 IO class : 2000 ID: EVIC in Chassis 0x00066a00db00001e, Slot 1, Ioc 1 service entries: 2 service[ 0]: 1000066a00000001 / InfiniNIC.InfiniConSys.Control:01 service[ 1]: 1000066a00000101 / InfiniNIC.InfiniConSys.
S 4–InfiniPath Cluster Setup and Administration OpenFabrics Drivers and Services Configuration and Startup Look at the qlgc_vnic.cfg.sample file to see how VNIC configuration files are written. This file can be found with the OFED documentation, or in the qlgc_vnictools subdirectory of the QLogicIB_Basic download.
A 4–InfiniPath Cluster Setup and Administration OpenFabrics Drivers and Services Configuration and Startup b. Format 2: Define an IOC using the IOCSTRING. Defining the IOC using the IOCSTRING allows VIO hardware to be hot-swapped in and out of a specific slot. The host attempts to connect to the specified IOC (1, 2, or 3) on the VIO hardware that currently resides in the specified slot of the specified chassis.
S 4–InfiniPath Cluster Setup and Administration OpenFabrics Drivers and Services Configuration and Startup 4. Create VirtualNIC interface configuration files. For each Ethernet interface defined in the /etc/sysconfig/qlgc_vnic.cfg file, create an interface configuration file, /etc/sysconfig/network-scripts/ifcfg- (or /etc/sysconfig/network/ifcfg- on Linux 2.6 kernels), where is the value of the NAME field specified in the CREATE block.
A 4–InfiniPath Cluster Setup and Administration OpenFabrics Drivers and Services Configuration and Startup If you start the InfiniBand network stack separately, then the correct starting order is: Start the InfiniBand stack. Start QLogic VNIC service. For example, if you use InfiniPath, the correct starting order is: # /etc/init.d/openibd start # /etc/init.d/qlgc_vnic start The correct stopping order is: Stop QLogic VNIC service. Stop the InfiniBand stack.
4–InfiniPath Cluster Setup and Administration Other Configuration: Changing the MTU Size S More information for configuration, starting and stopping the interface, and basic troubleshooting is found in the QLogic OFED+ User Guide. Other Configuration: Changing the MTU Size The Maximum Transfer Unit MTU is set to 4K and enabled in the driver by default.
A 4–InfiniPath Cluster Setup and Administration Managing the InfiniPath Driver Managing the InfiniPath Driver The startup script for ib_ipath is installed automatically as part of the software installation, and normally does not need to be changed. It runs as a system service. The primary configuration file for the InfiniPath driver ib_ipath and other modules and associated daemons is /etc/infiniband/openib.conf.
S 4–InfiniPath Cluster Setup and Administration Managing the InfiniPath Driver Start, Stop, or Restart InfiniPath Restart the software if you install a new InfiniPath release, change driver options, or do manual testing. You can start, stop, or restart (as a root user) InfiniPath support with: # /etc/init.d/openibd [start | stop | restart] This method will not reboot the system. The following set of commands shows how to use this script.
A 4–InfiniPath Cluster Setup and Administration More Information on Configuring and Loading Drivers Unload the Driver/Modules Manually You can also unload the driver/modules manually without using /etc/init.d/openibd.
4–InfiniPath Cluster Setup and Administration Performance Settings and Management Tips S Performance Settings and Management Tips The following sections provide suggestions for improving performance and simplifying cluster management. Many of these settings will be done by the system administrator. User level runtime performance settings are shown in “Performance Tuning” on page 5-21.
A 4–InfiniPath Cluster Setup and Administration Performance Settings and Management Tips Use a PCIe Max Read Request size of at least 512 bytes with the QLE7240 and QLE7280. QLE7240 and QLE7280 adapters can support sizes from 128 bytes to 4096 byte in powers of two. This value is typically set by the BIOS. Use PCIe Max Payload size of 256, where available, with the QLE7240 and QLE7280. The QLE7240 and QLE7280 adapters can support 128, 256, or 512 bytes.
S 4–InfiniPath Cluster Setup and Administration Performance Settings and Management Tips If you are using Network File System (NFS) or yellow pages (yp) passwords: rpcidmapd ypbind portmap nfs nfslock autofs To watch for disk problems, use: smartd readahead The service comprising the InfiniPath driver and SMA is: openibd Other services may be required by your batch queuing system or user community. If your system is running the daemon irqbalance, QLogic recommends turning it off.
A 4–InfiniPath Cluster Setup and Administration Host Environment Setup for MPI Host Environment Setup for MPI After the InfiniPath software and the GNU (GCC) compilers have been installed on all the nodes, the host environment can be set up for running MPI programs. Configuring for ssh Running MPI programs with the command mpirun on an InfiniPath cluster depends, by default, on secure shell ssh to launch node programs on the nodes.
4–InfiniPath Cluster Setup and Administration Host Environment Setup for MPI S The example in this section assumes the following: Both the cluster nodes and the front end system are running the openssh package as distributed in current Linux systems. All cluster end users have accounts with the same account name on the front end and on each node, by using Network Information Service (NIS) or another means of distributing the password file. The front end used in this example is called ip-fe.
A 4–InfiniPath Cluster Setup and Administration Host Environment Setup for MPI 812E5iw1GCZfNefBmWbegWVKFwGlNbqBnZK7kDRLSOKQtuhYbGPcrVlSjuVps fWEju64FTqKEetA8l8QEgAAAIBNtPDDwdmXRvDyc0gvAm6lPOIsRLmgmdgKXT GOZUZ0zwxSL7GP1nEyFk9wAxCrXv3xPKxQaezQKs+KL95FouJvJ4qrSxxHdd1 NYNR0DavEBVQgCaspgWvWQ8cL 0aUQmTbggLrtD9zETVU5PCgRlQL6I3Y5sCCHuO7/UvTH9nneCg== Change the file to mode 600 when you are finished editing. 4.
S 4–InfiniPath Cluster Setup and Administration Host Environment Setup for MPI 2. Enter a passphrase for your key pair when prompted. Note that the key agent does not survive X11 logout or system reboot: $ ssh-add 3. The following command tells ssh that your key pair should let you in: $ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys Edit the ~/.
A 4–InfiniPath Cluster Setup and Administration Checking Cluster and Software Status Checking Cluster and Software Status ipath_control InfiniBand status, link speed, and PCIe bus width can be checked by running the program ipath_control. Sample usage and output are as follows: $ ipath_control -iv $Id: QLogic OFED Release 1.4 $ $Date: 2009-03-10-10:15 $ 0: Version: ChipABI 2.0, InfiniPath_QLE7280, InfiniPath1 5.
S 4–InfiniPath Cluster Setup and Administration Checking Cluster and Software Status ibv_devinfo ibv_devinfo queries RDMA devices. Use the -v option to see more information. Sample usage: $ ibv_devinfo hca_id: ipath0 fw_ver: 0.0.
A 4–InfiniPath Cluster Setup and Administration Checking Cluster and Software Status The Intel Cluster Checker The Intel Cluster Checker checks and verifies Intel cluster ready certified clusters. To run the Intel Cluster Checker, create or edit (as a root user) the /etc/security/limits.conf file. The file will contain a single line: * soft memlock 131072 The asterisk (*) must be in the first column of the line.
4–InfiniPath Cluster Setup and Administration Checking Cluster and Software Status S Notes 4-32 IB6054601-00 H
5 Using QLogic MPI This section provides information on using the QLogic Message-Passing Interface (MPI). Examples are provided for setting up the user environment, and for compiling and running MPI programs. Introduction The MPI standard is a message-passing library or collection of routines used in distributed-memory parallel programming. It is used in data exchange and task synchronization between processes.
5–Using QLogic MPI Introduction S Other MPIs Other high-performance MPIs, such as HP-MPI version 2.3, Open MPI (release 1.2.8), Ohio State University MVAPICH (1.1), and Scali 5.6.4 (Platform) MPI, have been ported to the PSM interface. Open MPI, MVAPICH, HP-MPI, and Scali also run over IB Verbs (the Open Fabrics Alliance API that provides support for user level upper layer protocols like MPI). Intel MPI, although not ported to the PSM interface, is supported over uDAPL, which uses IB Verbs.
A 5–Using QLogic MPI Getting Started with MPI Getting Started with MPI This section shows how to compile and run some simple example programs that are included in the InfiniPath software product. Compiling and running these examples enables you to verify that QLogic MPI and its components have been properly installed on the cluster. See “QLogic MPI Troubleshooting” on page D-12 if you have problems compiling or running these examples.
5–Using QLogic MPI Getting Started with MPI S Then, run the program with several different specifications for the number of processes: $ mpirun -np 2 -m mpihosts ./cpi Process 0 on hostname1 Process 1 on hostname2 pi is approximately 3.1416009869231241, Error is 0.0000083333333309 wall clock time = 0.000149 In this example, ./cpi designates the executable of the example program in the working directory. The -np parameter to mpirun defines the number of processes to be used in the parallel computation.
A 5–Using QLogic MPI QLogic MPI Details fpi.f is a Fortran 77 program that computes pi in a way similar to cpi.c. Compile and link, and run it as follows: $ mpif77 -o fpi fpi.f $ mpirun -np 2 -m mpihosts ./fpi pi3f90.f90 is a Fortran 90 program that does the same computation. Compile and link, and run it as follows: $ mpif90 -o pi3f90 pi3f90.f90 $ mpirun -np 2 -m mpihosts ./pi3f90 The C++ program hello++.cc is a parallel processing version of the traditional “Hello, World” program.
S 5–Using QLogic MPI QLogic MPI Details Use Wrapper Scripts for Compiling and Linking The scripts in Table 5-1 invoke the compiler and linker for programs in each of the respective languages, and take care of referring to the correct include files and libraries in each case. Table 5-1. QLogic MPI Wrapper Scripts Wrapper Script Name Language mpicc C mpicxx C++ mpif77 Fortran 77 mpif90 Fortran 90 mpif95 Fortran 95 On x86_64, these scripts (by default) call the GNU compiler and linker.
A 5–Using QLogic MPI QLogic MPI Details Configuring MPI Programs for QLogic MPI When configuring an MPI program (generating header files and/or Makefiles) for QLogic MPI, you usually need to specify mpicc, mpicxx, and so on as the compiler, rather than gcc, g++, etc.
S 5–Using QLogic MPI QLogic MPI Details To Use Another Compiler QLogic MPI supports a number of compilers, in addition to the default GNU Compiler Collection (GCC) versions 3.3.x, 3.4.x, 4.0, and 4.1 and gfortran. These include the PathScale Compiler Suite 3.0 and 3.1; PGI 5.2, 6.0, and 7.1; and Intel 9.x and 10.1. NOTE: The GNU 4.x environment is supported in the PathScale Compiler Suite 3.x release. However, the 2.x PathScale compilers are not currently supported on SLES 10 systems that use the GNU 4.
A 5–Using QLogic MPI QLogic MPI Details Table 5-4. Portland Group (PGI) (Continued) Compiler Command Fortran 77 mpif77 -fc=pgf77 ..... Fortran 90/95 mpif90 -f90=pgf90 ..... mpif95 -f95=pgf95 ..... Table 5-5. Intel Compiler Command C $ mpicc -cc=icc ..... C++ $ mpicc -CC=icpc Fortran 77 $ mpif77 -fc=ifort ..... Fortran 90/95 $ mpif90 -f90=ifort ..... $ mpif95 -f95=ifort ..... Also, use mpif77, mpif90, or mpif95 for linking; otherwise, .true. may have the wrong value.
S 5–Using QLogic MPI QLogic MPI Details Compiler and Linker Variables When you use environment variables (e.g., $MPICH_CC) to select which compiler mpicc (and others) will use, the scripts will also set the matching linker variable (for example, $MPICH_CLINKER), if it is not already set. When both the environment variable and command line options are used (-cc=gcc), the command line variable is used.
A 5–Using QLogic MPI QLogic MPI Details When running a job in a batch system environment where multiple jobs may be running simultaneously, it is useful to restrict the number of InfiniPath contexts that are made available on each node of an MPI. See “Restricting InfiniPath Hardware Contexts in a Batch Environment” on page 5-12. Errors that may occur with context sharing are covered in “Context Sharing Error Messages” on page 5-13. There are multiple ways of specifying how processes are allocated.
5–Using QLogic MPI QLogic MPI Details S Enabling and Disabling Software Context Sharing By default, context sharing is enabled; it can also be specifically disabled. Context Sharing Enabled: The MPI library provides PSM the local process layout so that InfiniPath contexts available on each node can be shared if necessary; for example, when running more node programs than contexts.
A 5–Using QLogic MPI QLogic MPI Details Additionally, you can explicitly configure for the number of contexts you want to use with the cfgports module parameter. This will override the default settings (on the QLE7240 and QLE7280) based on the number of CPUs present on each node. See “InfiniPath Hardware Contexts on the QLE7240 and QLE7280” on page 5-11.
S 5–Using QLogic MPI QLogic MPI Details mpihosts File Details As noted in “Create the mpihosts File” on page 5-3, an mpihosts file (also called a machines file, nodefile, or hostsfile) has been created in your current working directory. This file names the nodes on which the node programs may run. The two supported formats for the mpihosts file are: hostname1 hostname2 ... or hostname1:process_count hostname2:process_count ...
A 5–Using QLogic MPI QLogic MPI Details In the absence of both the -m option and the MPIHOSTS environment variable, mpirun uses the file ./mpihosts, if it exists. If none of these three methods of specifying the hosts file are used, mpirun looks for the file ~/.mpihosts. If you are working in the context of a batch queuing system, it may provide a job submission script that generates an appropriate mpihosts file.
S 5–Using QLogic MPI QLogic MPI Details Generally, mpirun tries to distribute the specified number of processes evenly among the nodes listed in the mpihosts file. However, if the number of processes exceeds the number of nodes listed in the mpihosts file, then some nodes will be assigned more than one instance of the program.
A 5–Using QLogic MPI QLogic MPI Details mpirun monitors the parallel MPI job, terminating when all the node programs in that job exit normally, or if any of them terminates abnormally. Killing the mpirun program kills all the processes in the job. Use CTRL+C to kill mpirun. Console I/O in MPI Programs mpirun sends any output printed to stdout or stderr by any node program to the terminal.
S 5–Using QLogic MPI QLogic MPI Details Environment variables are specified in descending order, as follows: 1. Set in the default shell environment on a remote node, e.g., ~/.bashrc or equivalents. 2. Set in -rcfile. 3. Set the current shell environment for the mpirun command. 4. If nothing has been set (none of the previous sets have been performed), the default value of the environment variable is used.
A 5–Using QLogic MPI QLogic MPI Details Table 5-7. Environment Variables (Continued) Name IPATH_UNIT Description This variable is for context sharing. When multiple InfiniPath devices are present, this variable restricts the use to a specific InfiniPath unit. By default, all configured units are used in round robin order. Default: Unset LD_LIBRARY_PATH This variable specifies the path to the run-time library. It is often set in the .mpirunrc file.
S 5–Using QLogic MPI QLogic MPI Details Table 5-7. Environment Variables (Continued) Name PSM_SHAREDCONTEXTS_MAX Description This variable restricts the number of InfiniPath contexts that are made available on each node of an MPI job.
A 5–Using QLogic MPI Performance Tuning To keep remote process checks but disable triggering quiescence for temporary IB link failures, use the -disable-mpi-progress-check option with a nonzero -q option. To disable quiescence triggering altogether, use -q 0. No matter how these options are used, link failures (temporary or other) are always logged to syslog. If the link is down when the job starts and you want the job to continue blocking until the link comes up, use the -t -1 option.
S 5–Using QLogic MPI Performance Tuning Use the taskset utility with mpirun to specify the mapping of MPI processes to logical processors. This combination makes the best use of available memory bandwidth or cache locality when running on dual-core Symmetric MultiProcessing (SMP) cluster nodes. The following example uses the NASA Advanced Supercomputing (NAS) Parallel Benchmark’s Multi-Grid (MG) benchmark and the -c option to taskset. $ mpirun -np 4 -ppn 2 -m $hosts taskset -c 0,2 bin/mg.B.
A 5–Using QLogic MPI MPD The complete list of options is contained in Appendix A. MPD The Multi-Purpose Daemon (MPD) is an alternative to mpirun for launching MPI jobs. It is described briefly in the following sections. MPD Description MPD was developed by Argonne National Laboratory (ANL) as part of the MPICH-2 system.
5–Using QLogic MPI QLogic MPI and Hybrid MPI/OpenMP Applications S NOTE: To use MPD, the software package mpi-frontend-*.rpm and python (available with your distribution) must be installed on all nodes. See the QLogic HCA and QLogic OFED Software Install Guide for more details on software installation. QLogic MPI and Hybrid MPI/OpenMP Applications QLogic MPI supports hybrid MPI/OpenMP applications, provided that MPI routines are called only by the master OpenMP thread.
A 5–Using QLogic MPI Debugging MPI Programs NOTE: When there are more threads than CPUs, both MPI and OpenMP performance can be significantly degraded due to over-subscription of the CPUs. Debugging MPI Programs Debugging parallel programs is substantially more difficult than debugging serial programs. Thoroughly debugging the serial parts of your code before parallelizing is good programming practice.
5–Using QLogic MPI QLogic MPI Limitations S The following features of QLogic MPI facilitate debugging: Stack backtraces are provided for programs that crash. The -debug and -debug-no-pause options are provided for mpirun. These options make each node program start with debugging enabled. The -debug option allows you to set breakpoints, and start running programs individually. The -debug-no-pause option allows postmortem inspection. Be sure to set -q 0 when using -debug.
6 Using Other MPIs This section provides information on using other MPI implementations. Introduction Support for multiple high-performance MPI implementations has been added. Most implementations run over both PSM and OpenFabrics Verbs (see Table 6-1). Table 6-1. Other Supported MPI Implementations MPI Implementation Open MPI 1.2.8 Runs Over PSM Verbs Compiled With GCC, PGI, PathScale, Intel GCC Comments Provides some MPI-2 functionality (one-sided operations and dynamic processes).
S 6–Using Other MPIs Installed Layout Table 6-1. Other Supported MPI Implementations (Continued) MPI Implementation Intel MPI version 3.1 Compiled With Runs Over uDAPL GCC (default) Comments Provides MPI-1 functionality. Available for purchase from Intel. Table Notes MVAPICH and Open MPI have been have been compiled for PSM to support the following versions of the compilers: (GNU) gcc 4.1.0 (PathScale) pathcc 3.0 (PGI) pgcc 7.2-5 (Intel) icc 11.0 (Version 11.
A 6–Using Other MPIs Open MPI Open MPI Open MPI is an open source MPI-2 implementation from the Open MPI Project. Pre-compiled versions of Open MPI version 1.2.8 that run over PSM and are built with the GCC, PGI, PathScale, and Intel compilers are available with the QLogic download. Open MPI that runs over Verbs and is pre-compiled with the GNU compiler is also available.
6–Using Other MPIs Open MPI S To compile your program in C, type: $ mpicc mpi_app_name.c -o mpi_app_name Running Open MPI Applications By default, Open MPI shipped with the InfiniPath software stack will run over PSM once it is installed. Here is an example of a simple mpirun command running with four processes: $ mpirun -np 4 -machinefile mpihosts mpi_app_name To specify the PSM transport explicitly, add --mca mtl psm to the above command line.
A 6–Using Other MPIs MVAPICH MVAPICH Pre-compiled versions of MVAPICH 1.1 built with the GNU, PGI, PathScale, and Intel compilers, and that run over PSM, are available with the QLogic download. MVAPICH that runs over Verbs and is pre-compiled with the GNU compiler is also available. MVAPICH can be managed with the mpi-selector utility, as described in “Managing Open MPI, MVAPICH, and QLogic MPI with the mpi-selector Utility” on page 6-6.
6–Using Other MPIs Managing Open MPI, MVAPICH, and QLogic MPI with the mpi-selector Utility S To compile your program in C, type: $ mpicc mpi_app_name.c -o mpi_app_name To check the default configuration for the installation, check the following file: /usr/mpi/<$compiler>/<$mpi>/etc/mvapich.conf Running MVAPICH Applications By default, the MVAPICH shipped with the InfiniPath software stack runs over PSM once it is installed.
A 6–Using Other MPIs HP-MPI Following is an example for listing and selecting an MPI: $ mpi-selector --list mpi-1.2.3 mpi-3.4.5 $ mpi-selector --set mpi-4.5.6 The new default take effect in the next shell that is started. See the mpi-selector man page for more information. For QLogic MPI inter-operation with the mpi-selector utility, you must install all QLogic MPI RPMs using a prefixed installation.
S 6–Using Other MPIs HP-MPI Installation Follow the instructions for downloading and installing HP-MPI from the HP web site. Setup No special setup is needed. Compiling HP-MPI Applications As with QLogic MPI, QLogic recommends that you use the included wrapper scripts that invoke the underlying compiler (see Table 6-4). Table 6-4.
A 6–Using Other MPIs Platform (Scali) MPI Platform (Scali) MPI Platform MPI was formerly known as Scali MPI Connect. The version tested with this release is 5.6.4. Installation Follow the instructions for downloading and installing Platform MPI from the Platform (Scali) web site. Setup When run over PSM, no special setup is needed. If running over IB Verbs, Platform MPI needs to know which IB adapter to use. This is achieved by creating the file /opt/scali/etc/iba_params.
S 6–Using Other MPIs Intel MPI Running Platform MPI Applications Here is an example of a simple mpirun command running with four processes, over PSM: $ mpirun -np 4 -machinefile mpihosts mpi_app_name Once installed, Platform MPI uses the PSM transport by default. To specify PSM explicitly, add -networks infinipath to the above command.
A 6–Using Other MPIs Intel MPI compat-dapl-devel-static-1.2.12-1.x86_64.rpm compat-dapl-utils-1.2.12-1.x86_64.rpm 2. Verify that there is a /etc/dat.conf file. It should be installed by the dapl- RPM. The file dat.conf contains a list of interface adapters supported by uDAPL service providers. In particular, it must contain mapping entries for OpenIB-cma for dapl 1.2.x, in a form similar to this (all on one line): OpenIB-cma u1.2 nonthreadsafe default libdaplcma.so.1 dapl.1.2 "ib0 0" "" 3.
S 6–Using Other MPIs Improving Performance of Other MPIs Over IB Verbs Table 6-6. Intel MPI Wrapper Scripts (Continued) Wrapper Script Name Language mpif77 Fortran 77 mpif90 Fortran 90 mpiicc C (uses Intel C compiler) mpiicpc C++ (uses Intel C++ compiler) mpiifort Fortran 77/90 (uses Intel Fortran compiler) To compile your program in C using the default compiler, type: $ mpicc mpi_app_name.
A 6–Using Other MPIs Improving Performance of Other MPIs Over IB Verbs The maximum supported MTU size of InfiniPath adapter cards are: QHT7140: 2K QLE7140, QLE7240, QLE7280: 4K Support for 4K IB MTU requires switch support for 4K MTU. The method to set the IB MTU size varies by MPI implementation: Open MPI defaults to the lower of either the IB MTU size or switch MTU size. MVAPICH defaults to an IB MTU size of 1024 bytes.
6–Using Other MPIs Improving Performance of Other MPIs Over IB Verbs S Notes 6-14 IB6054601-00 H
A mpirun Options Summary This section summarizes the most commonly used options to mpirun. See the mpirun (1) man page for a complete listing. Job Start Options -mpd This option is used after running mpdboot to start a daemon, rather than using the default ssh protocol to start jobs. See the mpdboot(1) man page for more information. None of the other mpirun options (with the exception of -h) are valid when using this option.
S A–mpirun Options Summary Spawn Options -np np This option specifies the number of processes to spawn. If this option is not set, then the environment variable MPI_NPROCS is checked. If MPI_NPROCS is not set, the default is to determine the number of processes based on the number of hosts in the machinefile -M or the list of hosts -H. -ppn processes-per-node This option creates up to the specified number of processes per node.
A A–mpirun Options Summary Quiescence Options Quiescence Options -disable-mpi-progress-check This option disables the MPI communication progress check without disabling the ping reply check. If quiescence or a lack of ping reply is detected, the job and all compute processes are terminated. -i, -ping-interval,seconds This options specifies the seconds to wait between ping packets to mpirun (if -q > 0).
S A–mpirun Options Summary Stats Options -k, -kill-timeout seconds This option indicates the time to wait for other ranks after the first rank exits. Default: 60 -listen-addr | This option specifies which hostname (or IPv4 address) to listen on for incoming socket connections. It is useful for an mpirun front-end multihomed host. By default, mpirun assumes that ranks can independently resolve the hostname obtained on the head node with gethostname(2).
A A–mpirun Options Summary Tuning Options -statsfile file-prefix This option specifies an alternate file to receive the output from the -print-stats option. Default: stderr -statsmode absolute|diffs When printing process statistics with the -print-stats option, this option specifies if the printed statistics have the absolute values of the QLogic adapter chip counters and registers or if there are differences between those values at the start and end of the process.
S A–mpirun Options Summary Shell Options Shell Options -shell shell-name This option specifies the name of the program to use to log into remote hosts. Default: ssh, unless $MPI_SHELL is defined. -shellx shell-name This option specifies the name of program to use to log into remote hosts with X11 forwarding. This option is useful when running with -debug or in xterm. Default: ssh, unless $MPI_SHELL_X is defined.
A A–mpirun Options Summary Format Options Format Options -l, -label-output This option labels each line of output on stdout and stderr with the rank of the MPI process that produces the output. -y, -labelstyle string This option specifies the label that is prefixed to error messages and statistics. Process rank is the default prefix.
A–mpirun Options Summary Other Options S Notes A-8 IB6054601-00 H
B Benchmark Programs Several MPI performance measurement programs are installed from the mpi-benchmark RPM. This appendix describes these benchmarks and how to run them. These programs are based on code from the group of Dr. Dhabaleswar K. Panda at the Network-Based Computing Laboratory at the Ohio State University. For more information, see: http://mvapich.cse.ohio-state.edu/ These programs allow you to measure the MPI latency and bandwidth between two or more nodes in your cluster.
B–Benchmark Programs Benchmark 1: Measuring MPI Latency Between Two Nodes S defined. The program uses a loop, executing many such exchanges for each message size, to get an average. The program defers the timing until the message has been sent and received a number of times, to be sure that all the caches in the pipeline have been filled. This benchmark always involves two node programs.
A B–Benchmark Programs Benchmark 2: Measuring MPI Bandwidth Between Two Nodes Benchmark 2: Measuring MPI Bandwidth Between Two Nodes The osu_bw benchmark measures the maximum rate at which you can pump data between two nodes.
B–Benchmark Programs Benchmark 3: Messaging Rate Microbenchmarks S Benchmark 3: Messaging Rate Microbenchmarks mpi_multibw is the microbenchmark that highlights QLogic’s messaging rate results. This benchmark is a modified form of the OSU NOWlab’s osu_bw benchmark (as shown in the previous example). It has been enhanced with the following additional functionality: The messaging rate and the bandwidth are reported. N/2 is dynamically calculated at the end of the run.
A B–Benchmark Programs Benchmark 4: Measuring MPI Latency in Host Rings Searching for N/2 bandwidth. MB/s... Maximum Bandwidth of 1958.400649 Found N/2 bandwidth of 992.943275 MB/s at size 153 bytes This microbenchmark is available and can be downloaded from the QLogic web site: http://www.qlogic.com Benchmark 4: Measuring MPI Latency in Host Rings The program mpi_latency measures latency in a ring of hosts.
B–Benchmark Programs Benchmark 4: Measuring MPI Latency in Host Rings S Notes B-6 IB6054601-00 H
C Integration with a Batch Queuing System Most cluster systems use some kind of batch queuing system as an orderly way to provide users with access to the resources they need to meet their job’s performance requirements. One task of the cluster administrator is to allow users to submit MPI jobs through these batch queuing systems. Two methods are described in this document: Use mpiexec within the Portable Batch System (PBS) environment.
S C–Integration with a Batch Queuing System Using SLURM for Batch Queuing NOTE: This level of support is specific to QLogic MPI, and not to other MPIs that currently support InfiniPath. For more usage information, see the OSC mpiexec documentation. For more information on PBS, go to: http://www.pbsgridworks.com/ Using SLURM for Batch Queuing The following is an example of the some of the functions that a batch queuing script might perform.
A C–Integration with a Batch Queuing System Using SLURM for Batch Queuing Allocating Resources When the mpirun command starts, it requires specification of the number of node programs it must spawn (via the -np option) and specification of an mpihosts file listing the nodes on which the node programs may be run. (See “Environment for Node Programs” on page 5-17 for more information.
C–Integration with a Batch Queuing System Using SLURM for Batch Queuing S The sort | uniq -c component determines the number of times each unique line was printed. The awk command converts the result into the mpihosts file format used by mpirun. Each line consists of a node name, a colon, and the number of processes to start on that node. NOTE: This is one of two formats that the file can use. See “Console I/O in MPI Programs” on page 5-17 for more information.
A C–Integration with a Batch Queuing System Lock Enough Memory on Nodes when Using SLURM The following command terminates all processes using the QLogic interconnect: # /sbin/fuser -k /dev/ipath For more information, see the man pages for fuser(1) and lsof(8). Note that hard and explicit program termination, such as kill -9 on the mpirun Process ID (PID), may result in QLogic MPI being unable to guarantee that the /dev/shm shared memory file is properly removed.
S C–Integration with a Batch Queuing System Lock Enough Memory on Nodes when Using SLURM The following message displays after installation: $ mpirun -m ~/tmp/sm -np 2 -mpi_latency 1000 1000000 node-00:1.ipath_update_tid_err: failed: Cannot allocate memory mpi_latency: /fs2/scratch/infinipath-build-1.3/mpi-1.3/mpich/psm/src mq_ips.c:691: mq_ipath_sendcts: Assertion ‘rc == 0’ failed. MPIRUN: Node program unexpectedly quit. Exiting. You can check the ulimit -l on all the nodes by running ipath_checkout.
D Troubleshooting This appendix describes some of the tools you can use to diagnose and fix problems. The following topics are discussed: Using LEDs to Check the State of the Adapter BIOS Settings Kernel and Initialization Issues OpenFabrics and InfiniPath Issues System Administration Troubleshooting Performance Issues QLogic MPI Troubleshooting Troubleshooting information for hardware and software installation is found in the QLogic HCA and QLogic OFED Software Install Guide.
S D–Troubleshooting BIOS Settings Table D-1. LED Link and Data Indicators (Continued) LED States Green ON Amber OFF Indication Signal detected and the physical link is up. Ready to talk to SM to bring the link fully up. If this state persists, the SM may be missing or the link may not be configured. Use ipath_control -i to verify the software state. If all host channel adapters are in this state, then the SM is not running. Check the SM configuration, or install and run opensmd.
A D–Troubleshooting Kernel and Initialization Issues Kernel and Initialization Issues Issues that may prevent the system from coming up properly are described in the following sections. Driver Load Fails Due to Unsupported Kernel If you try to load the InfiniPath driver on a kernel that InfiniPath software does not support, the load fails. Error messages similar to this display: modprobe: error inserting ’/lib/modules/2.6.3-1.1659-smp/kernel/drivers/infiniband/hw/ipath/ ib_ipath.
S D–Troubleshooting Kernel and Initialization Issues If there is no output at all, the driver initialization failed. For more information on driver problems, see “Driver Load Fails Due to Unsupported Kernel” on page D-3 or “InfiniPath ib_ipath Initialization Failure” on page D-5. If the output is similar to one of these lines, then interrupts are not being delivered to the driver.
A D–Troubleshooting Kernel and Initialization Issues OpenFabrics Load Errors if ib_ipath Driver Load Fails When the ib_ipath driver fails to load, the other OpenFabrics drivers/modules will load and be shown by lsmod, but commands like ibstatus, ibv_devinfo, and ipath_control -i will fail as follows: # ibstatus Fatal error: device ’*’: sys files not found (/sys/class/infiniband/*/ports) # ibv_devinfo libibverbs: Fatal: couldn’t read uverbs ABI version.
S D–Troubleshooting OpenFabrics and InfiniPath Issues If the driver loaded, but MPI or other programs are not working, check to see if problems were detected during the driver and QLogic hardware initialization with the command: $ dmesg | grep -i ipath This command may generate more than one screen of output.
A D–Troubleshooting OpenFabrics and InfiniPath Issues Manual Shutdown or Restart May Hang if NFS in Use If you are using NFS over IPoIB and use the manual /etc/init.d/openibd stop (or restart) command, the shutdown process may silently hang on the fuser command contained within the script. This is because fuser cannot traverse down the tree from the mount point once the mount point has disappeared. To remedy this problem, the fuser process itself needs to be killed.
S D–Troubleshooting OpenFabrics and InfiniPath Issues SDP Module Not Loading If the settings for debug level and the zero copy threshold from InfiniPath release 2.0 are present in the release 2.2 /etc/modprobe.conf file (RHEL) or /etc/modprobe.conf.local (SLES) file, the SDP module may not load: options ib_sdp sdp_debug_level=4 sdp_zcopy_thrsh_src_default=10000000 To solve the problem, remove this line.
A D–Troubleshooting System Administration Troubleshooting System Administration Troubleshooting The following sections provide details on locating problems related to system administration. Broken Intermediate Link Sometimes message traffic passes through the fabric while other traffic appears to be blocked. In this case, MPI jobs fail to run. In large cluster configurations, switches may be attached to other switches to supply the necessary inter-node connectivity.
S D–Troubleshooting Performance Issues The exact symptoms can vary with BIOS, amount of memory, etc. When the driver starts, you may see these errors: ib_ipath 0000:04:01.0: infinipath0: Performance problem: bandwidth to PIO buffers is only 273 MiB/sec infinipath: mtrr_add(feb00000,0x100000,WC,0) failed (-22) infinipath: probe of 0000:04:01.0 failed with error -22 If you do not see any of these messages on your console, but suspect this problem, check the /var/log/messages file.
A D–Troubleshooting Performance Issues Erratic Performance Sometimes erratic performance is seen on applications that use interrupts. An example is inconsistent SDP latency when running a program such as netperf. This may be seen on AMD-based systems using the QLE7240 or QLE7280 adapters. If this happens, check to see if the program irqbalance is running. This program is a Linux daemon that distributes interrupts across processors.
S D–Troubleshooting QLogic MPI Troubleshooting Here is an example: # cat /sys/class/infiniband/ipath*/device/irq 98 # ls /proc/irq 0 10 11 13 15 233 4 50 7 8 90 1 106 12 14 2 3 5 58 66 74 9 (Note that you cannot find 98.) # cat /proc/interrupts|grep ib_ipath|awk \ ’{print $1}’|sed -e ’s/://’ 106 # echo 01 > /proc/irq/106/smp_affinity Using the echo command immediately changes the processor affinity of an IRQ.
A D–Troubleshooting QLogic MPI Troubleshooting Mixed Releases of MPI RPMs Make sure that all of the MPI RPMs are from the same release. When using mpirun, an error message will occur if different components of the MPI RPMs are from different releases. In the following example, mpirun from release 2.1 is being used with a 2.2 library. $ mpirun -np 2 -m ~/tmp/x2 osu_latency MPI_runscript-xqa-14.0: ssh -x> Cannot detect InfiniPath interconnect. MPI_runscript-xqa-14.
D–Troubleshooting QLogic MPI Troubleshooting S Resolving Hostname with Multi-Homed Head Node By default, mpirun assumes that ranks can independently resolve the hostname obtained on the head node with gethostname. However, the hostname of a multi-homed head node may not resolve on the compute nodes. To address this problem, the following new option has been added to mpirun: -listen-addr This address will be forwarded to the ranks.
A D–Troubleshooting QLogic MPI Troubleshooting Compiler/Linker Mismatch If the compiler and linker do not match in C and C++ programs, the following error message appears: $ export MPICH_CC=gcc $ mpicc mpiworld.c /usr/bin/ld: cannot find -lmpichabiglue_gcc3 collect2: ld returned 1 exit status Compiler Cannot Find Include, Module, or Library Files RPMs can be installed in any location by using the --prefix option.
S D–Troubleshooting QLogic MPI Troubleshooting Compiling on Development Nodes If the mpi-devel-* RPM is installed with the --prefix /path/to/devel option, then mpicc, etc. must be passed in -I/path/to/devel/include for the compiler to find the MPI include files, as in this example: $ mpicc myprogram.c -I/path/to/devel/include If you are using Fortran 90 or Fortran 95, a similar option is needed for the compiler to find the module files: $ mpif90 myprogramf90.
A D–Troubleshooting QLogic MPI Troubleshooting For the second option, change the file /etc/ld.so.conf on the compute nodes rather than using the -Wl,-rpath, option when compiling on the development node. It is assumed that the mpi-lib-* RPM is installed on the compute nodes with the same --prefix /path/to/libs option as on the development nodes. Then, on the computer nodes, add the following lines to the file /etc/ld.so.
S D–Troubleshooting QLogic MPI Troubleshooting The shell strips off the double quotes before handing the arguments to the mpicc script, thus causing the problem. The workaround is to escape the double quotes and white space by using backslashes, so that the shell does not process them. (Also note the single quote (‘) around the -D, since the scripts do an eval rather than directly invoking the underlying compiler.) Use this command instead: $ mpicc -show -DMYDEFINE=\"some\ value\" test.
A D–Troubleshooting QLogic MPI Troubleshooting In the next case, mpi_nxnlatbw.c is compiled with the HP-MPI version of mpicc, and given the name hpmpi-mpi_nxnlatbw, so that it is easy to see which version was used. However, it is run with the QLogic mpirun, which produces errors similar to this: $ /opt/hpmpi/bin/mpicc \ /usr/share/mpich/examples/performance/mpi_nxnlatbw.c -o hpmpi-mpi_nxnlatbw $ mpirun -m ~/host-bbb -np 4 ./hpmpi-mpi_nxnlatbw .
S D–Troubleshooting QLogic MPI Troubleshooting See “Compiler Cannot Find Include, Module, or Library Files” on page D-15, “Compiling on Development Nodes” on page D-16, and “Specifying the Run-time Library Path” on page D-16 for additional information. Process Limitation with ssh MPI jobs that use more than eight processes per node may encounter an ssh throttling mechanism that limits the amount of concurrent per-node connections to 10.
A D–Troubleshooting QLogic MPI Troubleshooting Using MPI.mod Files MPI.mod (or mpi.mod) are the Fortran 90/Fortran 95 mpi modules files. These files contain the Fortran 90/Fortran 95 interface to the platform-specific MPI library. The module file is invoked by ‘USE MPI’ or ‘use mpi’ in your application. If the application has an argument list that does not match what mpi.mod expects, errors such as this can occur: $ mpif90 -O3 -OPT:fast_math -c communicate.
D–Troubleshooting QLogic MPI Troubleshooting S If you know that an argument can accept a data type that the MPI module does not explicitly allow, you can extend the interface for yourself.
A D–Troubleshooting QLogic MPI Troubleshooting This is equally applicable if the module mpi provides only a lower-rank interface and you want to add a higher-rank interface, for example, when the module explicitly provides for 1-D and 2-D integer arrays, but you need to pass a 3-D integer array. Add a higher-rank interface only under the following conditions: The module mpi provides an explicit Fortran 90 style interface for mpi_bcast.
S D–Troubleshooting QLogic MPI Troubleshooting The following message displays after installation: $ mpirun -m ~/tmp/sm -np 2 -mpi_latency 1000 1000000 node-00:1.ipath_update_tid_err: failed: Cannot allocate memory mpi_latency: /fs2/scratch/infinipath-build-1.3/mpi-1.3/mpich/psm/src mq_ips.c:691: mq_ipath_sendcts: Assertion ‘rc == 0’ failed. MPIRUN: Node program unexpectedly quit. Exiting. You can check the ulimit -l on all the nodes by running ipath_checkout.
A D–Troubleshooting QLogic MPI Troubleshooting NOTE: It is important that /dev/shm be writable by all users, or else error messages like the ones in this section can be expected. Also, non-QLogic MPIs that use PSM may be more prone to stale shared memory files when processes are abnormally terminated.
S D–Troubleshooting QLogic MPI Troubleshooting General Error Messages The following message may be generated by ipath_checkout or mpirun: PSM found 0 available contexts on InfiniPath device The most likely cause is that the cluster has processes using all the available PSM contexts. Error Messages Generated by mpirun The following sections describe the mpirun error messages.
A D–Troubleshooting QLogic MPI Troubleshooting The following message indicates a mismatch between the QLogic interconnect hardware in use and the version for which the software was compiled: Number of buffer avail registers is wrong; have n, expected m build mismatch, tidmap has n bits, ts_map m These messages indicate a mismatch between the InfiniPath software and hardware versions. Consult Technical Support after verifying that current drivers and libraries are installed.
D–Troubleshooting QLogic MPI Troubleshooting S The following message usually indicates a node failure or malfunctioning link in the fabric: Couldn’t connect to (LID=::). Time elapsed 00:00:30. Still trying... IP is the MPI rank’s IP address, and are the rank’s lid, port, and subport.
A D–Troubleshooting QLogic MPI Troubleshooting The following indicates an unknown host: $ mpirun -np 2 -m ~/tmp/q mpi_latency 100 100 MPIRUN: Cannot obtain IP address of : Unknown host 15:35_~.1019 The following indicates that there is no route to a valid host: $ mpirun -np 2 -m ~/tmp/q mpi_latency 100 100 ssh: connect to host port 22: No route to host MPIRUN: Some node programs ended prematurely without connecting to mpirun.
S D–Troubleshooting QLogic MPI Troubleshooting The following indicates that one program on one node died: $ mpirun -np 2 -m ~/tmp/q mpi_latency 100000 1000000 MPIRUN: node program unexpectedly quit: Exiting. The quiescence detected message is printed when an MPI job is not making progress. The default timeout is 900 seconds. After this length of time, all the node processes are terminated. This timeout can be extended or disabled with the -quiescence-timeout option in mpirun.
A D–Troubleshooting QLogic MPI Troubleshooting This message occurs when a cable is disconnected, a switch is rebooted, or when there are other problems with the link. The job continues retrying until the quiescence interval expires. See the mpirun -q option for information on quiescence. If a hardware problem occurs, an error similar to this displays: infinipath: [error strings ] Hardware error In this case, the MPI program terminates.
D–Troubleshooting QLogic MPI Troubleshooting S Statistics other than MPI-level statistics are fairly low level; most users will not understand them. Contact QLogic Technical Support for more information. Message statistics are available for transmitted and received messages. In all cases, the MPI rank number responsible for a minimum or maximum value is reported with the relevant value. For application runs of at least three ranks, a median is also available.
E Write Combining Introduction Write combining improves write bandwidth to the QLogic chip by writing multiple words in a single bus transaction (typically 64 bytes). Write combining applies only to x86_64 systems. The x86 Page Attribute Table (PAT) mechanism that allocates Write Combining (WC) mappings for the PIO buffers has been added and is now the default.
S E–Write Combining PAT and Write Combining . . If you do not see any of these messages on your console, but suspect this problem, check the /var/log/messages file. Some systems suppress driver load messages but still output them to the log file. Methods for enabling and disabling the two write combining mechanisms are described in the following sections. There are no conflicts between the two methods.
A E–Write Combining MTRR Mapping and Write Combining For systems with very large amounts of memory (32GB or more), it may also be necessary to adjust the BIOS setting for the PCI hole granularity to 2GB. This setting allows the memory to be mapped with fewer MTRRs, so that there will be one or more unused MTRRs for the InfiniPath driver. Some BIOS’ do not have the MTRR mapping option. It may have a different name, depending on the chipset, vendor, BIOS, or other factors.
E–Write Combining MTRR Mapping and Write Combining S Notes E-4 IB6054601-00 H
F Useful Programs and Files The most useful programs and files for debugging, and commands for common tasks, are presented in the following sections. Many of these programs and files have been discussed elsewhere in the documentation. This information is summarized and repeated here for your convenience. Check Cluster Homogeneity with ipath_checkout Many problems can be attributed to the lack of homogeneity in the cluster environment. Use the following items as a checklist for verifying homogeneity.
S F–Useful Programs and Files Summary and Descriptions of Useful Programs Summary and Descriptions of Useful Programs Useful programs are summarized in Table F-1. Names in blue text are linked to a corresponding section that provides further details. Check the man pages for more information on the programs. Table F-1. Useful Programs Program Name Function chkconfig Checks the configuration state and enables/disables services, including drivers. Can be useful for checking homogeneity.
A F–Useful Programs and Files Summary and Descriptions of Useful Programs Table F-1. Useful Programs (Continued) Program Name Function lsmod Shows status of modules in the Linux kernel. Use to check whether drivers are loaded. modprobe Adds or removes modules from the Linux kernel. mpi_stress An MPI stress test program designed to load up an MPI interconnect with point-to-point messages while optionally checking for data integrity.
F–Useful Programs and Files Summary and Descriptions of Useful Programs S ibhosts This tool determines if all the hosts in your InfiniBand fabric are up and visible to the subnet manager and to each other. It is installed from the openib-diag RPM.
A F–Useful Programs and Files Summary and Descriptions of Useful Programs ibtracert The tool ibtracert determines the path that InfiniBand packets travel between two nodes. It is installed from the openib-diag RPM. The InfiniBand LIDs of the two nodes in this example are determined by using the ipath_control -i command on each node.
S F–Useful Programs and Files Summary and Descriptions of Useful Programs ident The ident strings are available in ib_ipath.ko. Running ident provides driver information similar to the following. For QLogic RPMs, it will look like: $ ident /lib/modules/$(uname-r)/updates/kernel/drivers/ infiniband/hw/ipath/ib_ipath.ko /lib/modules/2.6.16.46-0.12-smp/updates/kernel/drivers/infiniband/ hw/ipath/ib_ipath.ko: $Id: QLogic OFED Release 1.4 $ $Date: Fri Feb 27 16:14:31 PST 2009 $ $Id: QLogic OFED Release 1.
A F–Useful Programs and Files Summary and Descriptions of Useful Programs If you cannot see the problem, send the stdout output to your reseller, along with information on the version of the InfiniPath software you are using. ipath_checkout The ipath_checkout tool is a bash script that verifies that the installation is correct and that all the nodes of the network are functioning and mutually connected by the InfiniPath fabric. It is installed from the infinipath RPM.
S F–Useful Programs and Files Summary and Descriptions of Useful Programs The options available with ipath_checkout are shown in Table F-2. Table F-2. ipath_checkout Options Command Meaning -h, --help These options display help messages describing how a command is used. -v, --verbose -vv, --vverbose -vvv, --vvverbose These options specify three successively higher levels of detail in reporting test results.
A F–Useful Programs and Files Summary and Descriptions of Useful Programs Most of the functionality is accessed via the /sys filesystem. This shell script gathers the same information contained in these files: /sys/class/infiniband/ipath0/device/boardversion /sys/class/infiniband/ipath0/device/status_str /sys/class/infiniband/ipath0/device/driver/version These files are also documented in Table F-4 and Table F-5. Other than the -i option, this script must be run with root permissions.
S F–Useful Programs and Files Summary and Descriptions of Useful Programs ipath_mtrr NOTE: Use ipath_mtrr if you are not using the default PAT mechanism to enable write combining. MTRR is used by the InfiniPath driver to enable write combining to the QLogic on-chip transmit buffers. This option improves write bandwidth to the QLogic chip by writing multiple words in a single bus transaction (typically 64 bytes). This option applies only to x86_64 systems. It can often be set in the BIOS.
A F–Useful Programs and Files Summary and Descriptions of Useful Programs ipath_pkt_test This program is installed from the infinipath RPM. Use ipath_pkt_test to do one of the following: Test the InfiniBand link and bandwidth between two InfiniPath host channel adapters. Using an InfiniBand loopback connector, test the link and bandwidth within a single InfiniPath host channel adapter.
F–Useful Programs and Files Summary and Descriptions of Useful Programs S mpirun mpirun determines whether the program is being run against a QLogic or non-QLogic driver. It is installed from the mpi-frontend RPM. Sample commands and results are shown in the following paragraphs. QLogic-built: $ mpirun -np 2 -m /tmp/id1 -d0x101 mpi_latency 1 0 asus-01:0.ipath_setaffinity: Set CPU affinity to 1, port 0:2:0 (1 active chips) asus-01:0.
A F–Useful Programs and Files Common Tasks and Commands rpm To check the contents of an installed RPM, use these commands: $ rpm -qa infinipath\* mpi-\* $ rpm -q --info infinipath # (etc) The option-q queries. The option --qa queries all. To query a package that has not yet been installed, use the -qpl option. strings Use the strings command to determine the content of and extract text from a binary file. The command strings can also be used.
S F–Useful Programs and Files Common Tasks and Commands Table F-3.
A F–Useful Programs and Files Summary and Descriptions of Useful Files Summary and Descriptions of Useful Files Useful files are summarized in Table F-4. Names in blue text are linked to a corresponding section that provides further details. Table F-4. Useful Files File Name Function boardversion File that shows the version of the chip architecture.
S F–Useful Programs and Files Summary and Descriptions of Useful Files Table F-5 shows the possible contents of the file, with brief explanations of the entries. Table F-5. status_str File Contents File Contents Description Initted The driver has loaded and successfully initialized the IBA6110 or IBA7220 ASIC. Present The IBA6110 or IBA7220 ASIC has been detected (but not initialized unless Initted is also present).
A F–Useful Programs and Files Summary of Configuration Files Table F-6. Status—Other Files (Continued) File Name Contents nguid The number of GUIDs that are used. If nguids == 2 and two chips are discovered, the first chip is assigned the requested GUID (from eeprom, or ipath_sma), and the second chip is assigned GUID+1. serial The serial number of the QHT7140, QLE7140, QLE7240, or QLE7280 adapter. unit A unique number for each card or chip in a system.
S F–Useful Programs and Files Summary of Configuration Files Table F-7. Configuration Files (Continued) Configuration File Name /etc/modprobe.conf Description Specifies options for modules when added or removed by the modprobe command. Also used for creating aliases. The PAT write-combing option is set here. For Red Hat systems. /etc/modprobe.conf.local Specifies options for modules when added or removed by the modprobe command. Also used for creating aliases. The PAT write-combing option is set here.
A F–Useful Programs and Files Summary of Configuration Files Table F-7. Configuration Files (Continued) Sample and Template Files Description qlgc_vnic.cfg.sample Sample VNIC config file. It can be found with the OFED documentation, or in the qlgc_vnictools subdirectory of the QLogicIB_Basic download. It is also installed in /etc/infiniband. /usr/share/doc/initscripts-*/ sysconfig.txt File that explains many of the entries in the configuration files For Red Hat systems.
F–Useful Programs and Files Summary of Configuration Files S Notes F-20 IB6054601-00 H
G Recommended Reading Reference material for further reading is provided in this appendix. References for MPI The MPI Standard specification documents are located at: http://www.mpi-forum.org/docs The MPICH implementation of MPI and its documentation are located at: http://www-unix.mcs.anl.gov/mpi/mpich/ The ROMIO distribution and its documentation are located at: http://www.mcs.anl.
G–Recommended Reading Clusters S Clusters Gropp, William, Ewing Lusk, and Thomas Sterling, Beowulf Cluster Computing with Linux, Second Edition, 2003, MIT Press, ISBN 0-262-69292-9 Networking The Internet Frequently Asked Questions (FAQ) archives contain an extensive Request for Command (RFC) section. Numerous documents on networking and configuration can be found at: http://www.faqs.org/rfcs/index.html Rocks Extensive documentation on installing Rocks and custom Rolls can be found at: http://www.
Glossary A glossary is provided for technical terms used in the documentation. Italicized terms in the definitions are defined in the glossary. If you are viewing this document as a PDF file, the blue terms are linked to the corresponding definition. bandwidth The rate at which data can be transmitted. This represents the capacity of the network connection. Theoretical peak bandwidth is fixed, but the effective bandwidth, the ideal rate, is modified by overhead in hardware and the computer operating system.
Glossary EE – kDAPL EE Stands for End to End EEC Stands for End to End Context fabric The InfiniBand interconnect infrastructure, consisting of a set of host channel adapters (and possibly target channel adapters) connected by switches, such that each end node can directly reach all other nodes. front end node The machine or machines that launch jobs. funneled thread model Only the main (master) thread may execute MPI calls.
A latency The delay inherent in processing network data. In terms of MPI, it is the time required to send a message from one node to another, independent of message size. Latency can be further split into sender and receiver processing overheads, as well as wire and switch overhead. launch node Same as front end node layered driver A driver that does not directly manage any target devices. The layered driver calls another driver’s routines, which in turn manage the target devices.
S Glossary MR – RC MR Stands for Memory Region MTRR Stands for Memory Type Range Registers. Used by the InfiniPath driver to enable write combining to the QLogic on-chip transmit buffers. This improves write bandwidth to the QLogic chip by writing multiple words in a single bus transaction (typically 64). Applies only to x86_64 systems. MTU Stands for Maximum Transfer Unit. The largest packet size that can be transmitted over a given network.
A RDMA Stands for Remote Direct Memory Access. A communications protocol that enables data transmission from the memory of one computer to the memory of another without involving the CPU. The most common form of RDMA is over InfiniBand. RPM Stands for Red Hat Package Manager. A tool for packaging, installing, and managing software for Linux distributions. SDP Stands for Sockets Direct Protocol. An InfiniBand-specific upper layer protocol.
Glossary unmanaged switch – VNIC S unmanaged switch A switch that does not have an active Subnet Manager (SM) Verbs In the InfiniBand specification, Verbs are abstract descriptions of the functions needed to configure, manage, and operate an adapter. The OpenFabrics Alliance has created a User Level Verbs API, which provides support for user level upper layer protocols like MPI, Cluster File Systems, OpenSM, and other user-level utilities.
Index Symbols !!!ERROR!!! Lockable memory less than 4096KB on x nodes error message C-6, D-24 .
QLogic Host Channel Adapter and QLogic OFED Software Users Guide QLogic OFED Version 1.
A QLogic Host Channel Adapter and QLogic OFED Software Users Guide QLogic OFED Version 1.
QLogic Host Channel Adapter and QLogic OFED Software Users Guide QLogic OFED Version 1.
A QLogic Host Channel Adapter and QLogic OFED Software Users Guide QLogic OFED Version 1.
QLogic Host Channel Adapter and QLogic OFED Software Users Guide QLogic OFED Version 1.4 performance improvement over IB Verbs 6-12 Platform (Scali) 6-9 programming documentation G-1 using 3-2 MPI mpi_stress F-3 MPI.mod files, using D-21 NPROCS 5-19 runscript-xqa-14.
A QLogic Host Channel Adapter and QLogic OFED Software Users Guide QLogic OFED Version 1.
QLogic Host Channel Adapter and QLogic OFED Software Users Guide QLogic OFED Version 1.4 PCIe Glossary-4 bus width 4-23 bus width, checking 4-29 Max Payload size 4-23 Max Read Request size 4-23 Performance issues D-9 MVAPICH D-10 settings 4-22 Platform (Scali) MPI 6-9 compiling applications 6-9 installation 6-9 running applications 6-10 setup 6-9 wrapper scripts 6-9 Please install mpirun on ...
A QLogic Host Channel Adapter and QLogic OFED Software Users Guide QLogic OFED Version 1.
QLogic Host Channel Adapter and QLogic OFED Software Users Guide QLogic OFED Version 1.
Corporate Headquarters QLogic Corporation 26650 Aliso Viejo Parkway Aliso Viejo, CA 92656 949.389.6000 Europe Headquarters QLogic (UK) LTD. Quatro House Lyon Way, Frimley Camberley Surrey, GU16 7ER UK www.qlogic.com +44 (0) 1276 804 670 © 2005-2009 QLogic Corporation. Specifications are subject to change without notice. All rights reserved worldwide. QLogic, the QLogic logo, and InfiniPath are registered trademarks of QLogic Corporation.
