Product specifications
Table Of Contents
- Table of Contents
- 1 Introduction
- 2 Feature Overview
- 3 Step-by-Step Cluster Setup and MPI Usage Checklists
- 4 InfiniPath Cluster Setup and Administration
- Introduction
- Installed Layout
- Memory Footprint
- BIOS Settings
- InfiniPath and OpenFabrics Driver Overview
- OpenFabrics Drivers and Services Configuration and Startup
- Other Configuration: Changing the MTU Size
- Managing the InfiniPath Driver
- More Information on Configuring and Loading Drivers
- Performance Settings and Management Tips
- Host Environment Setup for MPI
- Checking Cluster and Software Status
- 5 Using QLogic MPI
- Introduction
- Getting Started with MPI
- QLogic MPI Details
- Use Wrapper Scripts for Compiling and Linking
- Configuring MPI Programs for QLogic MPI
- To Use Another Compiler
- Process Allocation
- mpihosts File Details
- Using mpirun
- Console I/O in MPI Programs
- Environment for Node Programs
- Environment Variables
- Running Multiple Versions of InfiniPath or MPI
- Job Blocking in Case of Temporary InfiniBand Link Failures
- Performance Tuning
- MPD
- QLogic MPI and Hybrid MPI/OpenMP Applications
- Debugging MPI Programs
- QLogic MPI Limitations
- 6 Using Other MPIs
- A mpirun Options Summary
- B Benchmark Programs
- C Integration with a Batch Queuing System
- D Troubleshooting
- Using LEDs to Check the State of the Adapter
- BIOS Settings
- Kernel and Initialization Issues
- OpenFabrics and InfiniPath Issues
- Stop OpenSM Before Stopping/Restarting InfiniPath
- Manual Shutdown or Restart May Hang if NFS in Use
- Load and Configure IPoIB Before Loading SDP
- Set $IBPATH for OpenFabrics Scripts
- ifconfig Does Not Display Hardware Address Properly on RHEL4
- SDP Module Not Loading
- ibsrpdm Command Hangs when Two Host Channel Adapters are Installed but Only Unit 1 is Connected to the Switch
- Outdated ipath_ether Configuration Setup Generates Error
- System Administration Troubleshooting
- Performance Issues
- QLogic MPI Troubleshooting
- Mixed Releases of MPI RPMs
- Missing mpirun Executable
- Resolving Hostname with Multi-Homed Head Node
- Cross-Compilation Issues
- Compiler/Linker Mismatch
- Compiler Cannot Find Include, Module, or Library Files
- Problem with Shell Special Characters and Wrapper Scripts
- Run Time Errors with Different MPI Implementations
- Process Limitation with ssh
- Number of Processes Exceeds ulimit for Number of Open Files
- Using MPI.mod Files
- Extending MPI Modules
- Lock Enough Memory on Nodes When Using a Batch Queuing System
- Error Creating Shared Memory Object
- gdb Gets SIG32 Signal Under mpirun -debug with the PSM Receive Progress Thread Enabled
- General Error Messages
- Error Messages Generated by mpirun
- MPI Stats
- E Write Combining
- F Useful Programs and Files
- G Recommended Reading
- Glossary
- Index
D–Troubleshooting
OpenFabrics and InfiniPath Issues
D-8 IB6054601-00 H
S
SDP Module Not Loading
If the settings for debug level and the zero copy threshold from InfiniPath
release 2.0 are present in the release 2.2 /etc/modprobe.conf file (RHEL) or
/etc/modprobe.conf.local (SLES) file, the SDP module may not load:
options ib_sdp sdp_debug_level=4
sdp_zcopy_thrsh_src_default=10000000
To solve the problem, remove this line.
ibsrpdm Command Hangs when Two Host Channel Adapters
are Installed but Only Unit 1 is Connected to the Switch
If multiple host channel adapters (unit 0 and unit 1) are installed and only unit 1 is
connected to the switch, the ibsrpdm command (to set up an SRP target) can
hang. If unit 0 is connected and unit 1 is disconnected, the problem does not
occur.
When only unit 1 is connected to the switch, use the
-d
option with
ibsrpdm
.
Then,
using the output from the
ibsrpdm
command, echo the new target information into
/sys/class/infiniband_srp/srp-ipath1-1/add_target
.
For example:
# ibsrpdm -d /dev/infiniband/umad1 -c
# echo \
id_ext=21000001ff040bf6,ioc_guid=21000001ff040bf6,dgid=fe800000000
0000021000001ff040bf6,pkey=ffff,service_id=f60b04ff01000021 >
/sys/class/infiniband_srp/srp-ipath1-1/add_target
Outdated ipath_ether Configuration Setup Generates Error
Ethernet emulation (ipath_ether) has been removed in this release, and, as a
result, an error may be seen if the user still has an alias set previously by
modprobe.conf (for example, alias eth2 ipath_ether).
When ifconfig or ifup are run, the error will look similar to this (assuming
ipath_ether was used for eth2):
eth2: error fetching interface information: Device not found
To prevent the error message, remove the following files (assuming
ipath_ether was used for eth2):
/etc/sysconfig/network-scripts/ifcfg-eth2 (for RHEL)
/etc/sysconfig/network/ifcfg-eth-eth2 (for SLES)
QLogic recommends using the IP over InfiniBand protocol (IPoIB-CM), included in
the standard OpenFabrics software releases, as a replacement for
ipath_ether.