Product specifications
Table Of Contents
- Table of Contents
- 1 Introduction
- 2 Feature Overview
- 3 Step-by-Step Cluster Setup and MPI Usage Checklists
- 4 InfiniPath Cluster Setup and Administration
- Introduction
- Installed Layout
- Memory Footprint
- BIOS Settings
- InfiniPath and OpenFabrics Driver Overview
- OpenFabrics Drivers and Services Configuration and Startup
- Other Configuration: Changing the MTU Size
- Managing the InfiniPath Driver
- More Information on Configuring and Loading Drivers
- Performance Settings and Management Tips
- Host Environment Setup for MPI
- Checking Cluster and Software Status
- 5 Using QLogic MPI
- Introduction
- Getting Started with MPI
- QLogic MPI Details
- Use Wrapper Scripts for Compiling and Linking
- Configuring MPI Programs for QLogic MPI
- To Use Another Compiler
- Process Allocation
- mpihosts File Details
- Using mpirun
- Console I/O in MPI Programs
- Environment for Node Programs
- Environment Variables
- Running Multiple Versions of InfiniPath or MPI
- Job Blocking in Case of Temporary InfiniBand Link Failures
- Performance Tuning
- MPD
- QLogic MPI and Hybrid MPI/OpenMP Applications
- Debugging MPI Programs
- QLogic MPI Limitations
- 6 Using Other MPIs
- A mpirun Options Summary
- B Benchmark Programs
- C Integration with a Batch Queuing System
- D Troubleshooting
- Using LEDs to Check the State of the Adapter
- BIOS Settings
- Kernel and Initialization Issues
- OpenFabrics and InfiniPath Issues
- Stop OpenSM Before Stopping/Restarting InfiniPath
- Manual Shutdown or Restart May Hang if NFS in Use
- Load and Configure IPoIB Before Loading SDP
- Set $IBPATH for OpenFabrics Scripts
- ifconfig Does Not Display Hardware Address Properly on RHEL4
- SDP Module Not Loading
- ibsrpdm Command Hangs when Two Host Channel Adapters are Installed but Only Unit 1 is Connected to the Switch
- Outdated ipath_ether Configuration Setup Generates Error
- System Administration Troubleshooting
- Performance Issues
- QLogic MPI Troubleshooting
- Mixed Releases of MPI RPMs
- Missing mpirun Executable
- Resolving Hostname with Multi-Homed Head Node
- Cross-Compilation Issues
- Compiler/Linker Mismatch
- Compiler Cannot Find Include, Module, or Library Files
- Problem with Shell Special Characters and Wrapper Scripts
- Run Time Errors with Different MPI Implementations
- Process Limitation with ssh
- Number of Processes Exceeds ulimit for Number of Open Files
- Using MPI.mod Files
- Extending MPI Modules
- Lock Enough Memory on Nodes When Using a Batch Queuing System
- Error Creating Shared Memory Object
- gdb Gets SIG32 Signal Under mpirun -debug with the PSM Receive Progress Thread Enabled
- General Error Messages
- Error Messages Generated by mpirun
- MPI Stats
- E Write Combining
- F Useful Programs and Files
- G Recommended Reading
- Glossary
- Index
B–Benchmark Programs
Benchmark 2: Measuring MPI Bandwidth Between Two Nodes
IB6054601-00 H B-3
A
Benchmark 2: Measuring MPI Bandwidth
Between Two Nodes
The osu_bw benchmark measures the maximum rate at which you can pump
data between two nodes. This benchmark also uses a ping-pong mechanism,
similar to the osu_latency code, except in this case, the originator of the
messages pumps a number of them (64 in the installed version) in succession
using the non-blocking MPI_I send function, while the receiving node consumes
them as quickly as it can using the non-blocking MPI_Irecv function, and then
returns a zero-length acknowledgement when all of the sent data has been
received.
You can run this program by typing:
$ mpirun -np 2 -ppn 1 -m mpihosts osu_bw
Typical output might look like:
# OSU MPI Bandwidth Test (Version 2.0)
# Size Bandwidth (MB/s)
1 3.549325
2 7.110873
4 14.253841
8 28.537989
16 42.613030
32 81.144290
64 177.331433
128 348.122982
256 643.742171
512 1055.355552
1024 1566.702234
2048 1807.872057
4096 1865.128035
8192 1891.649180
16384 1898.205188
32768 1888.039542
65536 1931.339589
131072 1942.417733
262144 1950.374843
524288 1954.286981
1048576 1956.301287
2097152 1957.351171
4194304 1957.810999
The increase in measured bandwidth with the messages’ size is because the
latency’s contribution to the measured time interval becomes relatively smaller.