User guide

IB0054606-02 A ix
OFED+ Host Software Release 1.5.4
User Guide
Configuring SRP for Native IB Storage . . . . . . . . . . . . . . . . . . . . . . . . B-21
Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B-23
Additional Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B-24
Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B-24
OFED SRP Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B-24
C Integration with a Batch Queuing System
Clean Termination of MPI Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C-1
Clean-up PSM Shared Memory Files. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C-2
D Troubleshooting
Using LEDs to Check the State of the Adapter . . . . . . . . . . . . . . . . . . . . . . D-1
BIOS Settings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D-2
Kernel and Initialization Issues. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D-2
Driver Load Fails Due to Unsupported Kernel. . . . . . . . . . . . . . . . . . . D-3
Rebuild or Reinstall Drivers if Different Kernel Installed . . . . . . . . . . . D-3
InfiniPath Interrupts Not Working. . . . . . . . . . . . . . . . . . . . . . . . . . . . . D-3
OpenFabrics Load Errors if ib_qib Driver Load Fails . . . . . . . . . . . . D-4
InfiniPath ib_qib Initialization Failure. . . . . . . . . . . . . . . . . . . . . . . . D-5
MPI Job Failures Due to Initialization Problems . . . . . . . . . . . . . . . . . D-6
OpenFabrics and InfiniPath Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D-6
Stop Infinipath Services Before Stopping/Restarting InfiniPath . . . . . . D-6
Manual Shutdown or Restart May Hang if NFS in Use . . . . . . . . . . . . D-7
Load and Configure IPoIB Before Loading SDP . . . . . . . . . . . . . . . . . D-7
Set $IBPATH for OpenFabrics Scripts . . . . . . . . . . . . . . . . . . . . . . . . D-7
SDP Module Not Loading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D-7
ibsrpdm Command Hangs when Two Host Channel
Adapters are Installed but Only Unit 1 is Connected
to the Switch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D-8
Outdated ipath_ether Configuration Setup Generates Error . . . . . . . . D-8
System Administration Troubleshooting. . . . . . . . . . . . . . . . . . . . . . . . . . . . D-8
Broken Intermediate Link. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D-9
Performance Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D-9
Large Message Receive Side Bandwidth Varies with
Socket Affinity on Opteron Systems . . . . . . . . . . . . . . . . . . . . . . . . . D-9
Erratic Performance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D-10
Method 1. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D-10
Method 2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D-10
Immediately change the processor affinity of an IRQ. . . . . . . . . D-11
Performance Warning if ib_qib Shares Interrupts with eth0 . . . . . D-12