Sun™ Dual Port 4x IB Host Channel Adapter PCI-X Installation and User’s Guide Sun Microsystems, Inc. www.sun.com Part No. 819-1280-10 September 2005, Revision A Submit comments about this document at: http://www.sun.
Copyright 2005 Sun Microsystems, Inc., 4150 Network Circle, Santa Clara, California 95054, U.S.A. All rights reserved. Sun Microsystems, Inc. has intellectual property rights relating to technology that is described in this document. In particular, and without limitation, these intellectual property rights may include one or more of the U.S. patents listed at http://www.sun.com/patents and one or more additional patents or pending patent applications in the U.S. and in other countries.
Contents Regulatory Compliance Statements Declaration of Conformity Preface xi xv xvii How This Book Is Organized Using UNIX Commands Shell Prompts xvii xviii xviii Typographic Conventions Related Documentation xix xix Documentation, Support, and Training Third-Party Web Sites xx Contacting Sun Technical Support Sun Welcomes Your Comments 1.
LED Displays 3 Hardware, Software, and Cable Requirements 2. Adapter Installation 7 IB-HCA Bus and Slot Selection 3. ▼ Installing the Adapter ▼ Verifying the Installation ▼ 9 11 11 Configuring IPoIB 11 Managing IB Devices for IPoIB 4. 13 InfiniBand Software Overview InfiniBand Software 7 7 Configuring IP Over InfiniBand IPoIB Configuration 15 15 User Direct Access Programming Library IP over InfiniBand Driver 17 InfiniBand Management Framework Tavor HCA Driver 18 18 18 A.
Other Useful Utilities 26 cfgadm snoop 27 netstat kstat Index 26 27 27 29 Contents v
vi Sun Dual Port 4x IB Host Channel Adapter PCI-X Installation and User’s Guide • September 2005
Figures FIGURE 1-1 Sun Dual Port 4x IB Host Channel Adapter PCI-X Card FIGURE 1-2 LED Layout 4 FIGURE 2-1 Typical Adapter-to-Switch Configuration FIGURE 4-1 InfiniBand Software Stack FIGURE A-1 Sun 4x-to-12x InfiniBand Cable Pin Assignments 2 9 16 21 vii
viii Sun Dual Port 4x IB Host Channel Adapter PCI-X Installation and User’s Guide • September 2005
Tables TABLE 1-1 IB-HCA LEDs 3 TABLE 1-2 Hardware, Software, and Cable Requirements TABLE B-1 SunVTS Documentation 5 24 ix
x Sun Dual Port 4x IB Host Channel Adapter PCI-X Installation and User’s Guide • September 2005
Regulatory Compliance Statements Your Sun product is marked to indicate its compliance class: • • • • Federal Communications Commission (FCC) — USA Industry Canada Equipment Standard for Digital Equipment (ICES-003) — Canada Voluntary Control Council for Interference (VCCI) — Japan Bureau of Standards Metrology and Inspection (BSMI) — Taiwan Please read the appropriate section that corresponds to the marking on your Sun product before attempting to install the product.
ICES-003 Class B Notice - Avis NMB-003, Classe B This Class B digital apparatus complies with Canadian ICES-003. Cet appareil numérique de la classe B est conforme à la norme NMB-003 du Canada. BSMI Class A Notice The following statement is applicable to products shipped to Taiwan and marked as Class A on the product compliance label.
CCC Class A Notice The following statement is applicable to products shipped to China and marked with “Class A” on the product’s compliance label.
xiv Sun Dual Port 4x IB Host Channel Adapter PCI-X Installation and User’s Guide • September 2005
Declaration of Conformity Compliance Model Number: Product Family Name: IB-HCA Sun Dual Port 4x IB Host Channel Adapter PCI-X EMC USA—FCC Class A This equipment complies with Part 15 of the FCC Rules. Operation is subject to the following two conditions: 1. This equipment may not cause harmful interference. 2. This equipment must accept any interference that may cause undesired operation.
xvi Sun Dual Port 4x IB Host Channel Adapter PCI-X Installation and User’s Guide • September 2005
Preface The Sun™ Dual Port 4x IB Host Channel Adapter PCI-X Installation and User’s Guide provides installation instructions for the Sun Dual Port 4x IB Host Channel Adapter PCI-X card. This manual includes an overview of the adapter, supported platforms, cable information, and related documentation. These instructions are designed for system administrators with experience installing network hardware and software.
Using UNIX Commands This document might not contain information about basic UNIX® commands and procedures such as shutting down the system, booting the system, and configuring devices. Refer to the following for this information: ■ Software documentation that you received with your system ■ Solaris™ Operating System documentation, which is at: http://docs.sun.
Typographic Conventions Typeface 1 Meaning Examples AaBbCc123 The names of commands, files, and directories; on-screen computer output Edit your .login file. Use ls -a to list all files. % You have mail. AaBbCc123 What you type, when contrasted with on-screen computer output % su Password: AaBbCc123 Book titles, new words or terms, words to be emphasized Read Chapter 6 in the User’s Guide. These are called class options. You must be superuser to do this.
Documentation, Support, and Training Sun Function URL Description Documentation http://www.sun.com/documentation/ Download PDF and HTML documents, and order printed documents Support and Training http://www.sun.com/supportraining/ Obtain technical support, download patches, and learn about Sun courses Third-Party Web Sites Sun is not responsible for the availability of third-party web sites mentioned in this document.
Sun Welcomes Your Comments Sun is interested in improving its documentation and welcomes your comments and suggestions. You can submit your comments by going to: http://www.sun.
xxii Sun Dual Port 4x IB Host Channel Adapter PCI-X Installation and User’s Guide • September 2005
CHAPTER 1 Product Overview This chapter provides an overview of the Sun Dual Port 4x IB Host Channel Adapter PCI-X card and includes the following sections: ■ ■ ■ ■ ■ “Product Features” on page 1 “Hardware Description” on page 1 “I/O Interfaces” on page 3 “LED Displays” on page 3 “Hardware, Software, and Cable Requirements” on page 5 Product Features Sun Dual Port 4x IB Host Channel Adapter PCI-X card features include: ■ ■ ■ ■ ■ ■ Dual 10 Gb/sec (4x) InfiniBand ports Integrated serializer/deserializer
The board characteristics are: ■ ■ ■ ■ ■ ■ ■ Power: approx. 12 watts total Low profile PCI form-factor card (approx. 2.5 in. by approx. 6.7 in.) Requires 4x InfiniBand cables 3.3V PCI-X or PCI slot (Note that the IB-HCA can not be plugged into a 5V connector.) Link status indication LED Two 4x InfiniBand (IB) ports Two faceplate sizes: standard and low profile The standard PCI/PCI-X interface is used for data traffic, management traffic, and to generate and receive inbound configuration packets.
I/O Interfaces The IB-HCA includes the following interfaces: ■ ■ Two 4x InfiniBand copper connectors PCI-X 64-bit 133 MHz edge connector InfiniBand Interface The IB-HCA provides two 4x InfiniBand v1.1 connector (or ports) for external copper cables and is IBTA specification 1.1-compliant. PCI-X Interface The PCI-X bus is a PCI-X version 2.2-compliant 64-bit 133 MHz interface. The adapter can be either a master initiating the PCI bus operations or a slave responding to PCI bus operations.
Physical link (Green LED) FIGURE 1-2 4 Logical link (Amber LED) LED Layout Sun Dual Port 4x IB Host Channel Adapter PCI-X Installation and User’s Guide • September 2005
Hardware, Software, and Cable Requirements The hardware, software, and cable requirements for the Sun Dual Port 4x IB Host Channel Adapter PCI-X card are listed in TABLE 1-2. TABLE 1-2 Hardware, Software, and Cable Requirements Operating System Servers Solaris 10 SPARC Sun Fire™ V240, V440, V490, V890, V1280, E2900, E4900, E6900, E20K, E25K Solaris 10 X86 Sun Fire V20z, V40z InfiniBand Switch X3152A Sun™ InfiniBand Switch P9 Diagnostics Version SunVTS Version 6.
6 Sun Dual Port 4x IB Host Channel Adapter PCI-X Installation and User’s Guide • September 2005
CHAPTER 2 Adapter Installation This chapter describes how to install the adapter in your system and verify that it has been installed correctly. IB-HCA Bus and Slot Selection For optimal performance, the IB-HCA card should be installed in the fastest available bus, preferably a 133 MHz PCI-X bus. But the IB-HCA can operate on the slower PCI-X and PCI buses. The IB-HCA should also be installed on its own PCI leaf.
2. Remove the cover from the unit to access the card slots and connectors. 3. Select an available 3.3 Volt PCI-X or PCI slot and remove the blank front panel. If you are replacing an existing card, remove the card. 4. Install the IB-HCA card into the slot, pushing the card’s edge connector into the connector on the chassis. Ensure that the front plate on the IB-HCA card mounts flush with the chassis panel opening. 5. If applicable, install the screw in front plate to secure the IB-HCA card into the chassis.
This completes the hardware installation. Proceed to the verification instructions in “Verifying the Installation” on page 9. Sun 9-Port InfiniBand switch Sun 9-Port InfiniBand switch Sun server FIGURE 2-1 ▼ Typical Adapter-to-Switch Configuration Verifying the Installation Before you can verify the installation, you must install the adapter in the chassis, power the server, and cable it to an operational InfiniBand switch.
2. Check that the amber LED is illuminated for each port that is connected to the switch. If the amber LED is not on, refer to Appendix B, “Diagnostic Software and Troubleshooting” on page 23 for more information.
CHAPTER 3 Configuring IP Over InfiniBand This chapter contains the following sections: ■ ■ “IPoIB Configuration” on page 11 “Managing IB Devices for IPoIB” on page 13 IPoIB Configuration IP Over InfiniBand (IPoIB) is automatically configured when the system is booted. The following procedure can be used when making changes to your configuration. See System Administration Guide: Basic Administration (817-1985) for Solaris setup information. ▼ Configuring IPoIB 1. Confirm that the IB-HCA ports are up.
For example: Dec 10 16:41:05 hostname pcplusmp: [ID 637496 kern.info] pcplusmp: pci15b3,5a44 (tavor) instance 0 vector 0x1c ioapic 0x4 intin 0x0 is bound to cpu 1 Dec 10 16:41:05 hostname pci_pci: [ID 370704 kern.info] PCI-device: pci15b3,5a44@0, tavor0 Dec 10 16:41:05 hostname genunix: [ID 936769 kern.info] tavor0 is /pci@0,0/pci1022,7450@b/pci15b3,5a46@1/pci15b3,5a44@0 Dec 10 16:41:05 hostname tavor: [ID 548581 kern.info] tavor0: FW ver: 0003.0002.
5. Verify that the devices are plumbed. To do this, type the command: # ifconfig -a The output includes information on the IB devices. 6. Check that data can be send over the InfiniBand link. Use the ping command to send ICMP (ICMP6) ECHO_REQUEST packets and receive ICMP (ICMP6) ECHO_RESPONSE packets between InfiniBand hosts. For example: # ping host-ip-address where host-ip-address is the IP address of another IB host. See ping(1M) for details.
14 Sun Dual Port 4x IB Host Channel Adapter PCI-X Installation and User’s Guide • September 2005
CHAPTER 4 InfiniBand Software Overview This chapter includes the following topics: ■ “InfiniBand Software” on page 15 ■ “Related Software Documentation” on page 18 InfiniBand Software The InfiniBand software is included in the Solaris 10 Operating System.
uDAPL IPoIB IBTF IBMF Softw ar e Solaris 10 OS IB-HCA Port 1 FIGURE 4-1 F/W Port 2 Hardware Tavor HCA Driver InfiniBand Software Stack User Direct Access Programming Library This library supports service providers written according to the uDAPL 1.2 specification. Sun provides default service provider modules to support the Tavor HCA driver. IP over InfiniBand Driver The IPoIB driver implements the IETF IPoIB protocol and provides IPoIB services for all IB-HCA ports present in the system.
InfiniBand Transport Framework IBTF is the Solaris implementation of the InfiniBand Transport Layer (IBTL). IBTF consists of InfiniBand Device Manager (IBDM), InfiniBand Communications Manager (IBCM) and an InfiniBand Nexus driver. The Nexus driver along with IBDM creates and manages InfiniBand devices.
Related Software Documentation Manuals ■ System Administration Guide: Devices and Files System (817-5093) For information on InfiniBand devices, see the System Administration Guide: Devices and Files System (817-5093) ■ System Administration Guide: Network Services (816-4555) For general information on the InfiniBand network, see the System Administration Guide: Network Services (816-4555). ■ Sun Cluster 3.
APPENDIX A InfiniBand Cable Pin Assignments FIGURE A-1 shows the InfiniBand cable connectors and pin assignments.
P1 (12X) P2 (4X) G1 S1 S2 S47 G9 S16 S15 S2 S1 G1 S48 G25 PAIR 1 PAIR 2 PAIR 3 PAIR 4 PAIR 5 PAIR 6 PAIR 7 PAIR 8 FIGURE A-1 20 G S1 S2 G S3 S4 G S5 S6 G S7 S8 G S41 S42 G S43 S44 G S45 S46 G S47 S48 G SHELL G1 S16 S15 G2 S14 S13 G3 S12 S11 G4 S10 S9 G5 S8 S7 G6 S6 S5 G7 S4 S3 G8 S2 S1 G9 SHELL PAIR 1 PAIR 2 PAIR 3 PAIR 4 PAIR 5 PAIR 6 PAIR 7 PAIR 8 Sun 4x-to-12x InfiniBand Cable Pin Assignments Sun Dual Port 4x IB Host Channel Adapter PCI-X Installation and User’s Guide • Sep
APPENDIX B Diagnostic Software and Troubleshooting This appendix provides an overview of the SunVTS™ diagnostic application and troubleshooting tips for checking the adapter. There is also a section outlining some common troubleshooting issues.
TABLE B-1 SunVTS Documentation Title Descriptions SunVTS 6.0 PS1 Documentation Supplement (819-1804) Describes the new SunVTS features and tests, including the ibhcatest. SunVTS 6.0 User’s Guide (817-7664) Describes the SunVTS diagnostic environment. SunVTS 6.0 Test Reference Manual (817-7665) Describes each SunVTS test and describes the various test options and command-line arguments. SunVTS 6.0 Quick Reference Card (817-7686) Provides an overview of the user interface.
▼ Using the ibhcatest 1. Ensure that the SunVTS software and the Tavor driver are installed on your system. To do this, type the following command: # pkginfo SUNWvts SUNWvtsx SUNWtavor If a SunVTS software package is not installed, refer to the SunVTS User’s Guide for installation instruction. If the SUNWtavor package is not installed, check your Solaris Operating System documentation for software package information. 2.
If an InfiniBand software package is not installed, check your Solaris Operating System documentation for software package information. ■ Check the system log or console for Tavor driver error messages. See tavor(7D) for error messages and descriptions. When the driver is attached to a port on the adapter, the following message is sent. tavorn: port m up (link width 4x). In the message, n is the instance of the Tavor device number and m is the port number on the adapter.
snoop The snoop program captures and inspects network packets. See the snoop(1M) man page for details. For example: # snoop -d ibd1 Using device /dev/ibd1 (promiscuous mode) ib-1-167 -> * ARP C Who is 199.1.1.168, ib-1-168 ? ib-1-168 -> ib-1-167 ARP R 199.1.1.168, ib-1-168 is 0:2:4:7:0:0:0:0:a:4:7c:4f:0:2:c9:2:0:0:55:91 ib-1-167 -> ib-1-168 ICMP Echo request (ID: 35608 Sequence number: 0) ib-1-168 -> ib-1-167 ICMP Echo reply (ID: 35608 Sequence number: 0) netstat netstat shows network status.
28 Sun Dual Port 4x IB Host Channel Adapter PCI-X Installation and User’s Guide • September 2005
Index A adapter description, 1 features, 1 installation, 7 layout, 2 C cable requirements, 5 cables connectors, 19 part numbers, 5 pin assignments, 19 D diagnostics, ibhcatest, 24 documentation man pages, 18 related, xix, 18 SunVTS, 23 InfiniBand software overview, 15 InfiniBand Transport Layer, 17 installation, adapter, 7 interface InfiniBand, 3 PCI-X, 3 IPoIB, 15 configuring, 11 driver, 16 managing devices, 13 L LEDs descriptions, 3 layout, 4 verification, 10 O operating systems, 5 R I I/O interfac
versions, 5 T Tavor driver, 17 troubleshooting IB-HCA, 25 tasks, 25 utilities, 26 U uDAPL, 15, 16 utilities, 26 cfgadm, 26 kstat, 27 netstat, 27 snoop, 27 30 Sun Dual Port 4x IB Host Channel Adapter PCI-X Installation and User’s Guide • September 2005
