SGI® Altix® Systems Dual-Port Gigabit Ethernet PCI Express Board User’s Guide 007-5472-001
COPYRIGHT © 2008, Silicon Graphics, Inc. All rights reserved; provided portions may be copyright in third parties, as indicated elsewhere herein. No permission is granted to copy, distribute, or create derivative works from the contents of this electronic documentation in any manner, in whole or in part, without the prior written permission of Silicon Graphics, Inc.
Canadian Department of Communications Statement This digital apparatus does not exceed the Class A limits for radio noise emissions from digital apparatus as set out in the Radio Interference Regulations of the Canadian Department of Communications.
Record of Revision Version Description 001 March 2008 Initial release.
Contents Figures . . . . . . . . . . . . . . . . . . . . . . . . . . ix Tables . . . . . . . . . . . . . . . . . . . . . . . . . . xi About this book . . . . . . . Important Information . . . . . Chapter Descriptions . . . . . . Related Publications . . . . . . SGI Altix Hardware Documentation Conventions . . . . . . . . Product Support . . . . . . . Reader Comments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Contents 3. Operating the Gigabit Ethernet PCI Express Board Verifying Functionality . . . . . . . . Using LEDs to Determine Board Functionality Verifying Board Recognition . . . . . . Verifying Board Configuration and Enabling . Resetting the Board. . . . . . . . . . Configuration Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 . 15 . 16 .
Figures Figure 1-1 Figure 1-2 Figure 2-1 Figure 2-2 007-5472-001 Fiber-Optic Gigabit Ethernet PCI Express Board Copper Gigabit Ethernet Board . . . . . . . . . . . . . . . . . Connecting the Fiber-Optic Cable Connecting the Copper Cable . . . . . . . . . . . . . 13 . 14 . . . . . . . .
Tables Table 1-1 Table 1-2 Table 1-3 Table 1-4 Table 3-1 Table 3-2 Table A-1 Table A-2 Table A-3 Table A-4 Table A-5 Table A-6 007-5472-001 SGI 62.5-Micron Cable Options for Fiber-Optic Gigabit Ethernet. Fiber-Optic Operating Range, 1000-BASE-SX Standard . . . . . 8 8 SGI Twisted Pair Cable . . . . . . . . . . . . . Configuration Limits . . . . . . . . . . . . . . LEDs on the Fiber-optic Gigabit Ethernet PCI Express Board . . . LEDs on the Copper Gigabit Ethernet PCI Express Board . . . .
About this book This guide describes the two versions of the dual-port Gigabit Ethernet PCI Express board, shows you how to connect the boards to an Ethernet network, and explains how to operate the boards. You can use the dual-port Gigabit Ethernet PCI Express board to replace the built-in Ethernet network adapter in your system, or use it in addition to your current adapter. This guide is written for users of the dual-port Gigabit Ethernet PCI Express board.
About this book Chapter Descriptions This guide contains the following chapters: • Chapter 1, “Gigabit Ethernet PCI Express Board Features and Capabilities,” summarizes board features, lists the protocols and interfaces with which the board is compatible, and gives board configuration limits for various systems. • Chapter 2, “Connecting the Gigabit Ethernet PCI Express Board to a Network,” shows you how to connect the Gigabit Ethernet board to your network.
About this book • SGI Altix 450 System User’s Guide Provides an overview of the Altix 450 system components, and it describes how to set up and operate this system • SGI Altix 4700 User’s Guide This guide provides an overview of the architecture and descriptions of the major components that compose the SGI Altix 4700 family of servers.
About this book • You can also view man pages by typing man
on a command line. For example, to display the man page for the apropos command, type the following on a command line: man apropos Important system configuration files and commands are documented on man pages. References in the documentation to these pages include the name of the command and the section number in which the command is found.About this book Product Support SGI provides a comprehensive product support and maintenance program for its products: • If you are in North America, contact the Technical Assistance Center at +1 800 800 4SGI or contact your authorized service provider. • If you are outside North America, contact the SGI subsidiary or authorized distributor in your country. Reader Comments If you have comments about the technical accuracy, content, or organization of this document, contact SGI.
Chapter 1 1. Gigabit Ethernet PCI Express Board Features and Capabilities Gigabit Ethernet is technology that allows computer systems to communicate at speeds up to 1 gigabit per second (Gbps). This board is supported in the following systems: • SGI Altix XE210 • SGI Altix XE240 • SGI Altix XE250 • SGI Altix XE310 • SGI Altix XE320 • SGI Altix ICE 8200 • SGI Altix 450 • SGI Altix 4700 Note: Not every Altix system can immediately accept a PCIe card without optional hardware to support it.
1: Gigabit Ethernet PCI Express Board Features and Capabilities Note: On SGI Altix 450 or 4700 systems, if both ports are run concurrently, one of the ports will have performance degraded by approximately 40 percent. The Broadcom 5715 GbE controller implementation of a PCIe to PCI bridge is only capable of 40-bit wide addressing. Ports run individually achieve expected throughput of approximately 940 Mb/s.
Board Features • Hardware acceleration that can offload tasks from the host processor. The controllers can offload TCP/UDP/IP checksum calculations and TCP segmentation. • Server class reliability, availability and performance features: – Link Aggregation and Load Balancing: Switch dependent: 802.3ad (LACP), Generic Trunking ( GEC / FEC) Switch and NIC Independent. Failover • Priority queuing - 802.1p layer 2 priority encoding. • Virtual LANs - 802.1q VLAN tagging. • Jumbo Frame (9KB). • 802.
1: Gigabit Ethernet PCI Express Board Features and Capabilities Fiber-optic connector Figure 1-1 4 Fiber-Optic Gigabit Ethernet PCI Express Board 007-5472-001
Board Features Copper Board Features The technical specifications for the copper Gigabit Ethernet PCI Express board are from a technical specification page available at: http://www.silicom-usa.com/pgx.php?p2=147 The copper Gigabit Ethernet PCI Express board includes these features: • Independently Fiber Gigabit Ethernet channel/s support Gigabit Ethernet 1000Base-SX. • Small Form Factor (SFF) LC Connectors.
1: Gigabit Ethernet PCI Express Board Features and Capabilities • Jumbo Frame (9KB). • 802.x flow control. • Boot ROM embedded or optional can be used for Boot ROM applications. • PCI Power Management Interface. (v1.1) • Statistics for SNMP MIB II, Ethernet like MIB, and Ethernet MIB (802.3z, Clause 30) • LEDs indicators for link/Activity/Speed status. Note: The hardware MAC addresses for the two ports are lowest at the top of the card and highest at the bottom of the card.
Cabling Figure 1-2 shows the quad-port Copper Gigabit Ethernet board.
Fiber-Optic Board Cabling The fiber-optic Gigabit Ethernet board is connected to the network using fiber-optic cable. The cable, which is not included in the shipment, must be a 50-micron or 62.5-micron multimode duplex cable with an LC connector. Table 1-1 lists SGI fiber-optic cables. Table 1-1 SGI 62.5-Micron Cable Options for Fiber-Optic Gigabit Ethernet Length Marketing Code 3 m (9.8 ft) X-F-OPT-3M 10 m (39.
Configuration Limits Copper Board Cabling The Copper Gigabit Ethernet board is implemented using twisted pair cable. The cable, which is not included in the shipment, must be Category-5 cable plant (4-pair) with an RJ45 UTP connector at each end. Table 1-3 lists the SGI twisted pair cables. The operating range for 1000-Base-T is up to 100 m (328 ft).
1: Gigabit Ethernet PCI Express Board Features and Capabilities Table 1-4 10 Configuration Limits System Maximum Per Bus Maximum Number of Boards Altix XE320 1 1 Altix 450 1 16 Altix ICE 8200 1 4 Altix 4700 1 16 007-5472-001
Chapter 2 2. Connecting the Gigabit Ethernet PCI Express Board to a Network This chapter shows you how to connect the fiber-optic Gigabit Ethernet PCI Express board or the copper Gigabit Ethernet PCI Express board to a network, and how to configure your system for the board.
2: Connecting the Gigabit Ethernet PCI Express Board to a Network • Altix ICE 8400: See the instructions for installing a PCI-E card in the user’s or owner’s guide that came with your system. • Altix 450: See the instructions for installing a PCI-E card in the user’s or owner’s guide that came with your system. • Altix 4700: Your Gigabit Ethernet board must be installed by an SGI certified service provider.
Connecting to the Network Fiber-optic connector Figure 2-1 Connecting the Fiber-Optic Cable Connecting the Copper Board To connect your copper Gigabit Ethernet PCI Express board to a network, insert the RJ45 connector on one end of the copper cable into the Gigabit Ethernet board, as shown in Figure 2-2.
2: Connecting the Gigabit Ethernet PCI Express Board to a Network RJ45 connector Figure 2-2 14 Connecting the Copper Cable 007-5472-001
Chapter 3 3. Operating the Gigabit Ethernet PCI Express Board This chapter describes various issues that may occur when using a Gigabit Ethernet network.
3: Operating the Gigabit Ethernet PCI Express Board Using LEDs to Determine Board Functionality The fiber-optic and copper Gigabit Ethernet boards have light-emitting diodes (LEDs) that indicate whether the board is configured correctly and connected to an active Ethernet, as discussed in the following sections. Fiber-Optic Board LEDs and Connector Specifications Table 3-1 describes the functions of these LEDs.
Verifying Functionality Table 3-2 Components LEDs LEDs on the Copper Gigabit Ethernet PCI Express Board Purpose Three LEDs per port Link/Activity: Turns on any link speed, blinks on activity (green) 100Mbits/s: Turns on 100 Mbit/s link (green) 1000 Mbits/s: Turns on 1000 Mbit/s link (green) LEDs are located on the Copper board visible through holes in the metal bracket holder.
3: Operating the Gigabit Ethernet PCI Express Board systemA:~ # ifconfig -a eth0 Link encap:Ethernet HWaddr 00:04:23:D3:A2:E8 UP BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 b) TX bytes:0 (0.
Resetting the Board UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:153 errors:0 dropped:0 overruns:0 frame:0 TX packets:153 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:11259 (10.9 KiB) TX bytes:11259 (10.9 KiB) sit0 Link encap:IPv6-in-IPv4 NOARP MTU:1480 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 b) TX bytes:0 (0.
This shows the current settings of the pause (or flow control) parameters. You can then change these parameters with the ethtool -A option. As with any system configuration changes, make sure to have a back out strategy, read the most recent documentation for potential changes and pitfalls, and consult with a relevant Linux archives for examples of common usage.
Appendix A A. Specifications This appendix provides the following information: • “Physical and Performance Characteristics” on page 21 • “General Technical Specifications” on page 22 Physical and Performance Characteristics Table A-1 summarizes the physical and performance characteristics for the copper board. Table A-1 Copper Gigabit Ethernet Tech. Specs.
A: Specifications Table A-2 summarizes the physical and performance characteristics for the fiber-optic board. Table A-2 Fiber-Optic Gigabit Ethernet Tech Specs. - (1000Base-SX) Adapters Characteristic Value IEEE Standard / Network topology: Fiber Gigabit Ethernet, 1000Base-SX (850nM) Data Transfer Rate: 2000Mbit/s in full duplex mode per port Cables and Operating distance: Multimode fiber: 220m at 62.5 um 550m at 50 um Optical Output Power: Typical: -6 dBm Minimum: -9.
General Technical Specifications Table A-3 Copper Board Gigabit Ethernet General Technical Specifications Specification Description PCI Express Card Type X4 Lane PCI Express Voltage: +3.3V +-9%, PCI Connector: X4 Lane Controllers: Broadcom BCM5715C Holder: Metal Bracket Weight: 80 gram (2.82 oz) Power Consumption: 1.97A at 3.3V: Typical, port operates at 1000Mbit/s. 1.31A at 3.3V: Typical, port operates at 100Mbit/s. 1.14A at 3.3V: Typical, port operates at 100Mbit/s. 1.23A at 3.
A: Specifications Table A-3 Copper Board Gigabit Ethernet General Technical Specifications Specification Description EMC Certifications: FCC Part 15, Subpart B Class B Conducted Emissions Radiated Emissions CE EN 55022: 1998 Class B Amendments A1: 2000; A2: 2003 Conducted Emissions Radiated Emissions CE EN 55024: 1998 Amendments A1: 2000; A2: 2003 Immunity for ITE Amendment A1: 2001 CE EN 61000-3-2 2000, Class A Harmonic Current Emissions CE EN 61000 3-3 1995, Amendment A1: 2001 Voltage Fluctuations a
General Technical Specifications Table A-3 Copper Board Gigabit Ethernet General Technical Specifications Specification Description EMC Certifications: CE EN 55022: 1998 Class B: Amendment A1 2000, A2 2003 Conducted Emissions Radiated Emissions CE EN 55024: 1998 Amendment A1: 2001, Amendment A2: 2003 Immunity for ITE CE EN 61000-3-2 2000 Harmonic Current Emissions CE EN 61000 3-3 Voltage Fluctuations and Flicker CE IEC 6100-4-2: 1995 ESDAirDischarge8kV.ContactDischarge4kV.
A: Specifications Table A-3 Copper Board Gigabit Ethernet General Technical Specifications Specification Description MTBF: 213 (Years) *According to Telcordia SR-332 Issue 1 Environmental condition ? GB (Ground, Fixed, Controlled). Ambient temperature - 25˚C. Temperature rise of 10˚C above the system ambient temperature was assumed for the cards components. Table A-4 summarizes the LEDs and connector specifications for the copper board.
General Technical Specifications Table A-5 Fiber-optic Board Gigabit Ethernet General Technical Specifications Specification Description PCI Express Card Type X4 Lane PCI Express Voltage: +3.3V +-9% PCI Connector: X4 Lane Controllers: Broadcom BCM5715S Holder: Metal Bracket Weight: 90 gram (3.2 oz) Power Consumption: 1.55A at 3.3V: Typical, port operates at 1000Mbit/s. 1.5A at 3.3V: Typical, No link.
A: Specifications Table A-5 Fiber-optic Board Gigabit Ethernet General Technical Specifications Specification Description EMC Certifications: FCC Part 15, Subpart B Class B Conducted Emissions Radiated Emissions CE EN 55022: 1998 Class B Amendments A1: 2000; A2: 2003 Conducted Emissions Radiated Emissions CE EN 55024: 1998 Amendments A1: 2000; A2: 2003 Immunity for ITE Amendment A1: 2001 CE EN 61000-3-2 2000, Class A Harmonic Current Emissions CE EN 61000 3-3 1995, Amendment A1: 2001 Voltage Fluctuati
General Technical Specifications Table A-5 Fiber-optic Board Gigabit Ethernet General Technical Specifications Specification Description MTBF*: 136 (Years) *According to Telcordia SR-332 Issue 1 Environmental condition ? GB (Ground, Fixed, Controlled). Ambient temperature - 25˚C. Temperature rise of 10˚C above the system ambient temperature was assumed for the cards components (except for the optical transceivers). Temperature rise of 15˚C was assumed for the optical transceivers.
Glossary acknowledge (Ack) packet The Ack packet informs the PE that initiated a message that the destination PE accepted the message. autonegotiation The process by which two computers (or a computer and a switch) connected by Gigabit Ethernet determine the speed and other parameters with which they will communicate. CD-ROM (CD) A flat metallic disk that contains information that you can view and copy onto your own hard disk; you cannot change or add to the disk.
Glossary MAC Medium access control, also called the physical layer. MAC address The physical address of the Gigabit Ethernet board, which is distinct from the IP address. MTU Maximum Transmission Unit is a configuration parameter that controls the size of the Ethernet frames that the Gigabit Ethernet board can transmit and receive. man (manual) page An online document that describes how to use a particular Linux command. Also called reference page.
Index Numbers 1000-Base-SX operating range, 8 1000-Base-T configuration, 9 operating range, 9 C cabling range, 8 connector LC, 4, 8, 12 RJ45, 7, 9 SC, 13 copper board cables, 9 characteristics, 21 connecting, 13 LEDs, 7 LEDs/connector specifications, 26 customer service, xvii F fiber-optic board cables, 8 characteristics, 22 LEDs, 4, 16 007-5472-001 LEDs/connector specifications, 29 fiber-optic Gigabit Ethernet board connecting, 12 G Gigabit Ethernet board cabling, 12 copper, 9 fiber-optic, 8 configur
Index LEDs copper, 7 fiber-optic, 4, 16 troubleshooting with,, 16 P ports copper board, 9 fiber-optic board, 8 product support, xvii R RJ45 connector, 7, 9 S supported systems, 1 T technical support, xvii troubleshooting with LEDs, 16 34 007-5472-001