SGI 10-Gigabit Ethernet Network Adapter User’s Guide ® 007–4669–003
CONTRIBUTORS Written by Lori Johnson Illustrated by Dan Young and Chrystie Danzer Engineering contributions Arthur Kepner, Edward Mascarenhas, Steve Modica, Alain Renaud, James Yarbrough COPYRIGHT © 2004-2006 Silicon Graphics, Inc. All rights reserved; provided portions may be copyright in third parties, as indicated elsewhere herein.
TUV R geprufte Sicherheit NRTL/C International Special Committee on Radio Interference (CISPR) This equipment has been tested to and is in compliance with the Class A limits per CISPR publication 22. Canadian Department of Communications Statement This digital apparatus does not exceed the Class A limits for radio noise emissions from digital apparatus as set out in the Radio Interference Regulations of the Canadian Department of Communications.
New Features This revision adds support for the Release 2 card, which is supported with SGI ProPack 4 Service Pack 3 and later releases.
Record of Revision 007–4669–003 Version Description 001 May 2004 Initial publication 002 August 2005 Revision 003 July 2006 Revision vii
Contents About this Guide Audience . . . . . . . . . . . . . . . . . . . . . . . . . xi . . . . . . . . . . . . . . . . . . . . . . xi . . . . . . . . . . . . . . . . . . . . . . xi . . . . . . . . . . . . . . . . . . . . . . xii Related Publications . . . . . . . . . . . . . . . . . . . . . . xii Obtaining Publications . . . . . . . . . . . . . . . . . . . . . .
Contents Verifying that the Adapter is Properly Configured and Enabled IRIX Adapter Configuration Verification . Linux Adapter Configuration Verification Resetting the Adapter . . . . . . . 12 . . . . . . . . . . . . . . 12 . . . . . . . . . . . . . . 13 . . . . . . . . . . . . . . . . . . . . . . 13 . . . . . . . . . . . . . . . . . . . . . . 14 IRIX Configuration . . . . . . . . . . . . . . . . . . .
About this Guide This guide describes the SGI 10-Gigabit (Gbit) Ethernet network adapter. It operates under the IRIX and SGI ProPack for Linux operating systems, as specified in the operating system release notes. You can use the SGI 10-Gbit Ethernet network adapter in addition to your current adapter. This guide shows you how to connect the adapter to an Ethernet network and explains how to operate the adapter.
About this Guide Scope of this Guide This guide is written to facilitate installation of the adapter and does not cover detailed points of network configuration. It contains the following chapters: • Chapter 1, "Features and Capabilities of the SGI 10-Gbit Ethernet Network Adapter", summarizes features, cabling, configuration limits, and tools. • Chapter 2, "Connecting the Adapter to a Network", shows you how to connect the adapter to your network.
® SGI 10-Gigabit Ethernet Network Adapter User’s Guide • Standard Linux man pages that are useful for any Ethernet device: – ethtool(8) – ifconfig(8) – ip(8) Obtaining Publications You can obtain SGI documentation as follows: • See the SGI Technical Publications Library at http://docs.sgi.com. Various formats are available. This library contains the most recent and most comprehensive set of online books, release notes, man pages, and other information.
About this Guide user input xiv This bold, fixed-space font denotes literal items that the user enters in interactive sessions. (Output is shown in nonbold, fixed-space font.
® SGI 10-Gigabit Ethernet Network Adapter User’s Guide [] Brackets enclose optional portions of a command or directive line. ... Ellipses indicate that a preceding element can be repeated. Product Support SGI provides a comprehensive product support and maintenance program for its products: • If you are in North America, contact the Technical Assistance Center at +1 800 800 4SGI or contact your authorized service provider.
About this Guide • Send mail to the following address: Technical Publications SGI 1500 Crittenden Lane, M/S 535 Mountain View, California 94043–1351 SGI values your comments and will respond to them promptly.
Chapter 1 Features and Capabilities of the SGI 10-Gbit Ethernet Network Adapter The 10-Gbit Ethernet technology is an extension of Gigabit Ethernet (1000-Base-T) technology that allows over-the-wire speeds up to 10 Gbits per second (Gbps), which is theoretically ten times the rate of existing technology. Given the limitations of the bus and the CPUs, realistic performance with the SGI 10-Gbit Ethernet network adapter will not exceed 7 Gbps.
1: Features and Capabilities of the SGI 10-Gbit Ethernet Network Adapter • "Configuration Limits" on page 3 • "Tools" on page 3 Features The adapter includes the following features: • Full-duplex Gigabit Ethernet interface as defined in the IEEE P802.3ae approved standard • Jumbo frame support (up to 9600 bytes) • 16 DMA channels (8 for reading, 8 for writing) • Single LC fiber connection • Support for PCI 64-bit 66-MHz and PCIX 66/100/133-MHz • Universal dual-voltage signaling (3.
® SGI 10-Gigabit Ethernet Network Adapter User’s Guide Table 1-1 10-Gbit Cable Standards Diameter (Microns) Modal Bandwidth (MHz * km) Range (Meters) 50 400 2 to 66 50 500 2 to 82 50 2000 2 to 300 Configuration Limits The number of the 10-Gbit Ethernet network adapters supported varies by system. Consult with your SGI representative to determine the currently supported maximum for your configuration.
Chapter 2 Connecting the Adapter to a Network To install the SGI 10-Gbit Ethernet network adapter, refer to the instructions for installing a PCI card in the user’s or owner’s guide that came with the SGI system. To connect the adapter to a network, do the following: 1. Remove the protective end caps and save them. ! Caution: 10-Gbit optics are very sensitive. If you plan on leaving them disconnected for any length of time, you must replace the end caps.
2: Connecting the Adapter to a Network Fiber-optic connector Figure 2-1 Release 1 Card: Fibre Optic Connections 6 007–4669–003
® SGI 10-Gigabit Ethernet Network Adapter User’s Guide Rx Tx Link Fiber-optic connector Figure 2-2 Release 2 Card: Fibre Optic Connections For information about configuring the interfaces, see IRIX Admin: Networking and Mail and The Network Administrators’ Guide.
Chapter 3 Operating the Adapter This chapter describes various issues that may occur when using the SGI 10-Gbit Ethernet network adapter in a 10-Gbit Ethernet network.
3: Operating the Adapter Table 3-1 LEDs on the SGI 10-Gbit Ethernet Network Adapter LED State Purpose ACT Blinking Off Data detected No data detected LINK On Off Good link No link: faulty cable, faulty connector, or communication mismatch During normal operation, the link LED is on; the data LED blinks whenever the adapter is receiving traffic. Note: At the time of release, a bug in the driver prevents the link light from behaving correctly. The link light may be on or off due to this bug.
® SGI 10-Gigabit Ethernet Network Adapter User’s Guide A line similar to the following should appear, where N is the number of the adapter: 10 Gigabit Ethernet: xgN, module 001c02, 66 MHz PCI bus 2 slot 1 For example, the first adapter would appear as xg1: 10 Gigabit Ethernet: xg1, module 001c02, 66 MHz PCI bus 2 slot 1 Linux Adapter Recognition Verification Unlike IRIX, under Linux all Ethernet adapters all share the prefix ethN. To verify that the adapter has been recognized, do the following: 1.
3: Operating the Adapter RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) Interrupt:62 Enabling the Adapter To enable the adapter, enter the following: # ifconfig IPaddress broadcast broadcastaddress netmask netmaskvalue For other systems to see the new address, you must enter the new hosts addresses in DNS or in host files or NIS as required for your system.
® SGI 10-Gigabit Ethernet Network Adapter User’s Guide In this address, the organizationally unique identifier (OUI) of the adapter vendor is represented by the first three sets of numbers (for example, 08:00:69). The last three sets vary, depending on the system. See the netstat man page for more details.
3: Operating the Adapter • Linux: # ifconfig ethN down # ifconfig ethN up Configuration IRIX and Linux use different configuration tools. IRIX Configuration The configuration of the adapter is controlled by a number of parameters, which are discussed in the IRIX /etc/config/xgcmd.options file. To change the configuration of your adapter, use the xgcmd command. For more information, see the xgcmd man page. Linux Configuration To change the configuration of your adapter, use the ethtool(8) command.
® SGI 10-Gigabit Ethernet Network Adapter User’s Guide The output will display information about the network adapters currently installed in the system, as follows: Name Mtu xg1 1500 Network 10.50.1 Address Ipkts Ierrs Opkts Oerrs Coll system.name 0 0 1 0 0 2. To bring down the adapter, enter the following command, where N is the number of the adapter: ifconfig xgN down 3.
Chapter 4 Performance Tuning This chapter discusses performance tuning topics for IRIX and SGI ProPack for Linux. For good TCP performance, the socket buffers used by applications must be at least as large as the bandwidth-round trip delay product between the two endpoints. Normally, larger socket buffers are called for with 10–Gbit Ethernet than when lower-bandwidth network interface cards are used.
4: Performance Tuning Read/Write Size and IRIX In a point-to-point configuration, use a buffer length of 513920 bytes. In a multiclient configuration, use a buffer length of 64240 bytes. Socket Buffer Size and IRIX Socket buffer size is set either: • By the application • Via the tcp_sendspace and tcp_recvspace tunable parameters In a point-to-point configuration, use one of the following socket buffer sizes shown in Table 4-1.
® SGI 10-Gigabit Ethernet Network Adapter User’s Guide SGI ProPack Performance Tuning This section discusses the following: • "Socket Read and Write Buffer Sizes for SGI ProPack" on page 19 • "Jumbo Frames and SGI ProPack" on page 19 • "Read/Write Size and SGI ProPack" on page 19 • "TCP/IP Socket Buffer Size and SGI ProPack" on page 19 Socket Read and Write Buffer Sizes for SGI ProPack The largest-allowed socket read and write buffer sizes are controlled by the following files: • Read: /proc/sys/net/core
4: Performance Tuning wmem_max will limit the amount of memory available for each socket’s buffers, and can result in degraded network throughput. Unless it is required to limit memory usage, SGI recommends that you do not reduce these below the default SGI ProPack values. If you to adjust the socket buffers, use sysctl(8) command.
Glossary CD-ROM (CD) A flat metallic disk that contains information that you can view and copy onto your own hard disk; you cannot change or add to the disk. CD-ROM is an abbreviation for compact disc read-only memory. Ethernet A communication network used to connect computers. gigabit (Gbit) A communication rate of 2^30 bits per second. host Any system connected to the network. hostname The name that uniquely identifies each host (system) on the network.
Glossary MTU Maximum Transmission Unit is a configuration parameter that controls the size of the Ethernet frames that the SGI 10-Gigabit Ethernet network adapter can transmit and receive. man (manual) page An online document that describes how to use a particular IRIX command. Also called reference page. multiclient configuration A TCP/IP configuration in which the IRIX system is connected via 10-Gbit Ethernet to a switch that fans out to multiple clients via 1-Gbit Ethernet.
Index fibre optic connections, 6 fibre type, 3 frame sizes, 2 frames, 14 functionality verification, 9 1000-Base-SX, 3 A adapter enabling, 12 adapter recognition, 10 Altix systems, 1 Altix XE systems, 1 C H hinv, 11 I cabling, 2 capabilities, 1 configuration limits, 3 connector, 2, 6 IEEE standards, 2 ifconfig, 3, 12, 14 installation, 5 interface names, 10 ip, 3 IP address, 12 IRIX performance tuning, 17 D diagnostics, 3 DMA channels, 2 driver, 3 E enabling the adapter, 12, 13 /etc/ioconfig.
Index M MAC address, 12 MMR, 3 MTU sizes, 14, 17 multibuffer mode and tuning, 18 multiclient configuration, 17 N netstat, 12–14 network connection, 5 network interface names, 10 rmem_max, 21 S SGI ProPack performance tuning, 19 signaling, 2 socket buffer size IRIX performance tuning, 18 SGI ProPack performance tuning, 19 standard-size Ethernet frames, 14 supported systems, 1 sysctl, 20 T O Onyx systems, 1 operating, 9 Origin systems, 1 P PCI-X bus interface, 2 performance tuning IRIX, 17 SGI ProPack,
® SGI 10-Gigabit Ethernet Network Adapter User’s Guide X xgcmd diagnostic tool, 3 xg driver, 3 007–4669–003 25
