SGI® 10-Gigabit Ethernet PCI Express Network Adapter User’s Guide 007–4983–002
COPYRIGHT © 2007 SGI. All rights reserved; provided portions may be copyright in third parties, as indicated elsewhere herein. No permission is granted to copy, distribute, or create derivative works from the contents of this electronic documentation in any manner, in whole or in part, without the prior written permission of SGI.
Attention Cet appareil numérique n’émet pas de perturbations radioélectriques dépassant les normes applicables aux appareils numériques de Classe A préscrites dans le Règlement sur les interferences radioélectriques etabli par le Ministère des Communications du Canada.
New Features in this Guide This revision includes additional SGI performance tuning recommendations. See Chapter 4, "Performance Tuning" on page 13.
Record of Revision 007–4983–002 Version Description 001 April 2007 Original publication 002 October 2007 Revision vii
Contents About this Guide Audience . . . . . . . . . . . . . . . . . . . . . . . . . xv . . . . . . . . . . . . . . . . . . . . . . xv . . . . . . . . . . . . . . . . . . . . . . xv . . . . . . . . . . . . . . . . . . . . . . xvi Related Publications . . . . . . . . . . . . . . . . . . . . . . xvi Obtaining Publications . . . . . . . . . . . . . . . . . . . . . .
Contents Verifying that the Adapter is Properly Configured and Enabled Resetting the Adapter . . Changing the Configuration Setting MTU Sizes Troubleshooting . . . . . . . . . 9 . . . . . . . . . . . . . . . . . . . . 10 . . . . . . . . . . . . . . . . . . . . 10 . . . . . . . . . . . . . . . . . . . . . . . 10 . . . . . . . . . . . . . . . . . . . . . . . 11 . . . . . . 13 4. Performance Tuning . .
Figures Figure 2-1 007–4983–002 Fiber Optic Connections . . . . . . . . . . . . . . . . .
Tables Table 1-1 10-Gbit Cable Standards Table 3-1 Faceplate LEDs 007–4983–002 . . . . . . . . . . . . . . . . . . . . 3 . . . . . . . . . . . . . . . . .
About this Guide This guide describes the SGI 10-Gigabit (Gbit) Ethernet PCI Express network adapter. It requires one of the following SGI ProPack for Linux releases: • SGI ProPack 5 Service Pack 1 or later • SGI ProPack 4 Service Pack 3 or later You can use the SGI 10-Gbit Ethernet PCI Express network adapter in addition to your current adapter. This guide shows you how to connect the adapter to an Ethernet network and explains how to operate the adapter.
About this Guide Scope of this Guide This guide is written to facilitate installation of the adapter and does not cover detailed points of network configuration. It contains the following chapters: • Chapter 1, "Features and Capabilities", summarizes features, cabling, configuration limits, and tools. • Chapter 2, "Connecting the Adapter to a Network", shows you how to connect the adapter to your network.
SGI® 10-Gigabit Ethernet PCI Express Network Adapter User’s Guide • Standard Linux man pages that are useful for any Ethernet device: ethtool(8) ifconfig(8) ip(8) Obtaining Publications You can obtain SGI documentation as follows: • See the SGI Technical Publications Library at http://docs.sgi.com. Various formats are available. This library contains the most recent and most comprehensive set of online books, release notes, man pages, and other information.
About this Guide [] Brackets enclose optional portions of a command or directive line. ... Ellipses indicate that a preceding element can be repeated. Product Support SGI provides a comprehensive product support and maintenance program for its products: • If you are in North America, contact the Technical Assistance Center at +1 800 800 4SGI or contact your authorized service provider. • If you are outside North America, contact the SGI subsidiary or authorized distributor in your country.
Chapter 1 Features and Capabilities This chapter discusses the following: • "SGI Systems Supported" on page 1 • "Key Features" on page 1 • "10-Gbit Ethernet Technology" on page 2 • "Cabling" on page 2 • "Configuration Limits" on page 3 • "Tools" on page 3 SGI Systems Supported The SGI 10-Gigabit (Gbit) Ethernet PCI Express network adapter is supported in the following systems: • SGI Altix 4700 • SGI Altix 450 • SGI Altix XE210 • SGI Altix XE240 • SGI Altix XE310 • SGI Altix XE1200 • SGI Altix XE1300 Key
1: Features and Capabilities Note: SGI does not support dual-protocol or 10–Gigabit Myrinet. • Wire-speed performance. • Firmware-controlled offload engine.
SGI® 10-Gigabit Ethernet PCI Express Network Adapter User’s Guide from 400-MHz * km to 2000-MHz * km, depending on its length, as shown in Table 1-1. Table 1-1 10-Gbit Cable Standards Diameter (Microns) Modal Bandwidth (MHz * km) Range (Meters) 50 400 2 to 66 50 500 2 to 82 50 2000 2 to 300 Configuration Limits The number of the 10-Gbit Ethernet PCI Express network adapters supported varies by system.
Chapter 2 Connecting the Adapter to a Network To install the SGI 10-Gbit Ethernet PCI Express network adapter, refer to the instructions for installing a PCI card in the user’s or owner’s guide that came with the SGI system. To connect the adapter to a network, do the following: 1. Remove the protective end caps and save them. ! Caution: 10-Gbit optics are very sensitive. If you plan on leaving them disconnected for any length of time, you must replace the end caps.
2: Connecting the Adapter to a Network Fiber-optic connector Figure 2-1 Fiber Optic Connections For information about configuring the interfaces, see The Network Administrators’ Guide.
Chapter 3 Operating the Adapter This chapter describes various issues that may occur when using the SGI 10-Gbit Ethernet PCI Express network adapter in a 10-Gbit Ethernet network.
3: Operating the Adapter Table 3-1 Faceplate LEDs Label Color Meaning S Yellow Controlled by the adapter’s firmware L Green Link connectivity R Green Receive (RX) traffic T Green Transmit (TX) traffic Verifying Adapter Recognition To verify that the adapter has been recognized, do the following: 1. Use the /sbin/lspci command to ensure that the device has been recognized.
SGI® 10-Gigabit Ethernet PCI Express Network Adapter User’s Guide Enabling the Adapter To enable the adapter, enter the following: [root@linux root]# ifconfig IPaddress netmask netmaskvalue broadcast broadcastaddress mtu 1500|9000 For example: [root@linux root]# ifconfig eth2 10.0.0.1 netmask 0xffffff00 broadcast 10.0.0.255 mtu 9000 Note: Ethernet interfaces are named eth0, eth1, and so on, always with a common prefix of eth.
3: Operating the Adapter Resetting the Adapter In the unlikely event that you need to reset the adapter, enter the following, where N is the adapter number: # ifconfig ethN down # ifconfig ethN up Changing the Configuration To change the configuration of your adapter, use the ethtool(8) command. For more information, see the ethtool(8) man page. Setting MTU Sizes The maximum transmission unit (MTU) size is controlled by the mtu mtu_size switch of the ifconfig command.
SGI® 10-Gigabit Ethernet PCI Express Network Adapter User’s Guide 2. To change the MTU size of the 10-Gbit Ethernet adapter, enter the following, where N is the number of the adapter: [root@linux root]# ifconfig ethN mtu mtu_size For example: [root@linux root]# ifconfig eth2 mtu 1500 3.
Chapter 4 Performance Tuning The default settings have been carefully chosen to maximize throughput for the SGI 10-Gigabit (Gbit) Ethernet PCI Express network adapter so that no further tuning is required. However, if you wish to experiment with the settings found on Myricom’s web site at http://www.myri.com/serve/cache/511.html, it is important that you follow the information in this chapter.
4: Performance Tuning sizes were chosen to be compatible with those recommended by the card manufacturer and required by SGI’s software. To set the network buffer sizes: 1. Add or change the following entries in the /etc/sysctl.conf file: net.core.rmem_default = 524287 net.core.rmem_max = 524287 net.core.wmem_default = 524287 net.core.wmem_max = 524287 net.ipv4.tcp_rmem = 10000000 10000000 16777216 net.ipv4.tcp_wmem = 10000000 10000000 16777216 net.ipv4.tcp_mem = 10000000 10000000 16777216 net.core.
Glossary Ethernet A communication network used to connect computers. gigabit (Gbit) A communication rate of 10^9 bits per second. host Any system connected to the network. hostname The name that uniquely identifies each host (system) on the network. IP address A number that uniquely identifies each host (system) on a TCP/IP network. LED Light-emitting diode, a light on a piece of hardware that indicates status or error conditions. MAC Medium access control, also called the physical layer.
Glossary man (manual) page An online document that describes how to use a particular command. Also called reference page. multiclient configuration A TCP/IP configuration in which the system is connected via 10-Gbit Ethernet to a switch that fans out to multiple clients via 1-Gbit Ethernet. NIS Network Information Service is a distributed database mechanism for user accounts, host names, mail aliases, and so on. PCI Express Peripheral Component Interconnect Express is a high-performance I/O interconnect.
Index 10GBASE-SR, 5 1000-Base-SX, 3 I ifconfig, 3, 9–11 installation, 5 ip, 3 A adapter enabling, 9 adapter resetting, 10 Altix systems, 1 Altix XE systems, 1 C cabling, 2 configuration changes, 10 configuration limits, 3 connector, 2, 6 J jumbo Ethernet frames, 10 jumbo frames MTU sizes supported, 10 tuning, 13 L LC connector, 2, 5 LEDs, 7 lspci, 8 E enabling the adapter, 9 end caps, 5 ethtool, 3, 10 F features and capabilities, 1 fibre optic connections, 6 fibre type, 3 frames, 10 functionality ve
Index O standard-size Ethernet frames, 10 supported systems, 1 operating, 7 T P performance tuning, 13 ports, 3 /proc/net/dev, 8 TCP time stamps, 14 tools, 3 transmit (TX) traffic, 8 troubleshooting, 11 troubleshooting with LEDs, 7 tuning, 13 R read/write size, 13 receive (RX) traffic, 8 recv calls, 13 resetting the adapter, 10 S /sbin/ifconfig, 8 /sbin/lsmod, 8 18 U unknown device, 8 update-pciids, 8 V verifying functionality, 7 007–4983–002
