User Guide NetXtreme II January 2010 Broadcom Ne tXtrem e II™ N etwor k Adapter Use r G ui de • Introduction • Functionality and Features • Teaming • Virtual LANs (VLANs) • Manageability • Installing the Hardware • Installing the Driver Software • Broadcom Boot Agent Driver Software • NDIS2 Driver Software • ODI Driver Software • Linux Driver Software • NetWare Driver Software • Windows Driver Software • Installing Management Applications • Using iSCSI • Configuring Teaming
NetXtreme II User Guide January 2010 the application or use of any product or circuit described herein, neither does it convey any license under its patent rights or the rights of others. Broadcom, the pulse logo, Connecting everything, the Connecting everything logo, NetXtreme, Ethernet@Wirespeed, LiveLink, and Smart Load Balancing are among the trademarks of Broadcom Corporation and/or its affiliates in the United States, certain other countries, and/or the EU.
User Guide NetXtreme II January 2010 F un cti on al i t y an d Fe atu res : B ro adc om NetXtre me II™ Ne twork Adapter Us er Guide • Functional Description • Features FUNCTIONAL DESCRIPTION The Broadcom NetXtreme II adapter is a new class of Gigabit Ethernet (GbE) and 10 GbE converged network interface controller (C-NIC) that can simultaneously perform accelerated data networking and storage networking on a standard Ethernet network.
NetXtreme II User Guide January 2010 FEATURES The following is a list of the Broadcom NetXtreme II adapter features: • TCP Offload Engine (TOE) • Internet Small Computer Systems Interface (iSCSI) offload • Single-chip solution • • • • Integrated 10/100/1000BASE-T transceivers • Integrated 10GBASE-T transceivers • 10/100/1000 triple-speed MAC • Host interfaces • SerDes interface for optical transceiver connection • PCI Express 1.
User Guide NetXtreme II January 2010 • Integrated 96 KB frame buffer memory • GMII/MII Management Interface • Four unique MAC unicast addresses • Support for multicast addresses via 128 bits hashing hardware function • Serial flash NVRAM memory • JTAG support • PCI Power Management Interface (v1.1) • 64-bit BAR support • EM64T processor support • AMD-64 processor support • 1.2 V core voltage, 0.
NetXtreme II User Guide January 2010 POWER MANAGEMENT Adapter speed connection when the system is down waiting for a wake-up signal may be at 10 Mbps or 100 Mbps, but can return to 1000 Mbit/s when the system is up and running if connected to a 1000 Mbps capable switch. Systems intending to use Wake on LAN (WOL) should be connected to a switch capable of both 1000 and 10/100 Mbps speeds. NOTES: • For specific systems, see your system documentation for WOL support.
User Guide NetXtreme II January 2010 SUPPORTED OPERATING ENVIRONMENTS The Broadcom NetXtreme II adapter has software support for the following operating systems: • Microsoft® Windows® (32-bit and 64-bit extended) • Microsoft Windows Vista™ (32-bit and 64-bit extended) • Linux® (32-bit and 64-bit extended) • MS-DOS® • ESX Server (VMware) • NetWare® • SCO® UnixWare® • SCO OpenServer® Bro adco m Co rp or atio n Document ENGSRVT52-CDUM100-R Supported Operating Environments Page 7
NetXtreme II User Guide January 2010 NETWORK LINK AND ACTIVITY INDICATION For copper-wire Ethernet connections, the state of the network link and activity is indicated by the LEDs on the RJ-45 connector, as described in Table 1. For fiber optic Ethernet connections, the state of the network link and activity is indicated by a single LED located adjacent to the port connector, as described in Table 2.
User Guide NetXtreme II January 2010 C o nf i g uri n g Te a m i n g: B ro a d c o m N e t X t r e m e I I ™ Network Adapter Us er Guide • Broadcom Advanced Server Program Overview • Load Balancing and Fault Tolerance BROADCOM ADVANCED SERVER PROGRAM OVERVIEW Broadcom Advanced Server Program (BASP) is the Broadcom teaming software for the Windows family of operating systems. BASP runs within the Broadcom Advanced Control Suite 3 (BACS) utility.
NetXtreme II User Guide January 2010 NOTE: Although a team can be created with one adapter, it is not recommended since this defeats the purpose of teaming. A team consisting of one adapter is automatically created when setting up VLANs on a single adapter, and this should be the only time when creating a team with one adapter. TYPES OF TEAMS The available types of teams for the Windows family of operating systems are: • Smart Load Balancing and Failover • Link Aggregation (802.
User Guide NetXtreme II January 2010 LINK AGGREGATION (802.3AD) This mode supports link aggregation and conforms to the IEEE 802.3ad (LACP) specification. Configuration software allows you to dynamically configure which adapters you want to participate in a given team. If the link partner is not correctly configured for 802.3ad link configuration, errors are detected and noted. With this mode, all adapters in the team are configured to receive packets for the same MAC address.
NetXtreme II User Guide January 2010 SLB (AUTO-FALLBACK DISABLE) The SLB (Auto-Fallback Disable) type of team is identical to the Smart Load Balancing and Failover type of team, with the following exception—when the standby member is active, if a primary member comes back on line, the team continues using the standby member, rather than switching back to the primary member. All primary interfaces in a team participate in load-balancing operations by sending and receiving a portion of the total traffic.
User Guide NetXtreme II January 2010 NOTE: IPv6 is supported for addressing, but is not supported for load balancing.
NetXtreme II User Guide January 2010 B ro a d c o m G i ga bi t E t h e r n e t Te a m i ng S e r vi c e s : Broadcom Ne tXtrem e II™ N etwor k Adapter Use r G ui de • Executive Summary • Teaming Mechanisms • Teaming and Other Advanced Networking Properties • General Network Considerations • Application Considerations • Troubleshooting Teaming Problems • Frequently Asked Questions • Appendix A: Event Log Messages EXECUTIVE SUMMARY • Glossary • Teaming Concepts • Software Components •
User Guide NetXtreme II January 2010 GLOSSARY Table 1: Glossary Item Definition ARP Address Resolution Protocol BACS Broadcom Advanced Control Suite BASP Broadcom Advanced Server Program (intermediate driver) DNS domain name service G-ARP Gratuitous Address Resolution Protocol Generic Trunking (FEC/GEC)/802.3ad-Draft Static Switch-dependent load balancing and failover type of team in which the intermediate driver manages outgoing traffic and the switch manages incoming traffic.
NetXtreme II User Guide January 2010 Table 1: Glossary (Cont.
User Guide NetXtreme II January 2010 TEAMING CONCEPTS • Network Addressing • Teaming and Network Addresses • Description of Teaming Types • TOE Teaming The concept of grouping multiple physical devices to provide fault tolerance and load balancing is not new. It has been around for years. Storage devices use RAID technology to group individual hard drives. Switch ports can be grouped together using technologies such as Cisco Gigabit EtherChannel, IEEE 802.
NetXtreme II User Guide January 2010 team must use a unique MAC address to be IEEE compliant. It is important to note that ARP cache entries are not learned from received frames, but only from ARP requests and ARP replies. Description of Teaming Types • Smart Load Balancing and Failover • Generic Trunking • Link Aggregation (IEEE 802.
User Guide NetXtreme II January 2010 Smart Load Balancing enables both transmit and receive load balancing based on the Layer 3/Layer 4 IP address and TCP/ UDP port number. In other words, the load balancing is not done at a byte or frame level but on a TCP/UDP session basis. This methodology is required to maintain in-order delivery of frames that belong to the same socket conversation. Load balancing is supported on 2 to 8 ports.
NetXtreme II User Guide January 2010 Generic Trunking Generic Trunking is a switch-assisted teaming mode and requires configuring ports at both ends of the link: server interfaces and switch ports. This is often referred to as Cisco Fast EtherChannel or Gigabit EtherChannel. In addition, generic trunking supports similar implementations by other switch OEMs such as Extreme Networks Load Sharing and Bay Networks or IEEE 802.3ad Link Aggregation static mode.
User Guide NetXtreme II January 2010 TOE Teaming All four basic teaming modes support failover of traffic from a failed adapter to other working adapters. All four teaming modes also support bidirectional load-balancing of TCP/IP traffic. A primary difference between the modes is that the SLB modes use a Broadcom proprietary algorithm to control how both inbound and outbound traffic is balanced across the network interfaces in the team. This has several advantages.
NetXtreme II User Guide January 2010 set of library functions to enable the communications between either miniport drivers or intermediate drivers and the protocol stack. The protocol stack implements IP, IPX and ARP. A protocol address such as an IP address is assigned to each miniport device instance, but when an Intermediate driver is installed, the protocol address is assigned to the virtual team adapter and not to the individual miniport devices that make up the team.
User Guide NetXtreme II January 2010 HARDWARE REQUIREMENTS • Repeater Hub • Switching Hub • Router The various teaming modes described in this document place certain restrictions on the networking equipment used to connect clients to teamed systems. Each type of network interconnect technology has an effect on teaming as described in the following sections. Repeater Hub A Repeater Hub allows a network administrator to extend an Ethernet network beyond the limits of an individual segment.
NetXtreme II User Guide January 2010 TEAMING SUPPORT BY PROCESSOR All team types are supported by the IA-32, AMD-64, and EM64T processors. CONFIGURING TEAMING The Broadcom Advanced Control Suite 3 utility is used to configure teaming in the supported operating system environments. The Broadcom Advanced Control Suite 3 (BACS) utility is designed to run on 32-bit and 64-bit Windows family of operating systems. BACS 3 is used to configure load balancing and fault tolerance teaming, and VLANs.
User Guide NetXtreme II January 2010 Table 4: Comparison of Team Types (Cont.) Switch-Independent Dynamic Link Aggregation (IEEE 802.
NetXtreme II User Guide January 2010 SELECTING A TEAM TYPE The following flow chart provides the decision flow when planning for Layer 2 teaming. For TOE teaming, only Smart Load Balancing™ and Failover type team is supported. The primary rationale for teaming is the need for additional network bandwidth and fault tolerance. Teaming offers link aggregation and fault tolerance to meet both of these requirements.
User Guide NetXtreme II January 2010 TEAMING MECHANISMS • Architecture • Types of Teams • Attributes of the Features Associated with Each Type of Team • Speeds Supported for Each Type of Team ARCHITECTURE The Broadcom Advanced Server Program is implemented as an NDIS intermediate driver (see Figure 2). It operates below protocol stacks such as TCP/IP and IPX and appears as a virtual adapter. This virtual adapter inherits the MAC Address of the first port initialized in the team.
NetXtreme II User Guide January 2010 Figure 2: Intermediate Driver Outbound Traffic Flow The Broadcom Intermediate Driver manages the outbound traffic flow for all teaming modes. For outbound traffic, every packet is first classified into a flow, and then distributed to the selected physical adapter for transmission. The flow classification involves an efficient hash computation over known protocol fields. The resulting hash value is used to index into an Outbound Flow Hash Table.
User Guide NetXtreme II January 2010 The outbound TCP and UDP packets are classified using Layer 3 and Layer 4 header information. This scheme improves the load distributions for popular Internet protocol services using well-known ports such as HTTP and FTP. Therefore, BASP performs load balancing on a TCP session basis and not on a packet-by-packet basis. In the Outbound Flow Hash Entries, statistics counters are also updated after classification.
NetXtreme II User Guide January 2010 ICMP ------> PhysAdapter1 ------> 10.0.0.1 If the server also sends an TCP and UDP flow to the same 10.0.0.1 address, they can be on the same physical adapter as IGMP and ICMP, or on completely different physical adapters from ICMP and IGMP. The stream may look like this: IGMP ------> PhysAdapter1 ------> 10.0.0.1 ICMP ------> PhysAdapter1 ------> 10.0.0.1 TCP------> PhysAdapter1 ------> 10.0.0.1 UDP------> PhysAdatper1 ------> 10.0.0.
User Guide NetXtreme II January 2010 The BASP intermediate driver continually monitors the physical ports in a team for link loss. In the event of link loss on any port, traffic is automatically diverted to other ports in the team. The SLB teaming mode supports switch fault tolerance by allowing teaming across different switches- provided the switches are on the same physical network or broadcast domain.
NetXtreme II User Guide January 2010 Switch-Dependent Generic Static Trunking This mode supports a variety of environments where the adapter link partners are statically configured to support a proprietary trunking mechanism. This mode could be used to support Lucent’s Open Trunk, Cisco’s Fast EtherChannel (FEC), and Cisco’s Gigabit EtherChannel (GEC).
User Guide NetXtreme II January 2010 Dynamic Trunking (IEEE 802.3ad Link Aggregation) This mode supports link aggregation through static and dynamic configuration via the Link Aggregation Control Protocol (LACP). With this mode, all adapters in the team are configured to receive packets for the same MAC address. The MAC address of the first adapter in the team is used and cannot be substituted for a different MAC address.
NetXtreme II User Guide January 2010 ATTRIBUTES OF THE FEATURES ASSOCIATED WITH EACH TYPE OF TEAM The attributes of the features associated with each type of team are summarized in Table 5.
User Guide NetXtreme II January 2010 Table 5: Attributes (Cont.) Feature Attribute Hot replace Yes Hot add Yes Hot remove Yes Link speed support Different speeds Frame protocol All Incoming packet management Switch Outgoing packet management BASP Failover event Loss of link only Failover time <500 ms Fallback time 1.
NetXtreme II User Guide January 2010 TEAMING AND OTHER ADVANCED NETWORKING PROPERTIES • Checksum Offload • IEEE 802.1p QoS Tagging • Large Send Offload • TCP Offload Engine (TOE) • Jumbo Frames • IEEE 802.1Q VLANs • Wake On LAN • Preboot Execution Environment Before creating a team, adding or removing team members, or changing advanced settings of a team member, make sure each team member has been configured similarly.
User Guide NetXtreme II January 2010 CHECKSUM OFFLOAD Checksum Offload is a property of the Broadcom network adapters that allows the TCP/IP/UDP checksums for send and receive traffic to be calculated by the adapter hardware rather than by the host CPU. In high-traffic situations, this can allow a system to handle more connections more efficiently than if the host CPU were forced to calculate the checksums.
NetXtreme II User Guide January 2010 insertion of a tag into an Ethernet frame to identify the VLAN to which a frame belongs. If present, the 4-byte VLAN tag is inserted into the Ethernet frame between the source MAC address and the length/type field. The first 2-bytes of the VLAN tag consist of the IEEE 802.1Q tag type, whereas the second 2 bytes include a user priority field and the VLAN identifier (VID). Virtual LANs (VLANs) allow the user to split the physical LAN into logical subparts.
User Guide NetXtreme II January 2010 GENERAL NETWORK CONSIDERATIONS • Teaming with Microsoft Virtual Server 2005 • Teaming Across Switches • Spanning Tree Algorithm • Layer 3 Routing/Switching • Teaming with Hubs (for troubleshooting purposes only) • Teaming with Microsoft NLB TEAMING WITH MICROSOFT VIRTUAL SERVER 2005 The only supported BASP team configuration when using Microsoft Virtual Server 2005 is with a Smart Load Balancing (TM) team-type consisting of a single primary Broadcom adapte
NetXtreme II User Guide January 2010 types of teaming supported by Broadcom. Therefore, an interconnect link must be provided between the switches that connect to ports in the same team. In the configuration without the interconnect, an ICMP Request from Blue to Gray goes out port 82:83 destined for Gray port 5E:CA, but the Top Switch has no way to send it there because it cannot go along the 5E:C9 port on Gray. A similar scenario occurs when Gray attempts to ping Blue.
User Guide NetXtreme II January 2010 The addition of a link between the switches allows traffic from/to Blue and Gray to reach each other without any problems. Note the additional entries in the CAM table for both switches. The link interconnect is critical for the proper operation of the team. As a result, it is highly advisable to have a link aggregation trunk to interconnect the two switches to ensure high availability for the connection.
NetXtreme II User Guide January 2010 Figure 5 represents a failover event in which the cable is unplugged on the Top Switch port 4. This is a successful failover with all stations pinging each other without loss of connectivity.
User Guide NetXtreme II January 2010 SPANNING TREE ALGORITHM • Topology Change Notice (TCN) • Port Fast/Edge Port In Ethernet networks, only one active path may exist between any two bridges or switches. Multiple active paths between switches can cause loops in the network. When loops occur, some switches recognize stations on both sides of the switch. This situation causes the forwarding algorithm to malfunction allowing duplicate frames to be forwarded.
NetXtreme II User Guide January 2010 Topology Change Notice BPDUs are sent when a port that was forwarding changes to blocking or transitions to forwarding. A TCN BPDU does not initiate an STP recalculation. It only affects the aging time of the forwarding table entries in the switch.It will not change the topology of the network or create loops. End nodes such as servers or clients trigger a topology change when they power off and then power back on.
User Guide NetXtreme II January 2010 SLB Team Connected to a Single Hub SLB teams configured as shown in Figure 6 maintain their fault tolerance properties. Either server connection could potentially fail, and network functionality is maintained. Clients could be connected directly to the hub, and fault tolerance would still be maintained; server performance, however, would be degraded. Figure 6: Team Connected to a Single Hub Generic and Dynamic Trunking (FEC/GEC/IEEE 802.3ad) FEC/GEC and IEEE 802.
NetXtreme II User Guide January 2010 APPLICATION CONSIDERATIONS • Teaming and Clustering • Teaming and Network Backup TEAMING AND CLUSTERING • Microsoft Cluster Software • High-Performance Computing Cluster • Oracle Microsoft Cluster Software MSCS clusters support up to two nodes if you are using Windows 2000 Server. If you are using Windows Server 2003, MSCS clusters support up to eight nodes.
User Guide NetXtreme II January 2010 Figure 7 shows a 2-node Fibre-Channel cluster with three network interfaces per cluster node: one private and two public. On each node, the two public adapters are teamed, and the private adapter is not. Teaming is supported across the same switch or across two switches. Figure 8 shows the same 2-node Fibre-Channel cluster in this configuration.
NetXtreme II User Guide January 2010 High-Performance Computing Cluster Gigabit Ethernet is typically used for the following three purposes in high-performance computing cluster (HPCC) applications: • Inter-Process Communications (IPC): For applications that do not require low-latency, high-bandwidth interconnects (such as Myrinet, InfiniBand), Gigabit Ethernet can be used for communication between the compute nodes.
User Guide NetXtreme II January 2010 Oracle In our Oracle Solution Stacks, we support adapter teaming in both the private network (interconnect between RAC nodes) and public network with clients or the application layer above the database layer.
NetXtreme II User Guide January 2010 TEAMING AND NETWORK BACKUP • Load Balancing and Failover • Fault Tolerance When you perform network backups in a nonteamed environment, overall throughput on a backup server adapter can be easily impacted due to excessive traffic and adapter overloading.
User Guide NetXtreme II January 2010 Because there are four client servers, the backup server can simultaneously stream four backup jobs (one per client) to a multidrive autoloader. Because of the single link between the switch and the backup server; however, a 4-stream backup can easily saturate the adapter and link.
NetXtreme II User Guide January 2010 The same algorithm applies if a third and fourth backup operation is initiated from the backup server. The teamed interface on the backup server transmits a unicast G-ARP to backup clients to inform them to update their ARP cache. Each client then transmits backup data along a route to the target MAC address on the backup server.
User Guide NetXtreme II January 2010 Bro adco m Co rp or atio n Document ENGSRVT52-CDUM100-R Application Considerations Page 53
NetXtreme II User Guide January 2010 TROUBLESHOOTING TEAMING PROBLEMS • Teaming Configuration Tips • Troubleshooting Guidelines When running a protocol analyzer over a virtual adapter teamed interface, the MAC address shown in the transmitted frames may not be correct. The analyzer does not show the frames as constructed by BASP and shows the MAC address of the team and not the MAC address of the interface transmitting the frame.
User Guide NetXtreme II January 2010 and fallback. 15. Test the performance behavior of the team before placing into a production environment. 16. Network teaming is not supported when running iSCSI traffic via Microsoft iSCSI initiator or iSCSI offload. MPIO should be used instead of Broadcom network teaming for these ports. 17. For information on iSCSI boot and iSCSI offload restrictions, see iSCSI Protocol.
NetXtreme II User Guide January 2010 Question: Can I connect the teamed adapters to a hub? Answer: Teamed ports can be connected to a hub for troubleshooting purposes only. However, this practice is not recommended for normal operation because the performance would be degraded due to hub limitations. Connect the teamed ports to a switch instead. Question: Can I connect the teamed adapters to ports in a router? Answer: No.
User Guide NetXtreme II January 2010 Question: During backup operations, does the adapter teaming algorithm load balance data at a byte-level or a sessionlevel? Answer: When using adapter teaming, data is only load balanced at a session level and not a byte level to prevent out-oforder frames. Adapter teaming load balancing does not work the same way as other storage load balancing mechanisms such as EMC PowerPath.
NetXtreme II User Guide January 2010 APPENDIX A: EVENT LOG MESSAGES • Windows System Event Log messages • Base Driver (Physical Adapter/Miniport) • Intermediate Driver (Virtual Adapter/Team) WINDOWS SYSTEM EVENT LOG MESSAGES The known base and intermediate Windows System Event Log status messages for the Broadcom NetXtreme II adapters are listed in Table 8 and Table 9. As a Broadcom adapter driver loads, Windows places a status code in the system event viewer.
User Guide NetXtreme II January 2010 Table 8: Base Driver Event Log Messages (Cont.) Message Severity Number Message Cause Corrective Action 7 Informational Network controller configured for 10Mb fullduplex link. The adapter has been manually configured for the selected line speed and duplex settings. No action is required. 8 Informational Network controller configured for 100Mb halfduplex link. The adapter has been manually configured for the selected line speed and duplex settings.
NetXtreme II User Guide January 2010 Table 8: Base Driver Event Log Messages (Cont.) Message Severity Number Message Cause Corrective Action 20 Error Driver initialization failed. Unspecified failure during driver initialization. Reinstall the driver, update to a newer driver, run Broadcom Advanced Control Suite 3 diagnostics, or replace the adapter. 21 Informational Network controller configured for 10Gb fullduplex link.
User Guide NetXtreme II January 2010 INTERMEDIATE DRIVER (VIRTUAL ADAPTER/TEAM) The intermediate driver is identified by source BLFM, regardless of the base driver revision. Table 9 lists the event log messages supported by the intermediate driver, explains the cause for the message, and provides the recommended action.
NetXtreme II User Guide January 2010 Table 9: Intermediate Driver Event Log Messages (Cont.) System Event Message Number Severity Message Cause Corrective Action 14 Informational Network adapter does not support Advanced Failover. The physical adapter does not support the Broadcom NIC Extension (NICE). Replace the adapter with one that does support NICE. 15 Informational Network adapter is enabled via management interface.
User Guide NetXtreme II January 2010 Vir t ua l L ANs: Broa dc om Ne tXtrem e II ™ Networ k A d a p t e r U s e r G ui de • VLAN Overview • Adding VLANs to Teams VLAN OVERVIEW Virtual LANs (VLANs) allow you to split your physical LAN into logical parts, to create logical segmentation of workgroups, and to enforce security policies for each logical segment.
NetXtreme II User Guide January 2010 Figure 1 shows an example network that uses VLANs. In this example network, the physical LAN consists of a switch, two servers, and five clients. The LAN is logically organized into three different VLANs, each representing a different IP subnet. The features of this network are described in Table 1. Table 1: Example VLAN Network Topology Component Description VLAN #1 An IP subnet consisting of the Main Server, PC #3, and PC #5.
User Guide NetXtreme II January 2010 ADDING VLANS TO TEAMS Each team supports up to 64 VLANs (63 tagged and 1 untagged). Note that only Broadcom adapters and Alteon® AceNIC adapters can be part of a team with VLANs. With multiple VLANs on an adapter, a server with a single adapter can have a logical presence on multiple IP subnets. With multiple VLANs in a team, a server can have a logical presence on multiple IP subnets and benefit from load balancing and failover.
NetXtreme II User Guide January 2010 M a nag e a b il i t y : B ro a d c o m N e t X t r e m e I I™ Network Adapter Us er Guide • CIM • SNMP CIM The Common Information Model (CIM) is an industry standard defined by the Distributed Management Task Force (DMTF). Microsoft implements CIM on Windows server platforms. Broadcom will support CIM on Windows server platforms. Broadcom's implementation of CIM will provide various classes to provide information to users through CIM client applications.
User Guide NetXtreme II January 2010 SNMP BASP SUBAGENT The BASP subagent, baspmgnt.dll, is designed for the Windows 2000 Server and Windows Server 2003 SNMP service. It is required to install the SNMP service before installing the BASP subagent. The BASP subagent allows an SNMP manager software to actively monitor the configurations and performance of the Broadcom Advanced Server features.
NetXtreme II User Guide January 2010 BASP EXTENSIBLE-AGENT The Broadcom NetXtreme II Gigabit Ethernet Controller Extended Information SNMP extensible-agent (bcmif.dll) is designed for Windows 2000 Server, Windows Server 2003, and Windows Server 2008 SNMP service. It is required that Windows 2000 Server SNMP service is installed before installing the extensible-agent. The extensible-agent allows the SNMP manager software to actively monitor the configurations of the Broadcom NetXtreme II adapter.
User Guide NetXtreme II January 2010 Insta l l in g t h e Hardwa r e : B ro a d c o m N e t X t r e m e I I ™ N e t wo r k A da pt e r U s e r G u id e • Overview • System Requirements • Safety Precautions • Preinstallation Checklist • Installation of the Add-In NIC OVERVIEW This section applies to Broadcom NetXtreme II add-in network interface cards.
NetXtreme II User Guide January 2010 Novell NetWare Novell NetWare 6.5 with the latest support pack. Linux Although the adapter driver should work with many Linux kernel versions and distributions, it has only been tested on 2.4x kernels (starting from 2.4.24) and 2.6.x kernels. The driver may not compile on kernels older than 2.4.24. Testing is concentrated on i386 and x86_64 architectures. Only limited testing has been done on other architectures.
User Guide NetXtreme II January 2010 INSTALLATION OF THE ADD-IN NIC The following instructions apply to installing the Broadcom NetXtreme II adapter (add-in NIC) in most systems. Refer to the manuals that were supplied with your system for details about performing these tasks on your particular system. INSTALLING THE ADD-IN NIC 1. Review Safety Precautions and Preinstallation Checklist.
NetXtreme II User Guide January 2010 Table 1: 10/100/1000BASE-T and 10GBASE-T Cable Specifications Port Type Connector Media Maximum Distance 10BASE-T RJ-45 Category 3, 4, or 5 unshielded twisted pairs (UTP) 100m (328 ft) 100/1000BASE-T1 RJ-45 Category 52 UTP 100m (328 ft) 10GBASE-T RJ-45 Category 63 UTP 50m (164 ft) 100m (328 ft) Category 6A3 UTP 1 1000BASE-T signaling requires four twisted pairs of Category 5 balanced cabling, as specified in ISO/IEC 11801:2002 and ANSI/EIA/TIA-568-B.
User Guide NetXtreme II January 2010 B ro adc om B oo t A g en t D ri ver S o f t wa re: Broadcom Ne tXtrem e II™ N etwor k Adapter Use r G ui de • Overview • Setting Up MBA in a Client Environment • Setting Up MBA in a Server Environment OVERVIEW Broadcom NetXtreme II adapters support Preboot Execution Environment (PXE), Remote Program Load (RPL), iSCSI, and Bootstrap Protocol (BootP).
NetXtreme II User Guide January 2010 SETTING UP MBA IN A CLIENT ENVIRONMENT Setting up MBA in a client environment involves the following steps: 1. Enabling the MBA driver. 2. Configuring the MBA driver. 3. Setting up the BIOS for the boot order. ENABLING THE MBA DRIVER To enable or disable the MBA driver: 1. Insert the installation CD in the CD-ROM drive and boot up in DOS mode. NOTE: The uxdiag.exe file is on the installation CD. 2.
User Guide NetXtreme II January 2010 4. Use the UP ARROW, DOWN ARROW, LEFT ARROW, and RIGHT ARROW keys to move to and change the values for other menu items, as desired. 5. Press F4 to save your settings. 6. Press ESC when you are finished. SETTING UP THE BIOS To boot from the network with the MBA, make the MBA enabled adapter the first bootable device under the BIOS. This procedure depends on the system BIOS implementation. Refer to the user manual for the system for instructions.
NetXtreme II User Guide January 2010 i S C S I P ro t oc ol : B ro a d c o m N e t X t r e m e I I™ Network Adapter Us er Guide • iSCSI Boot • iSCSI Crash Dump • iSCSI Offload ISCSI BOOT Broadcom NetXtreme II Gigabit Ethernet adapters support iSCSI boot to enable network boot of operating systems to diskless systems. The iSCSI boot allows a Windows or Linux operating system boot from an iSCSI target machine located remotely over a standard IP network.
User Guide NetXtreme II January 2010 1. Create an iSCSI target. 2. Create a virtual disk. 3. Map the virtual disk to the iSCSI target created in step 1. 4. Associate an iSCSI initiator with the iSCSI target. 5. Record the iSCSI target name, TCP port number, iSCSI Logical Unit Number (LUN), initiator Internet Qualified Name (IQN), and CHAP authentication details.
NetXtreme II User Guide January 2010 Table 1: Configuration Options Option Description DHCP Vendor ID Controls how the iSCSI boot host software interprets the Vendor Class ID field used during DHCP. If the Vendor Class ID field in the DHCP Offer packet matches the value in the field, the iSCSI boot host software looks into the DHCP Option 43 fields for the required iSCSI boot extensions. If DHCP is disabled, this value does not need to be set.
User Guide NetXtreme II January 2010 • Boot to iSCSI target: Disabled • DHCP Vendor ID: BRCM ISAN • Link Up Delay Time: 0 • Use TCP Timestamp: Enabled • Target as First HDD: Disabled • LUN Busy Retry Count: 0 • IP Version: IPv6. (For IPv6, non-offload.) • Windows HBA Boot Mode: Disabled 4. Select ESC to return to the Main menu. 5. From the Main menu, select Initiator Parameters. 6.
NetXtreme II User Guide January 2010 NOTE: When using a DHCP server, the DNS server entries are overwritten by the values provided by the DHCP server. This occurs even if the locally provided values are valid and the DHCP server provides no DNS server information. When the DHCP server provides no DNS server information, both the primary and secondary DNS server values are set to 0.0.0.0.
User Guide NetXtreme II January 2010 Configure Parameters for a Secondary Adapter A second iSCSI boot adapter can be configured for redundancy in the event the primary adapter fails to boot. To configure the iSCSI boot parameters for a secondary adapter 1. From the MBA Configuration Menu, select CTRL+K. 2. From the Main menu, select Secondary Device Parameters. 3. From the Secondary Device Parameters screen, select Secondary Device. 4.
NetXtreme II User Guide January 2010 4. Select support for Multipath I/O (MPIO), if needed. Refer to Microsoft’s Initiator documentation for more information on MPIO. 5. Select the iSCSI boot option. 6. Select the Broadcom adapter as the iSCSI boot device. NOTES: • Do not manually create connections to the iSCSI target for iSCSI boot adapters. • If the image is used on other hardware, Sysprep is required. • It is recommended to always run iscsibcg.
User Guide NetXtreme II January 2010 5. Select ESC to return to the Main menu. 6. From the Main menu, select 1st Target Parameters. 7. From the 1st Target Parameters screen, type values for the following using the values used when configuring the iSCSI target: • CHAP ID (optional if two-way CHAP) • CHAP Secret (optional if two-way CHAP, and must be 12 characters in length or longer) 8. Select ESC to return to the Main menu. 9. Select ESC and select Exit and Save Configuration.
NetXtreme II User Guide January 2010 DHCP Option 43, Vendor-Specific Information DHCP option 43 (vendor-specific information) provides more configuration options to the iSCSI client than DHCP option 17. In this configuration, three additional suboptions are provided that assign the initiator IQN to the iSCSI boot client along with two iSCSI target IQNs that can be used for booting.
User Guide NetXtreme II January 2010 DHCP iSCSI Boot Configuration for IPv6 The DHCPv6 server can provide a number of options, including stateless or stateful IP configuration, as well s information to the DHCPv6 client. For iSCSI boot, Broadcom adapters support the following DHCP configurations: • DHCPv6 Option 16, Vendor Class Option • DHCPv6 Option 17, Vendor-Specific Information NOTE: The DHCPv6 standard Root Path option is not yet available.
NetXtreme II User Guide January 2010 Configuring the DHCP Server Configure the DHCP server to support Option 16 and Option 17. NOTE: • The format of DHCPv6 Option 16 and Option 17 are fully defined in RFC 3315. LINUX ISCSI BOOT SETUP Linux iSCSI boot is supported on Red Hat Enterprise Linux 5 update 1 and later and SUSE Linux Enterprise Server 10 SP1 and later.
User Guide NetXtreme II January 2010 Booting See Booting. OTHER ISCSI BOOT CONSIDERATIONS There are several other factors that should be considered when configuring a system for iSCSI boot. Virtual LANs Virtual LAN (VLAN) tagging is not supported for iSCSI boot with the Microsoft iSCSI Software Initiator.
NetXtreme II User Guide January 2010 x64 Architecture • • Disk1: "WDF Installation Disk" • bxvbd.cat • bxvbd.inf • bxvbda.sys • txtsetup.oem • wdf01000.sys • wdfldr.sys Disk2: "Broadcom iSCSI Driver" • bxois.cat • bxois.inf • bxois.sys • txtsetup.oem To perform an iSCSI remote installation 1. During system POST, enter the MBA configuration menu CTRL+S when prompted. 2. Set Boot protocol to iSCSI 3. Enter the iSCSI configuration menu by selecting CTRL+K. 4.
User Guide NetXtreme II January 2010 Windows Server 2008 (Non-OIS) To perform a remote install for non-offload 1. Configure iSCSI OpROM with all proper settings. • Boot to iSCSI Target: Disable • Target as First HDD: Enable • HBA: Disable 2. Order the boot order so that iSCSI precedes the DVD. 3. Copy the VBD and NDIS driver to a USB flash or a diskette. 4. Boot the system and connect to the target. The Windows Server 2008 DVD installation begins. 5. Continue with the installation.
NetXtreme II User Guide January 2010 when prompted. 11. Enter the iSCSI configuration menu by selecting CTRL+K. 12. Enter General Parameters and set Boot to iSCSI target to Enable. 13. Exit and save the iSCSI configuration. 14. Exit and save the MBA configuration. 15. Continue with Windows GUI mode installation as usual. TROUBLESHOOTING ISCSI BOOT The following troubleshooting tips are useful for iSCSI boot.
User Guide NetXtreme II January 2010 ISCSI OFFLOAD iSCSI offload is a technology that offloads iSCSI protocol processing overhead from host processors to the iSCSI host bus adapter to increase network performance and throughput while helping to optimize server processor utilization. This section covers Windows iSCSI offload for the NetXtreme II family of network adapters. For Linux iSCSI offload, see Linux iSCSI Offload.
NetXtreme II User Guide January 2010 Configuring Broadcom iSCSI Using BACS 3 The Broadcom Advanced Control Suite (BACS 3) is used to manage all of Broadcom’s network adapters and advanced features. For more information, see Using Broadcom Advanced Control Suite 3. 1. Open BACS 3. 2. Select the Broadcom NetXtreme II C-NIC iSCSI adapter.
User Guide NetXtreme II January 2010 3. Select the Configuration tab. 4. DHCP is the default for IP address assignment, but this can be changed to static IP address assignment, if this is the preferred method of IP address assignment. NOTE: The IP address assignment method cannot be changed if the adapter was used for boot.
NetXtreme II User Guide January 2010 5. Select Apply and close BACS.
User Guide NetXtreme II January 2010 Configure Microsoft Initiator to Use Broadcom’s iSCSI Offload Now that the IP address has been configured for the iSCSI adapter, you need to use Microsoft Initiator to configure and add a connection to the iSCSI target using Broadcom iSCSI adapter. See Microsoft’s user guide for more details on Microsoft Initiator. 1. Open Microsoft Initiator. 2. Configure the initiator IQN name according to your setup. To change, click on Change. 3. Enter the initiator IQN name.
NetXtreme II User Guide January 2010 4. Select the Discovery tab and click Add to add a target portal. 5. Enter the IP address of the target and click Advanced.
User Guide NetXtreme II January 2010 6. From the General tab, select Broadcom NetXtreme II C-NIC iSCSI Adapter from Local adapter.
NetXtreme II User Guide January 2010 7. Select the IP address for the adapter from Source IP.
User Guide NetXtreme II January 2010 8. Click OK to close Advanced setting and then OK to add the target portal.
NetXtreme II User Guide January 2010 9. From the Targets tab, select the target and click Log On to log into your iSCSI target using the Broadcom iSCSI adapter. 10. Click on Advanced.
User Guide NetXtreme II January 2010 11. On the General tab, select the Broadcom NetXtreme Ii C-NIC iSCSI adapter from Local adapter. 12. Click OK to close Advanced settings. 13. Click OK to close the Microsoft Initiator. 14. To format your iSCSI partition, use Disk Manager. NOTES: • No teaming support for iSCSI-enabled adapters is available at this point. MPIO handles all failover and link aggregation for iSCSI traffic. • Network teaming and MPIO are supported concurrently on different ports.
NetXtreme II User Guide January 2010 ISCSI OFFLOAD FAQS Q: How do I assign an IP address for iSCSI offload? A: Use the Configurations tab in Broadcom Advanced Control Suite 3 (BACS). Q: What tools should be used to create the connection to the target? A: Use Microsoft iSCSI Software Initiator (version 2.08 or later). Q: How do I know that the connection is offloaded? A: Use Microsoft iSCSI Software Initiator. From a command line, type iscsicli sessionlist.
User Guide NetXtreme II January 2010 I ns t a l li n g M a n ag em e n t A pp li c a t io n s : B ro a d c o m NetXtre me II™ Ne twork Adapter Us er Guide • Installing Broadcom Advanced Control Suite 3 and Related Management Applications • Modifying Management Applications • Repairing Management Applications • Removing Management Applications INSTALLING BROADCOM ADVANCED CONTROL SUITE 3 AND RELATED MANAGEMENT APPLICATIONS The Broadcom Advanced Control Suite 3 (BACS 3) software and related managem
NetXtreme II User Guide January 2010 .NET FRAMEWORK REQUIREMENT Microsoft .NET Framework 2.0 includes the runtime and associated files needed to run BACS 3, and must be installed on your system in order for BACS 3 to operate. For information on the minimum and recommended .NET Framework versions for your operating system, see Table 1. NOTE: For optimal performance of BACS 3, Broadcom recommends .NET Framework 2.0 SP1, .NET Framework 3.0 SP1, or .NET Framework 3.5, depending on your operating system.
User Guide NetXtreme II January 2010 7. Click Install. 8. Click Finish to close the wizard. USING SILENT INSTALLATION NOTES: • All commands are case sensitive. • User must “Run as Administrator” for Vista when using “msiexec” for “silent” install/uninstall(s). • For detailed instructions and information about unattended installs, refer to the Silent.txt file in the MgmtApps folder.
NetXtreme II User Guide January 2010 Type the following: setup /s /v"/qn ADDSOURCE=BACSa64,CHMa64,BASPa64,SNMPa64" To perform a silent install from within a batch file To perform a silent install from within a batch file and wait for the install to complete before continuing with the next command line, type the following: start /wait setup /s /w /v/qn MODIFYING MANAGEMENT APPLICATIONS To modify the management applications 1. In Control Panel, double-click Add or Remove Programs. 2.
User Guide NetXtreme II January 2010 1. In Control panel, double-click Add or Remove Programs. 2. Click Broadcom Management Programs, and then click Remove. 3. Reboot your system to complete the removal of management applications.
NetXtreme II User Guide January 2010 ND IS2 Dr iv er Softwa re: Broadcom Ne tXtr eme I I ™ N e t wo r k A da pt e r U s e r G u id e • Overview • Preinstallation Requirements • Installing the NDIS2 Driver Software for Use on MS-DOS Platforms • Using Keywords for the Drivers OVERVIEW Two drivers are discussed in this section: • BXND20X: Broadcom NetXtreme II Gigabit Ethernet driver • BNX2EV: Broadcom NetXtreme II 10 Gigabit Ethernet driver The examples used in this section refer to the BXND20X
User Guide NetXtreme II January 2010 • Windows NT Server 4.0 users. When running Setup for Microsoft Network Client v3.0 for MS-DOS, click any network card from the list (NE2000 Compatible, for example) to create the startup disk. • After creating the startup disk, follow the instructions in Modifying the Startup Disk. To create a startup disk 1. Create a folder called NCADMIN in the root of the C drive. 2. Copy the NCADMIN.CN_, NCADMIN.EX_, and NCADMIN.
NetXtreme II User Guide January 2010 Example Protocol.ini file for IPX [network.setup] version=0x3110 netcard=ms$ne2clone,1,MS$NE2CLONE,1 transport=ms$ndishlp,MS$NDISHLP transport=ms$nwlink,MS$NWLINK lana0=ms$ne2clone,1,ms$nwlink lana1=ms$ne2clone,1,ms$ndishlp [MS$NE2CLONE] DriverName=BXND20X$ [protman] DriverName=PROTMAN$ PRIORITY=MS$NDISHLP [MS$NDISHLP] DriverName=ndishlp$ BINDINGS=ms$ne2clone [ms$nwlink] DriverName=nwlink$ FRAME=Ethernet_802.2 BINDINGS=MS$NE2CLONE LANABASE=0 Example Protocol.
User Guide NetXtreme II January 2010 2. Edit A:\Net\System.ini. a. Change netcard= to netcard=BXND20X.dos. b. Check for references to C:\NET and change C:\NET to A:\NET if necessary. Example System.ini file [network] sizworkbuf=1498 filesharing=no printsharing=no autologon=yes computername=MYPC lanroot=A:\NET username=USER1 workgroup=WORKGROUP reconnect=yes dospophotkey=N lmlogon=0 logondomain= preferredredir=basic autostart=basic maxconnections=8 [network drivers] netcard=BXND20X.dos transport=ndishlp.
NetXtreme II User Guide January 2010 lastdrive=z INSTALLING THE DOS NDIS2 DRIVER SOFTWARE ON THE HARD DISK To install the DOS NDIS2 Driver Software on the hard disk 1. Verify that the system has Microsoft LAN Manager 2.2 installed, with a protocol such as NetBEUI configured. 2. Create a folder on your hard disk to store the NDIS 2.01 driver. Example: C:\LANMAN 3. Copy the BXND20X.dos file to this folder. Edit the Config.sys file by adding the following lines: DEVICE = C:\LANMAN\PROTMAN.
User Guide NetXtreme II January 2010 USING KEYWORDS FOR THE DRIVERS The Protocol.ini file contains certain keywords that are used by the BXND20X.dos AND BXND20X.dos drivers. These keywords are listed below: BusNum. Specifies the number of the PCI bus on which the network adapter is located. Requires a decimal number having a value ranging from 0 to 255. DevNum. Specifies the device number assigned to the network adapter when it is configured by the PCI BIOS.
NetXtreme II User Guide January 2010 AcceptAllMC. Informs the driver to deliver all multicast packets to the upper protocol.
User Guide NetXtreme II January 2010 ODI Driver Software : Broadcom Ne tXtr eme II™ Network Adapter Us er Guide • Overview • ODI Client32 • DOS ODI OVERVIEW Three drivers are discussed in this section: • BX2.LAN: Broadcom NetXtreme II Gigabit Ethernet 32-bit driver • BX2.COM: Broadcom NetXtreme II Gigabit Ethernet 16-bit driver • BNX2EV.COM: Broadcom NetXtreme II 10 Gigabit Ethernet 16-bit driver The examples used in the ODI Client32 section only refer to the BX2.LAN driver.
NetXtreme II User Guide January 2010 6. Type install, and then press ENTER. 7. Read the agreement, and then press ENTER to accept the agreement. 8. Using the UP ARROW or DOWN ARROW key as necessary, highlight Novell Client for DOS (required), and then press F10 to save and continue. 9. Highlight 32-bit LAN Drivers, and then press F10 to save and continue. 10. Insert the floppy disk containing the Broadcom 32-bit LAN driver files into the floppy disk drive. 11.
User Guide NetXtreme II January 2010 NOTES: • At the end of the ODI driver software installation process, note the BusNum and DevNum values that are displayed. Alternatively, use Broadcom Advanced Control Suite 3 (see Viewing Resource Information) to view the bus number, function (port) number, and device number assigned to each adapter (Windows users only). • The values for the BusNum, DevNum, and FuncNum keywords are displayed in hexadecimal notation when the driver loads on the adapter.
NetXtreme II User Guide January 2010 DOS ODI The Open Data-Link Interface (ODI) driver software allows LAN drivers to work together irrespective of differences between the hardware and/or the protocol stacks of communicating computers. An intervening layer, called the link support layer (LSL), is responsible for routing different packet types between LAN drivers and protocol stacks. NOTE: The examples used in this section refer to the BX2.COM driver, but also apply to the BNX2EV.COM driver.
User Guide NetXtreme II January 2010 PROTOCOL IPXODI 5. Restart the computer to complete the installation. 6. As the files are being loaded, the computer attempts to attach to a NetWare server using the frame types specified in Net.cfg. If the connection is successful, a dialog box appears. 7. Change to drive F and log onto the NetWare server.
NetXtreme II User Guide January 2010 Example: LINK DRIVER BX2 SUBVID 14E4 SUBVID cannot be used with BUSNUM, DEVNUM or FUNNUM. The MODEL keyword is for loading the driver only on the first network interface card found that has a subvendor ID (SVID) that matches the one set in MODEL. The driver ignores all other NICs that do not have a matching SVID. MODEL expects a decimal number.
User Guide NetXtreme II January 2010 Li nu x Dri ve r So ftw are : Broa dc om Ne tXtr eme I I ™ Network Adapter Us er Guide • Introduction • Limitations • Packaging • Installing Linux Driver Software • Unloading/Removing the Linux Driver • Patching PCI Files (Optional) • Network Installations • Setting Values for Optional Properties • Driver Defaults • Driver Messages • Teaming with Channel Bonding • Statistics • Linux iSCSI Offload INTRODUCTION This section discusses the Linux
NetXtreme II User Guide January 2010 LIMITATIONS • bnx2 Driver • bnx2x Driver • bnx2i Driver BNX2 DRIVER The current version of the driver has been tested on 2.4.x kernels (starting from 2.4.24) and all 2.6.x kernels. The driver may not compile on kernels older than 2.4.24. NOTE: Support for the 2.4.21 kernels is provided in Red Hat Enterprise Linux 3. Testing is concentrated on i386 and x86_64 architectures. Only limited testing has been done on other architectures.
User Guide NetXtreme II January 2010 PACKAGING The Linux driver is released in the packaging formats shown in Table 2. The netxtreme2 package contains the bnx2 (1 Gb network adapter) driver, the bnx2x (10 Gb network adapter) driver, and the bnx2i (iSCSI) driver for source RPM and compressed tar. Table 2: Linux Driver Packaging Format Drivers Source RPM netxtreme2-version.src.rpm Compressed tar netxtreme2-version.tar.gz Supplemental tar netxtreme2_sup-version.tar.
NetXtreme II User Guide January 2010 INSTALLING LINUX DRIVER SOFTWARE • Installing the Source RPM Package • Building the Driver from the Source TAR File NOTE: If a bnx2/bnx2x/bnx2i driver is loaded and the Linux kernel is updated, the driver module must be recompiled if the driver module was installed using the source RPM or the TAR package. INSTALLING THE SOURCE RPM PACKAGE NOTE: The examples in this procedure refer to the bnx2 driver, but also apply to the bnx2x and bnx2i drivers. 1.
User Guide NetXtreme II January 2010 2.6.16 kernels and newer (bnx2i driver) /lib/modules/kernel_version/kernel/drivers/iscsi/bnx2i.ko 4. Load the driver: insmod bnx2 or modprobe bnx2 5. Load the cnic driver (if applicable): insmod cnic.ko or modprobe cnic To configure the network protocol and address, refer to the documentation provided with your operating system.
NetXtreme II User Guide January 2010 1. Unload existing driver and kill user daemon, if necessary: Manual: rmmod bnx2i pkill -9 bnx2id Using service: service bnx2id stop 2. Load the iSCSI driver and the user daemon: Manual: bnx2id insmod bnx2i.
User Guide NetXtreme II January 2010 UNLOADING/REMOVING THE LINUX DRIVER • Unloading/Removing the Driver from an RPM Installation • Removing the Driver from a TAR Installation UNLOADING/REMOVING THE DRIVER FROM AN RPM INSTALLATION NOTES: • The examples used in this procedure refer to the bnx2 driver, but also apply to the bnx2x driver. • On 2.6 kernels, it is not necessary to bring down the eth# interfaces before unloading the driver module.
NetXtreme II User Guide January 2010 Next, the old files can be backed up and the new files can be renamed for use. cp cp cp cp /usr/share/hwdata/pci.ids /usr/share/hwdata/old.pci.ids /usr/share/hwdata/pci.ids.new /usr/share/hwdata/pci.ids /usr/share/hwdata/pcitable /usr/share/hwdata/old.pcitable /usr/share/hwdata/pcitable.
User Guide NetXtreme II January 2010 BNX2X DRIVER disable_tpa The disable_tpa parameter can be supplied as a command line argument to disable the Transparent Packet Aggregation (TPA) feature. By default, the driver will aggregate TCP packets. Use disable_tpa to disable the advanced TPA feature. Set the disable_tpa parameter to 1 as shown below to disable the TPA feature on all NetXtreme II network adapters in the system. The parameter can also be set in modprobe.conf.
NetXtreme II User Guide January 2010 BNX2I DRIVER Optional parameters en_tcp_dack, error_mask1, and error_mask2 can be supplied as command line arguments to the insmod or modprobe command for bnx2i. error_mask1 and error_mask2 "Config FW iSCSI Error Mask #", use to configure certain iSCSI protocol violation to be treated either as a warning or a fatal error. All fatal iSCSI protocol violations will result in session recovery (ERL 0). These are bit masks.
User Guide NetXtreme II January 2010 Note that Broadcom validation is limited to a power of 2; for example, 16, 32. event_coal_div "Event Coalescing Divide Factor", performance tuning parameter used to moderate the rate of interrupt generation by the iscsi firmware. Defaults: 1 Valid values: 1, 2, 4, 8 bnx2i_nopout_when_commands_active "iSCSI NOOP even when connection is not idle", this parameter enables the offload initiator to send iSCSI NOP-OUT on wire even when the link is not truly idle.
NetXtreme II User Guide January 2010 Coalesce RX Frames IRQ: 6 (range is 0–255) Coalesce TX Microseconds: 80 (range is 0–1023) Coalesce TX Microseconds IRQ: 80 (range is 0–1023) Coalesce TX Frames: 20 (range is 0–255) Coalesce TX Frames IRQ: 20 (range is 0–255) Coalesce Statistics Microseconds: 999936 (approximately 1 second) (range is 0–16776960 in increments of 256) MSI: Enabled (if supported by the 2.6 kernel and the interrupt test passes) TSO: Enabled (on 2.
User Guide NetXtreme II January 2010 • bnx2 and bnx2x Driver • bnx2i Driver BNX2 AND BNX2X DRIVER NOTE: The examples used in this procedure refer to the bnx2 driver, but also apply to the bnx2x driver. Driver Sign on Broadcom NetXtreme II Gigabit Ethernet Driver bnx2 v1.6.3c (July 23, 2007) CNIC Driver Sign on (bnx2 only) Broadcom NetXtreme II cnic v1.1.
NetXtreme II User Guide January 2010 NOTE: This message is displayed only when the user attempts to make an iSCSI connection.
User Guide NetXtreme II January 2010 bnx2i: bnx2i: bnx2i: bnx2i: bnx2i: bnx2i: bnx2i: bnx2i: bnx2i: bnx2i: bnx2i: bnx2i: bnx2i: bnx2i: bnx2i: bnx2i: bnx2i: bnx2i: bnx2i: bnx2i: bnx2i: bnx2i: bnx2i: bnx2i: bnx2i: bnx2i: iscsi_error iscsi_error iscsi_error iscsi_error iscsi_error iscsi_error iscsi_error iscsi_error iscsi_error iscsi_error iscsi_error iscsi_error iscsi_error iscsi_error iscsi_error iscsi_error iscsi_error iscsi_error iscsi_error iscsi_error iscsi_error iscsi_error iscsi_error iscsi_error i
NetXtreme II User Guide January 2010 TEAMING WITH CHANNEL BONDING With the Linux drivers, you can team adapters together using the bonding kernel module and a channel bonding interface. For more information, see the Channel Bonding information in your operating system documentation. STATISTICS Detailed statistics and configuration information can be viewed using the ethtool utility. See the ethtool man page for more information.
User Guide NetXtreme II January 2010 LINUX ISCSI OFFLOAD • User Application - bnx2id • Open iSCSI User Applications • Bind iSCSI Target to Broadcom NX2 iSCSI Transport Name • Making Connections to iSCSI Targets • Maximize Offload iSCSI Connections • Linux iSCSI Offload FAQ USER APPLICATION - BNX2ID Run the bnx2id daemon before attempting to create iSCSI connections. The driver will not be able to establish connections to the iSCSI target without the daemon's assistance.
NetXtreme II User Guide January 2010 BIND ISCSI TARGET TO BROADCOM NX2 ISCSI TRANSPORT NAME By default, the open-iscsi daemon connects to discovered targets using software initiator (transport name = 'tcp'). Users who wish to offload iSCSI connection onto CNIC device should explicitly change transport binding of the iSCSI node. This can be done using iscsiadm cli utility as follows, iscsiadm --mode node --targetname iqn.2004-06.com.broadcom:tg1 \ --portal 192.168.1.100 --op=update \ --name=node.
User Guide NetXtreme II January 2010 MAXIMIZE OFFLOAD ISCSI CONNECTIONS With default driver parameters set, which includes 128 outstanding commands, bnx2i can offload a maximum of 28 iSCSI connections. This is not a hard limit, but just a simple on-chip resource allocation math. bnx2i will be able to offload > 28 connections by reducing the shared queue size which in turn limits the maximum outstanding tasks on a connection.
NetXtreme II User Guide January 2010 NetWare D river Softwa re: Broadcom NetXtreme I I ™ N e t wo r k A da pt e r U s e r G u id e • Overview • Preinstallation Requirements • Installing NetWare Server 6.5 Driver Software • Viewing or Modifying Adapter Properties • Removing Drivers from Autoexec.ncf OVERVIEW This section discusses three NetWare drivers: • BX2.LAN: Broadcom NetXtreme II Gigabit Ethernet 32-bit driver • BX2.COM: Broadcom NetXtreme II Gigabit Ethernet 16-bit driver • BNX2EV.
User Guide NetXtreme II January 2010 INSTALLING NETWARE SERVER 6.5 DRIVER SOFTWARE You may want to create an archive disk by copying all the files from the CDROM\NetWare\Driver directory onto a floppy disk. If you elect to use the CD directly, ensure that the CDROM.NLM file is loaded and that you know the NetWare volume name for the CD. 1. From the NetWare Server console, type HDETECT, and then press ENTER. 2.
NetXtreme II User Guide January 2010 RxBuffers= This is to pre-allocate receive ECBs & Receive adapter resources. This setting may be affected by the NetWare Server maximum/minimum packet receive buffer settings. Min = 32 Max = 1000 Default = 200 Speed= This keyword forces the line speed of the adapter. When this keyword is used, the switch to which the adapter cable is connected must be forced. Otherwise, the adapter and switch may run at different duplex modes.
User Guide NetXtreme II January 2010 TxPacketsPer= Enables allowing an interrupt to occur after a specific amount of packets is transmitted. Min = 0 (Disabled) Max = 100 The default value is 20. RxPacketsPer= Enables allowing an interrupt to occur after a specific amount of packets is received. Min = 0 (Disabled) Max = 100 The default value is 75. CheckSum= Enables or disables the transmit and receive checksum offload feature.
NetXtreme II User Guide January 2010 NODE= A Novell NetWare keyword that allows an input Ethernet node address to replace the adapter factory-programmed Ethernet node address until a subsequent reboot. NODE=nnnnnnnnnnnn FRAME= A Novell NetWare keyword for specifying the frame type. ETHERNET_II ETHERNET_802.3 ETHERNET_802.2 ETHERNET_SNAP The default value is ETHERNET_802.2.
User Guide NetXtreme II January 2010 Spuriousfix= When this keyword is set to 1 (Spurious Fix is ON), the spurious interrupts count that is sometimes displayed on the NetWare console may be reduced. Setting this keyword to 0 (Spurious Fix is OFF) may enhance performance. Spuriousfix = 0 (Spurious Fix is off) Spuriousfix = 1 (Spurious Fix is on) The default value is 1 (Spurious Fix is on). Poll= Disables the interrupt driven mode.
NetXtreme II User Guide January 2010 AUTO (default) FORCE NOTE: If you modify any of the adapter properties, you must reboot the system before the changes take effect. If you make changes and do not reboot, you may experience problems. A valid Autoexec.ncf file is shown below. One set of load and bind commands (in bold) is added for each frame type the adapter is configured to support.
User Guide NetXtreme II January 2010 REMOVING DRIVERS FROM AUTOEXEC.NCF To remove the drivers from the Autoexec.ncf file, locate the load and bind command lines associated with the Broadcom driver and remark them out by inserting the # symbol at the beginning of each command line, or by deleting the statement. Example: # # # # # # # # # LOAD BIND LOAD BIND LOAD BIND LOAD BIND BIND BX2 SLOT=2 FRAME=Ethernet_802.2 NAME=BX2_1_E82 IPX BX2_1_E82 NET=FAFD3D25 BX2 SLOT=2 FRAME=Ethernet_802.
NetXtreme II User Guide January 2010 Wi n dows D r i v e r S of t w a r e : B ro a d c o m N e t X tr e m e I I ™ N e t wo r k A da pt e r U s e r G u id e • Installing the Driver Software • Removing the Device Drivers • Using the NetXtreme II Monolithic Driver • Inserting the NetXtreme II Monolithic Driver in a WinPE 2.
User Guide NetXtreme II January 2010 4. Click Next to continue. 5. After you review the license agreement, click I accept the terms in the license agreement and then click Next to continue. 6. Select how you want to install the NetXtreme II drivers and then click Next. 7. Click Install. 8. Click Finish to close the wizard. 9. The installer will determine if a system restart is necessary. Follow the on-screen instructions. USING SILENT INSTALLATION NOTES: • All commands are case sensitive.
NetXtreme II User Guide January 2010 NOTE: The REINSTALL switch should only be used if the same installer is already installed on the system. If upgrading an earlier version of the installer, use setup /s /v/qn as listed above. To perform a GUI reinstall of the same installer Type the following: setup /V"REINSTALL=ALL" If performing a silent upgrade or uninstall, ensure to do a manual reboot afterwards to avoid leaving the system in an inconsistent state.
User Guide NetXtreme II January 2010 known as Remote Installation Services (RIS), or for the deployment agent used in the Automated Deployment Services (ADS). Therefore, a separate driver was created to accommodate these Microsoft deficiencies. This driver is known as the NetXtreme II monolithic driver, but it is sometimes referred to as the "RIS" driver.
NetXtreme II User Guide January 2010 To insert Broadcom's NetXtreme II monolithic driver in a WinPE 2.0 image, download AIK from http://www.microsoft.com/ downloads/en/default.aspx and install. After installing AIK, copy the latest monolithic driver to a directory on the local hard drive of the system you installed the AIK. Follow the procedure below to insert the monolithic driver into a WinPE 2.0 boot image. 1. From All Programs, open Windows AIK and select Windows PE Tools Command prompt. 2.
User Guide NetXtreme II January 2010 To add a device driver to an offline Windows PE image (Windows Server 2008 R2) This procedure demonstrates how to use the Deployment Image Servicing and Management (DISM) tool to add a device driver (.inf) to an offline Windows PE image. Before running a DISM command, first mount the Windows PE image. 1. Mount the base image by using the DISM tool to a local Windows PE directory. For example: Dism /Mount-WIM /WimFile:c:\winpe_x86\winpe.
NetXtreme II User Guide January 2010 • Auto (1 Gbps is enabled when that speed is supported) = "0" • 10 Mbps Half Duplex = "65794" • 10 Mbps Full Duplex = "258" • 100 Mbps Half Duplex = "66050" • 100 Mbps Full Duplex = "514" Options for fiber interfaces: • Auto (1 Gbps is enabled when that speed is supported) = "0" • 1 Gbps Full Duplex = "771" • Auto with 1 Gbps Fallback = "33539" • Hardware default = "65283" An example is provided in the file showing how to configure a copper interface
User Guide NetXtreme II January 2010 NOTES: • The Power Management tab is available only for servers that support power management. • To enable Wake on LAN (WOL) when the computer is on standby, click Allow the device to bring the computer out of standby box. • If you select Only allow management stations to bring the computer out of standby, the computer can be brought out of standby only by Magic Packet.
NetXtreme II User Guide January 2010 U s i n g B ro a d c o m A dvan c e d C o nt rol S u i t e 3 : Broadcom Ne tXtrem e II™ N etwor k Adapter Use r G ui de • Broadcom Advanced Control Suite 3 Overview • Starting Broadcom Advanced Control Suite 3 • Using Broadcom Advanced Control Suite 3 • Configuring Teaming • Configuring With the Command Line Interface Utility BROADCOM ADVANCED CONTROL SUITE 3 OVERVIEW Broadcom Advanced Control Suite 3 (BACS 3) is an integrated utility that provides useful inf
User Guide NetXtreme II January 2010 STARTING BROADCOM ADVANCED CONTROL SUITE 3 In Control Panel, click Broadcom Control Suite 3. Microsoft .NET Framework 2.0 includes the runtime and associated files needed to run BACS 3, and must be installed on your system in order for BACS 3 to operate. For information on the minimum and recommended .NET Framework versions for your operating system, see Table 1. NOTES: • Starting BACS 3 without .NET Framework (version 2.
NetXtreme II User Guide January 2010 menu options, and device information are only available based on the types of devices installed or the item selected in the interface. Two buttons at the bottom of the interface control what you want to manage: click the Device Management button to manage the connected devices, or click the Team Management button to create and manage teams. Device Management The Device Management pane lists the devices that can be viewed, analyzed, tested, or adjusted by BACS 3.
User Guide NetXtreme II January 2010 The Connection View lists the server and client devices in a parent/child hierarchy, which makes it easier to see the relationships between the adapters. Just as in the Device Management pane, selecting items in the Connection View shows the options available for that item. Device Icons The icon next to each device in the Device Management pane shows its status. An icon next to a device name that appears normal means the device is connected and working. • X.
NetXtreme II User Guide January 2010 To create or edit teams • See Configuring Teaming. CONFIGURING BACS USER INTERFACE OPTIONS Enabling/Disabling the BACS Tray Icon BACS 3 places an icon in the Windows taskbar when the program is installed. Use the Options window to turn this icon on or off. To enable or disable the BACS tray icon 1. From the Tools menu, select Options. 2. In the Options window, select General. 3. Select or clear Enable Tray Icon (the option is enabled by default). 4. Click OK.
User Guide NetXtreme II January 2010 2. In the Options window, select Appearance. 3. Choose the appearance settings. Click Apply to set changes. Click OK to set changes and close the Options window. To reset the interface to the default appearance 1. From the Tools menu, select Options. 2. In the Options window, select General. 3. Click Restore Default. 4. Close and then restart BACS 3.
NetXtreme II User Guide January 2010 MAC Address. A physical MAC (media access control) address that is assigned to the adapter by the manufacturer. The physical address is never all 0s. Permanent MAC Address. The unique hardware address assigned to the network adapter. iSCSI MAC Address. If an iSCSI network adapter is loaded onto the system, this parameter will display the iSCSI MAC address. IPv4 DHCP. The IP address is from a DHCP server if the value is Enable. IP Address.
User Guide NetXtreme II January 2010 Offload Capabilities. The offload capabilities supported by the adapter. This information is only available for Broadcom NetXtreme II adapters. • TOE. TCP Offload Engine (TOE) allows simultaneous operation of up to 1024 fully offloaded TCP connections for 1-Gbps network adapters and 1880 fully offloaded TCP connections for 10-Gbps network adapters to the hardware. • iSCSI. iSCSI offload for block-level transfer of data. • LSO.
NetXtreme II User Guide January 2010 Driver Status. The status of the adapter driver. • Loaded. Normal operating mode. The adapter driver has been loaded by Windows and is functioning. • Not Loaded. The driver associated with the adapter has not been loaded by Windows. • Information Not Available. The value is not obtainable from the driver that is associated with the adapter. Driver Name. The file name of the adapter driver. Driver Version. The current version of the adapter driver. Driver Date.
User Guide NetXtreme II January 2010 window, select Information and then select Resources. NOTE: Some information may not be available for all Broadcom network adapters. Bus Type. The type of input/output (I/O) interconnect used by the adapter. Bridge. The bridge type, which is the PCI-E to PCI-X bridge. This information is only available for Broadcom NetXtreme II adapters. Bridge Lanes. The number of PCI-E lanes connected to the bridge.
NetXtreme II User Guide January 2010 Bus No. Indicates the number of the bus where the adapter is installed. Device No. The number assigned to the adapter by the operating system. Function No. The port number of the adapter. For a single-port adapter, the function number is 0. For a two-port adapter, the function number for the first port is 0, and the function number for the second port is 1. Interrupt Request. The interrupt line number that is associated with the adapter.
User Guide NetXtreme II January 2010 VIEWING HARDWARE INFORMATION The Hardware section of the Information tab displays information about the hardware settings for the selected network adapter. To view Hardware for any installed network adapter, click the name of the adapter listed in the Device Management pane, then click the Information tab. • If the Information tab is not visible, then from the View menu, select Navigate, then Device Management, and then Information.
NetXtreme II User Guide January 2010 Vendor ID. The vendor ID. Device ID. The adapter ID. Subsystem Vendor ID. The subsystem vendor ID. Subsystem ID. The subsystem ID. External PHY Firmware Version. The external PHY firmware version. TESTING THE NETWORK The Network Test option on the Diagnostics tab lets you verify IP network connectivity. This test verifies if the driver is installed correctly and tests connectivity to a gateway or other specified IP address on the same subnet.
User Guide NetXtreme II January 2010 Bro adco m Co rp or atio n Document ENGSRVT52-CDUM100-R Using Broadcom Advanced Control Suite 3 Page 169
NetXtreme II User Guide January 2010 RUNNING DIAGNOSTIC TESTS The Diagnostic Tests option on the Diagnostics tab lets you check the state of the physical components on a Broadcom network adapter. You can trigger the tests manually, or choose to have BACS 3 continuously perform them. If the test are performed continuously, then the number of passes and fails in the Status field for each test increments every time the tests are performed.
User Guide NetXtreme II January 2010 Control Registers. This test verifies the read and write capabilities of the network adapter registers by writing various values to the registers and verifying the results. The adapter driver uses these registers to perform network functions such as sending and receiving information. A test failure indicates that the adapter may not be working properly. MII Registers. This test verifies the read and write capabilities of the registers of the physical layer (PHY).
NetXtreme II User Guide January 2010 Test LED. This test causes all of the port LEDs to blink 5 times for the purpose of identifying the adapter. ANALYZING CABLES The Cable Analysis option on the Diagnostics tab lets you monitor the conditions of each wire pair in an Ethernet Category 5 cable connection within an Ethernet network. The analysis measures the cable quality and compares it against the IEEE 802.3ab specification for compliance.
User Guide NetXtreme II January 2010 Distance. The valid cable length in meters (except when the Noise result is returned). Status. The result of the analysis for the indicated pair. • Good. Good cable/PCB signal paths, but no gigabit link. • Crossed. Pin short or crosstalk along two or more cable/PCB signal paths. • Open. One or both pins are open for a twisted pair. • Short. Two pins from the same twisted pair are shorted together. • Noise.
NetXtreme II User Guide January 2010 SETTING ADAPTER PROPERTIES Advanced and iSCSI Boot Configuration on the Configurations tab allow you to view and change the values of the available properties of the selected adapter. The potentially available properties and their respective settings are described below. NOTES: • You must have administrator privileges to change the values for a property. • The list of available properties for your particular adapter may be different.
User Guide NetXtreme II January 2010 802.1p QOS. Enables quality of service, which is an Institute of Electrical and Electronics Engineering (IEEE) specification that treats different types of network traffic diversely to ensure required levels or reliability and latency according to the type of traffic. This property is disabled by default. Unless the network infrastructure supports QoS, do not enable this property. Otherwise, problems may occur. Ethernet@Wirespeed.
NetXtreme II User Guide January 2010 Flow Control. Enables or disables the receipt or transmission of PAUSE frames. PAUSE frames allow the network adapter and a switch to control the transmit rate. The side that is receiving the PAUSE frame momentarily stops transmitting. By enabling TOE, network performance improves, but with the increased performance, TOE performance is more susceptible to packet loss when flow control is disabled. Enable flow control to reduce the number of packets lost.
User Guide NetXtreme II January 2010 • The range is 00:00:00:00:00:01 to FF:FF:FF:FF:FF:FD. • Do not use a multicast address (least significant bit of the high byte = 1). • Do not use all 0s or all Fs. Receive Side Scaling. Allows configuring network load balancing across multiple CPUs. The default setting for this property is Enabled. NOTE: For all network adapters, IPv6 Receive Side Scaling is not supported on Windows Server 2003 due to a limitation in the operating system.
NetXtreme II User Guide January 2010 • Auto (default). Sets the speed to auto-negotiate with its link partner at the highest matching speed. • Auto with 1Gb Fallback Full. Sets the speed to auto-negotiate with its link partner, but if the attached link partner is forced at 1 Gbit/s, it will fall back to this mode. • Hardware Default. Sets the speed to negotiate according to the setting specified by the manufacturer (see manufacturer documentation for more information). iSCSI Crash Dump.
User Guide NetXtreme II January 2010 VIEWING STATISTICS The information provided on the Statistics tab allows you to view traffic statistics for both Broadcom network adapters and network adapters made by others. Statistical information and coverage are more comprehensive for Broadcom adapters. To view Statistics information for any installed network adapter, click the name of the adapter listed in the Device Management pane, then click the Statistics tab.
NetXtreme II User Guide January 2010 General Statistics General Statistics show the transmitted and received statistics to and from the adapter. Frames Tx OK. A count of the frames that were successfully transmitted. This counter is incremented when the transmit status is reported as Transmit OK. Frames Rx OK. A count of the frames that were successfully received.
User Guide NetXtreme II January 2010 Local Initiator login failures. The number of login failures likely caused by the initiator. Initiator Instance Statistics. The statistics in this area pertain to all sessions. Session digest errors. The number of sessions with errors due to an invalid payload or header. Session connection timeout error. The number of sessions that were terminated due to any of the many timeout errors. Session format error.
NetXtreme II User Guide January 2010 Frames Tx with more than one Collision. A count of the frames that were involved in more than one collision and were subsequently transmitted successfully. This counter is incremented when the transmit status is reported as Transmit OK, and the value of the attempts variable is greater than 2 and less than or equal to the attempt limit. Frames Tx after Deferral. A count of the frames that were delayed being transmitted on the first attempt because the medium was busy.
User Guide NetXtreme II January 2010 To view resource reservations 1. Click the name of the Broadcom NetXtreme II system device in the Device Management pane. 2. Click the Configurations tab. If the Configurations tab is not visible, then from the View menu, select Navigate, then Device Management, and then Configurations. 3. From the Resource Reservations section, select the property you want to set.
NetXtreme II User Guide January 2010 The Configurations tab for 10 Gbps network adapters is shown below.
User Guide NetXtreme II January 2010 Configuring the IP Address for iSCSI Offload For iSCSI-booted adapters, the Configurations tab is not available and you will not be able to perform this procedure. To set the IP address of the iSCSI HBA for iSCSI offload The iSCSI Management section of the Configurations tab allows you to set the IP address of the iSCSI HBA when using iSCSI protocol to offload network processing from the CPU to the Broadcom network adapter. 1.
NetXtreme II User Guide January 2010 VIEWING LICENSES NOTES: • The Licenses section of the Configurations tab is only available for Broadcom NetXtreme II adapters and VBD drivers. • Not all offload technologies are available with all adapters. The Licenses section shows the number of connections available for TOE and iSCSI offload technologies. To view licenses 1. Click the name of the Broadcom NetXtreme II system device in the Device Management pane. 2. Click the Configurations tab.
User Guide NetXtreme II January 2010 CONFIGURING TEAMING The teaming function allows you to group any available network adapters together to function as a team. Teaming is a method of creating a virtual NIC (a group of multiple adapters that functions as a single adapter). The benefit of this approach is that it enables load balancing and failover. Teaming is done through the Broadcom Advanced Server Program (BASP) software.
NetXtreme II User Guide January 2010 TEAM TYPES You can create four types of load balance teams: • Smart Load Balance and Failover • Link Aggregation (802.3ad) (TOE is not applicable) • Generic Trunking (FEC/GEC)/802.3ad-Draft Static (TOE is not applicable) • SLB (Auto-Fallback Disable) – The Auto-Fallback Disable feature is configured for Smart Load Balance and Failover type teams in the Teaming Wizard. NOTE: NetXtreme II network adapters with iSCSI enabled is supported only in an SLB team type.
User Guide NetXtreme II January 2010 Device Manager or Hot-Plug PCI. If any primary adapter assigned to a team is disabled, the team functions as a Smart Load Balancing and Failover type of team in which auto-fallback occurs. The LiveLink feature is supported for this type of team. STANDBY TEAM MEMBER AND AUTO-FALLBACK DISABLE MODE You can designate one team member in an SLB type of team to be the standby member.
NetXtreme II User Guide January 2010 NOTE: If you prefer to work without the wizard for now, click Expert Mode. If you want to always use Expert Mode to create a team, select Default to Expert Mode on next start. See Using Expert Mode. 4. To continue using the wizard, click Next.
User Guide NetXtreme II January 2010 5. Type the team name and then click Next. If you want to review or change any of your settings, click Back. Click Cancel to discard your settings and exit the wizard. NOTE: The team name cannot exceed 39 characters, cannot begin with spaces, and cannot contain any of the following characters: & \ / : * ? < > | 6. Select the type of team you want to create. If the team type is an SLB type team, click Next.
NetXtreme II User Guide January 2010 Bro adco m C orp or atio n Page 192 Configuring Teaming Document ENGSRVT52-CDUM100-R
User Guide NetXtreme II January 2010 7. From the Available Adapters list, click the adapter you want to add to the team and then click Add. Remove team members from the Team Members list by clicking the adapter and then clicking Remove. Click Next. NOTE: There must be at least one Broadcom network adapter assigned to the team.
NetXtreme II User Guide January 2010 10. If you want to configure LiveLink, select Yes, otherwise select No, then click Next. 11. Select the probe interval (the number of seconds between each retransmission of a link packet to the probe target) and the maximum number of probe retries (the number of consecutively missed responses from a probe target before a failover is triggered). 12. Set the Probe VLAN ID to allow for connectivity with probe targets residing on a tagged VLAN.
User Guide NetXtreme II January 2010 created with an identical VLAN tag value (see Step 18.). 13. Click the probe target at the top of the list, click Edit Target IP Address, type the target IP address in the IP Address box for one or all probe targets, and then click OK. Click Next. NOTE: Only the first probe target is required. You can specify up to three additional probe targets to serve as backups by assigning IP addresses to the other probe targets. 14.
NetXtreme II User Guide January 2010 15. If you want to create a VLAN on the team, select Add VLAN, or if you want to change the settings of an existing VLAN, select Edit VLAN, then click Next. If you do not want to create or edit a VLAN, select Skip Manage VLAN, then click Next, and continue with the wizard from the Finish screen (see Step 20. of this procedure). VLANs enable you to add multiple virtual adapters that are on different subnets.
User Guide NetXtreme II January 2010 16. Type the VLAN name and then click Next. NOTE: The team name cannot exceed 39 characters, cannot begin with spaces, and cannot contain any of the following characters: & \ / : * ? < > | 17. To tag the VLAN, select Tagged and then click Next. Otherwise, click Untagged, click Next, and continue with the wizard to add additional VLANs (see Step 19. of this procedure). 18. Type the VLAN tag value and then click Next. The value must be between 1 and 4094.
NetXtreme II User Guide January 2010 19. Select Yes to add or manage another VLAN and then click Next. Repeat until you do not want to add or manage any additional VLANs. NOTE: You can define up to 64 VLANs per team (63 VLANs that are tagged and 1 VLAN that is not tagged). Adding several VLANS may slow down the reaction time of the Windows interface due to memory and processor time usage for each VLAN. The degree to which Windows performance may suffer depends on system configuration.
User Guide NetXtreme II January 2010 20. To apply and commit the changes to the team, select Commit changes to system and Exit the wizard. To apply your changes but continue using the wizard, select Save changes and continue to manage more teams. Click Finish. NOTE: At any point in the Broadcom Teaming Wizard procedure, click Preview to get a visual representation of what the team will look like before committing any changes. 21.
NetXtreme II User Guide January 2010 USING EXPERT MODE Use Expert Mode to create a team, modify a team, add a VLAN, and configure LiveLink for a Smart Load Balance and Failover and SLB (Auto-Fallback Disable) team. To create a team using the wizard, see Using the Broadcom Teaming Wizard. To set the default Teaming Mode, select Options from the Tools menu. In the Options window, click the General tab, then select Expert Mode or Wizard Mode (the default is Wizard Mode).
User Guide NetXtreme II January 2010 4. Click Expert Mode. NOTE: If you want to always use Expert Mode to create a team, click Default to Expert Mode on next start. 5. Click the Create Team tab. NOTE: The Create Team tab appears only if there are teamable adapters available. 6. Click the Team Name field to enter a team name. 7. Click the Team Type field to select a team type. 8. Click Manage Members at the top of the window.
NetXtreme II User Guide January 2010 9. Assign any available adapter or adapters to the team by moving the adapter from the Available Adapters list to the Load Balance Members list. There must be at least one adapter in the Load Balance Members list. 10. You can assign any other available adapter to be a standby member by selecting it from the Standby Member list. NOTE: There must be at least one Broadcom network adapter assigned to the team.
User Guide NetXtreme II January 2010 • The maximum number of team members is 8. • When team configuration has been correctly performed, a virtual team adapter driver is created for each configured team. • If you disable a virtual team and later want to reenable it, you must first disable and reenable all team members before you reenable the virtual team. • When you create Generic Trunking and Link Aggregation teams, you cannot designate a standby member.
NetXtreme II User Guide January 2010 6. Make the desired changes, and then click Update. The changes have not yet been applied; click the Preview tab to view the updated team structure before applying the changes. 7. Click Apply/Exit to apply the updates and exit the Manage Teams window. 8. Click Yes when the message is displayed indicating that the network connection will be temporarily interrupted. Adding a VLAN You can add virtual LANs (VLANs) to a team.
User Guide NetXtreme II January 2010 7. Type the VLAN name, then select the type and ID. 8. Click Create to save the VLAN information. As VLANs are defined, they can be selected from the Team Name list, but they have not yet been created. 9. Continue this process until all VLANs are defined, then click OK to create them. 10. Click Yes when the message is displayed indicating that the network connection will be temporarily interrupted.
NetXtreme II User Guide January 2010 NOTE: If you delete a team, any VLANs configured for that team are also deleted. Configuring LiveLink for a Smart Load Balancing and Failover and SLB (Auto-Fallback Disable) Team LiveLink is a feature of BASP that is available for the Smart Load Balancing (SLB) and SLB (Auto-Fallback Disable) type of teaming. The purpose of LiveLink is to detect link loss beyond the switch and to route traffic only through team members that have a live link.
User Guide NetXtreme II January 2010 10. Select one of the listed team members and type the member IP address. NOTE: All of the member IP addresses must be in the same subnet as the probe targets. 11. Click Update. Repeat these steps for each of the other listed team members. 12. Click Apply/Exit. Saving and Restoring a Configuration To save a configuration 1. From the File menu, select Team Save As. 2. Type the path and file name of the new configuration file, and then click Save (a .
NetXtreme II User Guide January 2010 5. If a configuration is already loaded, a message is displayed that asks if you want to save your current configuration. Click Yes to save the current configuration. Otherwise, the configuration data that is currently loaded is lost. VIEWING BASP STATISTICS The Statistics section shows performance information about the network adapters that are on a team.
User Guide NetXtreme II January 2010 CONFIGURING WITH THE COMMAND LINE INTERFACE UTILITY An alternate method to BACS 3 for configuring Broadcom network adapters is with BACSCLI, which is a Broadcom utility that allows you to view information and configure network adapters using a console in either a non-interactive command line interface (CLI) mode or an interactive mode.
NetXtreme II User Guide January 2010 S p e c if i c a t io n s : B ro adcom NetXtreme II™ Network Adapter Us er Guide • 10/100/1000BASE-T and 10GBASE-T Cable Specifications • 1000/2500BASE-X Fiber Optic Specifications • Interface Specifications • NIC Physical Characteristics • NIC Power Requirements • Wake On LAN Power Requirements • Environmental Specifications 10/100/1000BASE-T AND 10GBASE-T CABLE SPECIFICATIONS Table 1: 10/100/1000BASE-T Cable Specifications Port Type Connector Media M
User Guide NetXtreme II January 2010 1000/2500BASE-X FIBER OPTIC SPECIFICATIONS Table 3: 1000/2500BASE-X Fiber Optic Specifications Port Type Connector Media Maximum Distance 1000BASE-X Small form-factor pluggable (SFP) transceiver with LC™ connection system (Infineon p/n V23818-K305L57) Multimode fiber (MMF) System optimized for 62.
NetXtreme II User Guide January 2010 NIC PHYSICAL CHARACTERISTICS Table 6: NIC Physical Characteristics NIC Type NIC Length NIC Width BCM5708 PCI Express 14.7 cm (5.79 inches) 6.4 cm (2.52 inches) BCM5709 PCI Express x4 low-profile 11.9 cm (4.7 inches) 6.9 cm (2.7 inches) BCM57710/BCM57711 PCI Express x8 low profile 16.8 cm (6.6 inches) 5.1 cm (2.0 inches) NIC POWER REQUIREMENTS Table 7 shows the power requirements for the BCM5708C add-in NIC.
User Guide NetXtreme II January 2010 WAKE ON LAN POWER REQUIREMENTS Table 10: BCM5708C Wake On LAN Power Requirements (Nominal Conditions) 100 Mbit Link NIC 3.3V Current (mA) 10 Mbit Link NIC Power (W) 236 NIC 3.3V Current (mA) NIC Power (W) 150 0.5 0.78 Table 11: BCM5709C Wake On LAN Power Requirements (Nominal Conditions) 100 Mbit Link NIC 3.3V Current (mA) 10 Mbit Link NIC Power (W) 0 NIC 3.3V Current (mA) NIC Power (W) 0 0.85 0.
NetXtreme II User Guide January 2010 R e gu l a t o r y I n for m a t i on : Broadcom NetXtreme I I ™ N e t wo r k A da pt e r U s e r G u id e • FCC Notice • VCCI Notice • CE Notice • Canadian Regulatory Information (Canada Only) • Korea Communications Commission (KCC) Notice (Republic of Korea Only) FCC NOTICE FCC, CLASS B Broadcom NetXtreme II Gigabit Ethernet Controller Broadcom Corporation 190 Mathilda Place Sunnyvale, California 94086 USA The equipment complies with Part 15 of the FCC Rule
User Guide NetXtreme II January 2010 FCC, CLASS A Broadcom NetXtreme II Gigabit Ethernet Controller Broadcom NetXtreme II 10 Gigabit Ethernet Controller Broadcom Corporation 190 Mathilda Place Sunnyvale, California 94086 USA This device complies with Part 15 of the FCC Rules. Operations is subject to the following two conditions: 1) This device may not cause harmful interference, and 2) This device must accept any interference received, including interference that may cause undesired operation.
NetXtreme II User Guide January 2010 The equipment is a Class B product based on the standard of the Voluntary Control Council for Interference from Information Technology Equipment (VCCI). If used near a radio or television receiver in a domestic environment, it may cause radio interference. Install and use the equipment according to the instruction manual.
User Guide NetXtreme II January 2010 CLASS A Broadcom NetXtreme II Gigabit Ethernet Controller Broadcom NetXtreme II 10 Gigabit Ethernet Controller Broadcom Corporation 190 Mathilda Place Sunnyvale, California 94086 USA This equipment is a Class A product based on the standard of the Voluntary Control Council for interference by Information Technology Equipment (VCCI). If used in a domestic environment, radio disturbance may arise. Install and use the equipment according to the instruction manual.
NetXtreme II User Guide January 2010 Bro adco m C orp or atio n Page 218 CE Notice Document ENGSRVT52-CDUM100-R
User Guide NetXtreme II January 2010 Bro adco m Co rp or atio n Document ENGSRVT52-CDUM100-R CE Notice Page 219
NetXtreme II User Guide January 2010 Bro adco m C orp or atio n Page 220 CE Notice Document ENGSRVT52-CDUM100-R
User Guide NetXtreme II January 2010 Bro adco m Co rp or atio n Document ENGSRVT52-CDUM100-R CE Notice Page 221
NetXtreme II User Guide January 2010 Bro adco m C orp or atio n Page 222 CE Notice Document ENGSRVT52-CDUM100-R
User Guide NetXtreme II January 2010 CANADIAN REGULATORY INFORMATION (CANADA ONLY) INDUSTRY CANADA, CLASS B Broadcom NetXtreme II Gigabit Ethernet Controller Broadcom Corporation 190 Mathilda Place Sunnyvale, California 94086 USA This Class B digital apparatus complies with Canadian ICES-003. Notice: The Industry Canada regulations provide that changes or modifications not expressly approved by Broadcom could void your authority to operate this equipment.
NetXtreme II User Guide January 2010 INDUSTRY CANADA, CLASSE B Broadcom NetXtreme II Gigabit Ethernet Controller Broadcom Corporation 190 Mathilda Place Sunnyvale, California 94086 USA Cet appareil numérique de la classe B est conforme à la norme canadienne ICES-003. Avis : Dans le cadre des réglementations d'Industry Canada, vos droits d'utilisation de cet équipement peuvent être annulés si des changements ou modifications non expressément approuvés par Broadcom y sont apportés.
User Guide NetXtreme II January 2010 KOREA COMMUNICATIONS COMMISSION (KCC) NOTICE (REPUBLIC OF KOREA ONLY) B CLASS DEVICE Broadcom NetXtreme II Gigabit Ethernet Controller Broadcom Corporation 190 Mathilda Place Sunnyvale, California 94086 USA Note that this device has been approved for non-business purposes and may be used in any environment, including residential areas.
NetXtreme II User Guide January 2010 A CLASS DEVICE Broadcom NetXtreme II Gigabit Ethernet Controller Broadcom NetXtreme II 10 Gigabit Ethernet Controller Broadcom Corporation 190 Mathilda Place Sunnyvale, California 94086 USA Bro adco m C orp or atio n Page 226 Korea Communications Commission (KCC) Notice (Republic of Korea Only) Document ENGSRVT52-CDUM100-R
User Guide NetXtreme II January 2010 U s e r D i agn o s t i c s : B ro a d c o m N e tXt r e m e I I ™ Network Adapter Us er Guide • Introduction • System Requirements • Performing Diagnostics • Diagnostic Test Descriptions INTRODUCTION Broadcom NetXtreme II User Diagnostics is an MS-DOS based application that runs a series of diagnostic tests (see Table 3) on the Broadcom NetXtreme II network adapters in your system.
NetXtreme II User Guide January 2010 PERFORMING DIAGNOSTICS At the MS-DOS prompt, type uxdiag (for BCM5706/BCM5708/BCM5709 network adapters) or uediag (for BCM57710 network adapters) followed by the command options. The uxdiag command options are shown in Table 1 and the uediag command options are shown in Table 2.
User Guide NetXtreme II January 2010 Table 1: uxdiag Command Options (Cont.) Command Options Description uxdiag -mbas Sets the MBA/PXE speed. 0 = Auto 1 = 10H 2 = 10F 3 = 100H 4 = 100F 6 = 1000F uxdiag -mbav <1|0> Enables/disables MBA VLAN. 1 = Enable 0 = Disable uxdiag -mbavval Sets MBA VLAN (<65536). uxdiag -mfw <1/0> Enables/disables management firmware. 1 = Enable 0 = Disable uxdiag -t Disables certain groups/tests.
NetXtreme II User Guide January 2010 Table 2: uediag Command Options Command Options Description uediag Performs all tests on all Broadcom NetXtreme II adapters in your system. uediag -c Specifies the adapter (device#) to test. Similar to -dev (for backward compatibility). uediag -cof Allows tests to continue after detecting a failure. uediag -dev Specifies the adapter (device#) to test. uediag -F Forces an upgrade of the image without checking the version.
User Guide NetXtreme II January 2010 Table 2: uediag Command Options (Cont.) Command Options Description uediag -wol <1/0> Enables/disable Magic Packet WOL.
NetXtreme II User Guide January 2010 DIAGNOSTIC TEST DESCRIPTIONS The diagnostic tests are divided into four groups: Basic Functional Tests (Group A), Memory Tests (Group B), Block Tests (Group C), and Ethernet Traffic Tests (Group D). The diagnostic tests are listed and described in Table 3.
User Guide NetXtreme II January 2010 Table 3: Diagnostic Tests (Cont.) Test Description Number Name C1 CPU Logic and DMA Interface Verifies the basic logic functionality of all the on-chip CPUs. It also exercises the DMA interface exposed to those CPUs. The internal CPU tries to initiate DMA activities (both read and write) to system memory and then compares the values to confirm that the DMA operation completed successfully.
NetXtreme II User Guide January 2010 Table 3: Diagnostic Tests (Cont.) Test Description Number Name D4 LSO Verifies the functionality of the adapter’s Large Send Offload (LSO) support by enabling MAC loopback mode and transmitting large TCP packets. As the packets are received back by Broadcom NetXtreme II User Diagnostics, they are checked for proper segmentation (according to the selected MSS size) and any other errors. The adapter should not be connected to a network.
User Guide NetXtreme II January 2010 Tro ubl e s ho ot i n g: B ro a d c o m N e tXt r e m e I I ™ Network Adapter Us er Guide • Hardware Diagnostics • Checking Port LEDs • Troubleshooting Checklist • Checking if Current Drivers are Loaded • Running a Cable Length Test • Testing Network Connectivity • Microsoft Virtualization with Hyper-V • Removing the Broadcom NetXtreme II Device Drivers • Upgrading Windows 2000 Server to Windows Server 2003 • Upgrading Windows Operating Systems •
NetXtreme II User Guide January 2010 TROUBLESHOOTING CHECKLIST CAUTION! Before you open the cabinet of your server to add or remove the adapter, review Safety Precautions. The following checklist provides recommended actions to take to resolve problems installing the Broadcom NetXtreme II adapter or running it in your system. • Inspect all cables and connections. Verify that the cable connections at the network adapter and the switch are attached properly.
User Guide NetXtreme II January 2010 LAN Protocol: ARP (see note) LAN Protocol: IP Addr: (see note) NOTE: The LAN protocol status is displayed after an IP address is assigned to the adapter. LINUX To verify that the bnx2.o driver is loaded properly, run: lsmod If the driver is loaded, a line similar to the one below is displayed, where size is the size of the driver in bytes, and n is the number of adapters configured.
NetXtreme II User Guide January 2010 NETWARE Ping an IP host on the network to verify connection has been established. From the command line, type ping IP address, and then press ENTER. The ping statistics that are displayed indicate whether or not the network connection is working. LINUX To verify that the Ethernet interface is up and running, run ifconfig to check the status of the Ethernet interface. It is possible to use netstat -i to check the statistics on the Ethernet interface.
User Guide NetXtreme II January 2010 MICROSOFT VIRTUALIZATION WITH HYPER-V Microsoft Virtualization is a hypervisor virtualization system for Windows Server 2008 and Windows Server 2008 R2. This section is intended for those who are familiar with Hyper-V, and it addresses issues that affect the configuration of NetXtreme II network adapters and teamed network adapters when Hyper-V is used. For more information on Hyper-V, see http://www.microsoft.com/windowsserver2008/en/us/hyperv.aspx.
NetXtreme II User Guide January 2010 SINGLE NETWORK ADAPTER Windows Server 2008 When configuring a NetXtreme II network adapter on a Hyper-V system, be aware of the following: • An adapter that is to be bound to a virtual network should not be configured for VLAN tagging through the driver’s advanced properties. Instead, Hyper-V should manage VLAN tagging exclusively.
User Guide NetXtreme II January 2010 TEAMED NETWORK ADAPTERS Table 2 identifies Hyper-V supported features that are configurable for NetXtreme II teamed network adapters. This table is not an all-inclusive list of Hyper-V features. Table 2: Configurable Teamed Network Adapter Hyper-V Features Supported in Windows Server 2008 Supported in Windows Server 2008 R2 Smart Load Balancing and Failover (SLB) team type Yes Yes Multi-member SLB team allowed with latest BASP6 version.
NetXtreme II User Guide January 2010 Windows Server 2008 When configuring a team of NetXtreme II network adapters on a Hyper-V system, be aware of the following: • Create the team prior to binding the team to the Hyper-V virtual network. • Create a team only with an adapter that is not already assigned to a Hyper-V virtual network. • A TOE-enabled team that is bound to a Hyper-V virtual network will report TOE as an offload capability in BACS; however, TOE will not work.
User Guide NetXtreme II January 2010 REMOVING THE BROADCOM NETXTREME II DEVICE DRIVERS Uninstall the Broadcom NetXtreme II device drivers from your system only through the InstallShield wizard. Uninstalling the device drivers with Device Manager or any other means may not provide a clean uninstall and may cause the system to become unstable. For information on uninstalling Broadcom NetXtreme II device drivers, see Removing the Device Drivers.
NetXtreme II User Guide January 2010 For an unattended installation, place the Broadcom NetXtreme II driver files in the $OEM$\$1\Drivers\NIC directory. For Sysprep, the drivers are located in Drivers\NIC at the root of the system drive. The Broadcom NetXtreme II driver files are listed below according to the operating system to be installed: Windows 2000: bxvbd.inf, bxvbdx.sys, bxvbd.cat, bxnd.inf, bxnd.cat, bxnd50x.sys, bxndcox.dll, bxdiag.cat, bxdiag.inf, bxdiagx.sys, wdfcoinstaller01005.
User Guide NetXtreme II January 2010 Problem: A BASP team in Windows 2003 may not function properly if a team member driver property is modified. Solution: Due to a limitation in Windows 2003, the features of team members should remain static during the entire life of the team. To change the characteristics of a team member, remove the team member from the team, modify the team member, and then re-add the adapter to the team. The limitation does not exist in Windows 2008 and later.
NetXtreme II User Guide January 2010 LINUX Problem: On kernels older than 2.6.16 when 16 partitions are created on a server containing two BCM57711 network adapters, not all partitions would come up and an error indicating a shortage of space would display. Solution: On architectures where the default vmalloc size is relatively small and not sufficient to load many interfaces, use vmalloc= during boot to increase the size.
User Guide NetXtreme II January 2010 Problem: The Broadcom NetXtreme II adapter may not perform at optimal level on some systems if it is added after the system has booted. Solution: The system BIOS in some systems does not set the cache line size and the latency timer if the adapter is added after the system has booted. Reboot the system after the adapter has been added. Problem: Intelligent Platform Management Interface (IPMI) is not functioning properly.
