Front cover Implementing IBM System Networking 10Gb Ethernet Switches Introduction to IBM System Networking RackSwitch hardware Sample network design and implementation Switch troubleshooting and maintenance Sangam Racherla Ionut Bunea David A Gray Sebastian Oglaza Jose Rodriguez Ruibal ibm.
International Technical Support Organization Implementing IBM System Networking 10Gb Ethernet Switches June 2012 SG24-7960-00
Note: Before using this information and the product it supports, read the information in “Notices” on page ix. First Edition (June 2012) This edition applies to IBM System Networking 10Gb Top-of-Rack, and Embedded Switches from the IBM System Networking portfolio of products. © Copyright International Business Machines Corporation 2012. All rights reserved. Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.
Contents Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix Trademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .x Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi The team who wrote this book . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.1.2 VLANs and Port VLAN ID numbers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 2.1.3 Protocol-based VLANs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 2.2 Spanning Tree Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 2.2.1 Rapid Spanning Tree Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 2.2.
Chapter 3. Reference architectures. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 Overview of the reference architectures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Top-of-Rack architecture. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.1 Layer 1 architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.2 Layer 2 architecture . . . . . . . .
Chapter 6. IBM Virtual Fabric 10Gb Switch Module implementation . . . . . . . . . . . . . 6.1 Purpose of this implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2 Stacking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2.1 Stacking overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2.2 Stacking requirements . . . . . . . . . . . .
Locating the web material . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Other publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Online resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
viii Implementing IBM System Networking 10Gb Ethernet Switches
Notices This information was developed for products and services offered in the U.S.A. IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used.
Trademarks IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. These and other IBM trademarked terms are marked on their first occurrence in this information with the appropriate symbol (® or ™), indicating US registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries.
Preface In today’s infrastructure, it is common to build networks based on 10 Gb Ethernet technology. The IBM® portfolio of 10 Gb systems networking products includes Top-of-Rack switches, and the embedded switches in the IBM BladeCenter® family. In 2010, IBM formed the IBM System Networking business (by acquiring BLADE Network Technologies), which is now focused on driving data center networking by using the latest Ethernet technologies.
Sangam Racherla is an IT Specialist and Project Leader working at the ITSO in San Jose, CA. He has 12 years of experience in the IT field and has been with the ITSO for the past eight years. Sangam has extensive experience in installing and supporting the ITSO lab equipment for various IBM Redbooks projects. He has expertise in working with Microsoft Windows, Linux, IBM AIX®, IBM System x®, and IBM System p® servers, and various SAN and storage products.
Figure 1 shows the team. Figure 1 Jose, David, Sangam, Sebastian, and Ionut Thanks to the following people for their contributions to this project: Ann Lund, Jon Tate, David Watts International Technical Support Organization, San Jose Nghiem V. Chu, Kam-Yee (Johnny) Chung, Michael Easterly, David Faircloth, David Iles, Jeffery M. Jaurigui, Harry W. Lafnear, Lan T. Nguyen, Tuan A. Nguyen, Pushkar B. Patil, William V.
Comments welcome Your comments are important to us! We want our books to be as helpful as possible. Send us your comments about this book or other IBM Redbooks publications in one of the following ways: Use the online Contact us review Redbooks form found at: ibm.com/redbooks Send your comments in an email to: redbooks@us.ibm.com Mail your comments to: IBM Corporation, International Technical Support Organization Dept.
1 Chapter 1. Introduction to IBM System Networking 10Gb Ethernet products Networks are changing. Voice, video, storage, and data are quickly converging onto a single backbone. Growth in cloud services and Web 2.0 multimedia content is pushing bandwidth demand to the edge. These bandwidth demands are also increasing as clients employ virtualization and focus on maximizing server usage. The next level of network consolidation has to do with I/O and storage.
1.1 Overview In today’s infrastructure, it is common to build networks based on 10 Gb Ethernet technology. The IBM portfolio of 10 Gb systems networking products includes Top-of-Rack switches, and the embedded switches in the IBM BladeCenter family. In 2010, IBM formed the IBM System Networking business (by acquiring BLADE Network Technologies), which is now focused on driving data center networking by using the latest in Ethernet technologies.
1.1.4 IBM RackSwitch Implementation Chapter 5, “IBM System Networking RackSwitch implementation” on page 155 provides information and instructions for implementing the 10 Gb Ethernet with Top-of-Rack switch models G8264R/F and G8124. As described in the reference architecture in Chapter 3, “Reference architectures” on page 107, we present a step by step guide for implementing and configuring the most important functions of the IBM Networking OS.
1.2 IBM System Networking 10Gb RackSwitch information In this section, we provide detailed information about the IBM System Networking 10Gb Switches summarized in Table 1-1.
1 m IBM Passive DAC SFP+ Cable 90Y9427 3 m IBM Passive DAC SFP+ Cable 90Y9430 5 m IBM Passive DAC SFP+ Cable 90Y9433 8.
These features are covered in more detail in Chapter 2, “IBM System Networking Switch 10Gb Ethernet switch features” on page 51. 1.3 IBM System Networking RackSwitch G8052 This switch is a Top-of-Rack switch designed for a data center. It combines great performance, server-like airflow for cooling, and low-power consumption in a virtualization-ready package. Figure 1-1 shows the IBM System Networking RackSwitch G8052 Top-of-Rack (TOR) Switch.
Mini-USB to DB9 serial cable (3 m) Comes with an IBM limited 3-year hardware warranty and includes a 3-year software license, providing entitlement to upgrades over that period Two power cords, depending on the country of purchase (Make sure that you include these cords in your configuration.) Transceivers: Small form-factor pluggable plus (SFP+) transceivers are not included in the purchase of the switch. All 1/10 Gb transceivers require LC-to-LC cables. 1.3.
Performance The performance features and specifications of the RackSwitch G8052 are: Single switch ASIC design Full line rate performance 176 Gbps (full duplex) switching architecture Low latency: 1.7 ms Hardware features The hardware features of the RackSwitch G8052 are: Models: – RackSwitch G8052F (for front-to-rear cooling). The ports at the front of the rack match the airflow of IBM RackSwitch iDataPlex®. – RackSwitch G8052R (for rear-to-front cooling).
Figure 1-3 shows the rear view of the switch. Figure 1-3 RackSwitch G8052 connections and modules (rear view) Software features The software features for the RackSwitch G8052 are: Security: – LDAP – 802.
– Fast uplink convergence – PVRST+ Quality of service: – QoS 802.
9 K Jumbo Frames 802.3X Flow Control 1.3.4 RackSwitch G8052 LED status details Figure 1-4 shows the LED indicators as they appear on the switch. Their meanings are explained in Figure 1-5. Figure 1-4 The location of LEDs on the RackSwitch G8052 Stacking master indicator Stacking member indicator Figure 1-5 Indicator LEDs and their meanings Table 1-4 shows the different System LED statuses for the RackSwitch G8052.
Function Stacking master indicatora Stacking member indicatora Power supply Service Non-Stack Member Off Off N/A Off a. Stacking for the RackSwitch G8052 is not currently supported, but these indicators remain for possible future feature releases. 1.3.5 More information For more about the RackSwitch G8052 and the LED status information, see the following resources: IBM System Networking RackSwitch G8052 Announcement Letter: http://www.ibm.
With support for 1G or 10G, this switch is designed for those clients that are using 10G Ethernet today or plan to do so in the future. This switch was the first Top-of-Rack 10Gb switch for IBM System x designed to support IBM Virtual Fabric, which helps clients reduce cost and complexity when it comes to the I/O requirements of most virtualization deployments today.
1.4.2 IBM System Networking RackSwitch G8124 features The RackSwitch G8124 offers the following feature benefits: High performance: The 10G Low Latency (<700 ns) switch provides the best combination of low latency, non-blocking line-rate switching and ease of management. Lower power and better cooling: The RackSwitch G8124 uses as little power as two 60 W light bulbs, which is a fraction of the power consumption of most competitive offerings.
1.4.3 Features and specifications In this section, we list some of the hardware and software features and specifications of the RackSwitch G8124. For more details about these features, see Chapter 2, “IBM System Networking Switch 10Gb Ethernet switch features” on page 51.
Power: – The AC-Powered G8124 has dual load-sharing internal power modules, with 50 - 60 Hz and 100 - 240 VAC auto-switching per module. – The nominal power for the G8124 ranges from 115 W to 168 W depending on the speed of the port (1G/10G), type of transceivers (SR or DAC), and number of active ports. Mean time between failures (MTBF): 189,060 hrs with ambient operating temperature of 40 ºC.
– – – – – – – SCP Wirespeed filtering: Allow and deny SSH v1 and v2 HTTPS Secure BBI Secure interface login and password MAC address move notification Shift B Boot menu (password recovery/factory default) VLANs: – – – – Port-based VLANs 4096 VLAN IDs supported 1 k VLANs (802.1Q) Private VLAN Edge FCoE/Lossless Ethernet: – – – – – – – 802.
Monitoring: – Port mirroring – ACL-based mirroring – sFlow Version 5 Virtualization: – VMready with VI API support – vNIC MIB support for SNMP Management features – Netboot Upgrades: – Upgrade firmware through serial or TFTP – Dual software images Management features RackSwitch G8124 supports the following management clients: IBM System Networking Element Manager isCLI (Cisco-like) Scriptable CLI Browser-based client or Telnet Standard protocols RackSwitch G8124 supports the following stand
Their meanings are explained in Figure 1-10. Power supplies and AC power input status Fans status Figure 1-10 Indicator LEDs and their meanings Table 1-6 shows the different System LED status for the RackSwitch G8124.
IBM System Networking RackSwitch G8124/G8124E Browser-Based Interface Quick Guide: http://www-01.ibm.com/support/docview.wss?uid=isg3T7000389 IBM System Networking RackSwitch G8124/G8124E ISCLI Command Reference: http://www-01.ibm.com/support/docview.wss?uid=isg3T7000390 IBM System Networking RackSwitch G8124/G8124E Menu-Based CLI Reference Guide: http://www-01.ibm.com/support/docview.wss?uid=isg3T7000391 1.
Table 1-7 shows the part numbers used to order the IBM System Networking RackSwitch G8264. Table 1-7 IBM System Networking RackSwitch G8264 part numbers Description IBM Part No. Power Systems MTM/FC IBM System Networking RackSwitch G8264R (Rear-to-Front) 7309G64 1455-64C IBM System Networking RackSwitch G8264F (Front-to-Rear) 730964F 1.5.
Seamless interoperability: IBM System Networking RackSwitches interoperate seamlessly with other vendors' upstream switches. Fault tolerance: IBM System Networking RackSwitches learn alternative routes automatically and perform faster convergence in the unlikely case of a link, switch, or power failure. The switch uses proven technologies, such as L2 trunk failover, advanced VLAN-based failover, VRRP, and Hot Link.
– One mini-USB Console port for serial access, which provides an additional means to install software and configure the switch module. This USB-style connector enables connection of a special serial cable that is supplied with the switch module. – Server-like port orientations, enabling short and simple cabling. Dimensions: 17.3 in. wide, 19 in. deep, 1 RU high Weight: 9.98 kg (22 lb) Rack Installation Kit: – Generic Rack Mount Kit (2-post).
Figure 1-13 shows the rear view of the switch. Figure 1-13 RackSwitch G8264 rear panel Software features The software features for the RackSwitch G8264 are: Security: – – – – – – – – – RADIUS TACACS+ SCP Wire Speed Filtering: Allow and Deny SSH v1 and v2 HTTPS Secure BBI Secure interface login and password MAC address move notification Shift B Boot menu (Password Recovery/ Factory Default) VLANs: – – – – – Port-based VLANs 4096 VLAN IDs supported 1024 Active VLANs (802.1Q) 802.
– Fast Uplink Convergence – BPDU guard Quality of Service: – QoS 802.
1.5.4 IBM System Networking RackSwitch G8264 LED status details Figure 1-14 shows the LED indicators as they appear on the switch. Figure 1-14 Location of LEDs on the RackSwitch G8264 Their meanings are explained in Figure 1-15. Service indicator Power supplies and power input status Figure 1-15 Indicator LEDs and their meanings Table 1-8 shows the different System LED statuses for the RackSwitch G8264.
b. If service is required is because of a power supply failure, this LED Flash. Otherwise, it is solid green. c. If service is required because of a fan failure, this LED Flash. Otherwise, it is solid green. d. If service is required because of a stacking error, this LED Flash or is solid green, depending on its last known good state. e. If an operations command is sent to the unit, this LED is solid blue. It can be used to locate the device. 1.5.
Using the CEE and FCoE functionality, you can transfer storage, network, Voice over IP (VoIP), video, and other data over the common Ethernet infrastructure. With the use of the QLogic Virtual Fabric Extension Module, clients can achieve FCoE gateway functionality inside the BladeCenter chassis. The IBM Virtual Fabric 10Gb Switch Module can be used both in IBM Virtual Fabric Mode and Switch Independent Mode. The switch module can be managed by using a command-line interface (CLI) or web browser interface.
Important Notices document Documentation CD-ROM Transceivers: Small form-factor pluggable plus (SFP+) transceivers are not included in the purchase of the switch. All 1/10 Gb transceivers require LC-to-LC cables. To communicate outside of the chassis, you must have either SFP+ transceivers or SFP+ direct-attach copper (DAC) cables connected. DAC cables have SFP+ transceivers on both ends.
Availability and redundancy: – VRRP for Layer 3 router redundancy. – IEEE 802.1D STP for providing Layer 2 redundancy with PVRST+. – IEEE 802.1s Multiple STP (MSTP) for topology optimization, up to 128 STP instances supported by single switch. – IEEE 802.1w Rapid STP (RSTP), providing rapid STP convergence for critical delay-sensitive, traffic-like voice or video. – Layer 2 trunk failover to support active and standby configurations of network adapter teaming on blades.
Manageability: – Simple Network Management Protocol (SNMP V1, V2, and V3). – HTTP and HTTPS Browser-Based Interface (BBI). – Industry standard CLI and IBM Networking OS/AlteonOS CLI. – Telnet interface for CLI. – SSH v1/v2. – Serial interface for CLI. – Scriptable CLI. – Firmware image update (Trivial File Transfer Protocol (TFTP) and File Transfer Protocol (FTP)). – Network Time Protocol (NTP) for switch clock synchronization. – IBM System Networking Element Manager support.
VMready compatibility with Virtual Fabric solutions is as follows: VMready is not supported with IBM Virtual Fabric Mode. VMready is supported with Switch Independent Mode. The switch module supports the following IEEE standards: IEEE 802.1D STP with PVRST+ IEEE 802.1s MSTP IEEE 802.1w RSTP IEEE 802.1p Tagged Packets IEEE 802.1Q Tagged VLAN (frame tagging on all ports when VLANs are enabled) IEEE 802.1x port-based authentication IEEE 802.2 Logical Link Control IEEE 802.
IBM Virtual Fabric 10Gb Switch Module for IBM BladeCenter isCLI Reference: http://download.boulder.ibm.com/ibmdl/pub/systems/support/system_x_pdf/bmd00191 .pdf IBM Virtual Fabric 10Gb Switch Module for IBM BladeCenter BBI Quick Guide: http://download.boulder.ibm.com/ibmdl/pub/systems/support/system_x_pdf/bmd00192 .pdf 1.6.
The part number includes the following items: One IBM 1/10 Uplink Ethernet Switch Module for IBM BladeCenter 3 m USB-to-DB9 serial console cable Printed documentation Documentation CD-ROM Features and specifications The IBM 1/10 Uplink Ethernet Switch Module includes the following standard features and functions: Internal ports: – 14 internal full-duplex Gigabit ports, one connected to each of the blade servers. – Two internal full-duplex 10/100 Mbps ports connected to the management module.
– Layer 2 Trunk Failover to support active/standby configurations of network adapter teaming on blades. – Interchassis redundancy (L2 and L3). VLAN support: – Up to 1024 VLANs supported per switch, with VLAN numbers ranging 1 - 4095 (4095 is used for management module’s connection only). – 802.1Q VLAN tagging support on all ports. – Private VLANs. Security: – VLAN-based, MAC-based, and IP-based ACLs. – 802.1X port-based authentication. – Multiple user IDs and passwords. – User access control.
– Change tracking and remote logging with syslog feature. – POST diagnostic tests. Special functions: Support for Serial over LAN (SOL) Standards supported: The switch module supports the following IEEE standards: – IEEE 802.1D Spanning Tree Protocol (STP). – IEEE 802.1s Multiple STP (MSTP). – IEEE 802.1w Rapid STP (RSTP). – IEEE 802.1p Tagged Packets. – IEEE 802.1Q Tagged VLAN (frame tagging on all ports when VLANs are enabled). – IEEE 802.1x port-based authentication. – IEEE 802.
The front panel contains the following components: LEDs that display the status of the switch module and the network. These LEDs are OK, which indicates that the switch module passed the power-on self-test (POST) with no critical faults and is operational, and switch module error, which indicates that the switch module failed the POST or detected an operational fault. One USB RS-232 console port that provides an additional means to install software and configure the switch module.
IBM 1/10Gb Uplink Ethernet Switch Module for IBM BladeCenter Installation Guide: ftp://ftp.software.ibm.com/systems/support/system_x_pdf/dw1gymst.pdf IBM 1/10Gb Uplink Ethernet Switch Module for IBM BladeCenter Application Guide: http://www-947.ibm.com/systems/support/supportsite.wss/docdisplay?brandind=5000 008&lndocid=MIGR-5076214 IBM 1/10Gb Uplink Ethernet Switch Module for IBM BladeCenter Command Reference: http://www-947.ibm.com/systems/support/supportsite.
High-speed redundant midplane connections: Based on 4X InfiniBand, the midplane supports up to 40 Gb bandwidth and provides four 10 Gb data channels to each blade. By giving each blade two physical connections to the midplane that connects all blades and modules together internally, a failure of one connector alone cannot bring down the server.
The features of the module can be accessed either locally or remotely across a network. One module comes standard. A second module can be added for hot-swap/redundancy and failover. The module uses USB ports for keyboard and mouse. Two module bays for blower modules: Two hot-swap/redundant blower modules come standard with the chassis. They can provide efficient cooling for up to 14 blades. These modules replace the need for each blade and switch to contain its own fans.
Blade server default ports: Ethernet Figure 1-21 shows a blade server in an IBM BladeCenter H chassis. The example blade shows two onboard NICs (NIC0 and NIC1). NIC0 is connected through the IBM BladeCenter H midplane to switch bay one. NIC1 is connected to switch bay two. These connections are hardwired in the chassis mid-plane and cannot be changed. Switch bays one and two in the IBM BladeCenter H are dedicated Ethernet switch bays; only Ethernet capable devices can be installed in these two switch bays.
Blade server high speed ports: 10 Gb Ethernet / 1 Gb Ethernet / Fibre Channel Switch bays seven to ten are optional high-speed switch bays. These switch bays are used for additional Ethernet connectivity by installing the IBM Virtual Fabric 10Gb Switch Module into the appropriate high-speed switch bays and an optional Combo Form Factor Horizontal (CFF-h) High Speed Ethernet daughter card on the blade server.
1.6.4 IBM BladeCenter HT IBM BladeCenter HT (Figure 1-22) is a carrier grade, rugged 12U chassis designed for challenging central office and networking environments. Figure 1-22 IBM BladeCenter HT chassis front view It provides: NEBS Level 3/ETSI-tested: Designed for the Network Equipment Provider (NEP) and Service Provider (SP) environments. Also ideal for government, military, aerospace, industrial automation and robotics, medical imaging, and finance.
Two module bays for Advanced Management Modules: The management modules provide advanced systems management and KVM capabilities for not only the chassis itself, but for all of the blades and other modules installed in the chassis. Four bays for Fan Modules: All four hot-swap/redundant fan modules come standard with the chassis. These modules replace the need for each blade to contain its own fans.
Up to eight module bays for communication and I/O switches or bridges: The modules interface with all of the blade servers in the chassis and alleviate the need for external switches or expensive and cumbersome cabling. All connections are done internally through the midplane. Two module slots are reserved for hot-swap/redundant Gigabit Ethernet switch modules. Two slots support either high-speed bridge modules or traditional Gigabit Ethernet, Myrinet, Fibre Channel, InfiniBand, and other switch modules.
Blade server default ports: Ethernet Figure 1-25 shows a blade server in an IBM BladeCenter HT chassis. The example blade shows two onboard NICs (NIC0 and NIC1). NIC0 is connected through the IBM BladeCenter HT midplane to switch bay one. NIC1 is connected to switch bay two. These connections are hardwired in the chassis mid-plane and cannot be changed.
Blade server high speed ports: 10 Gb Ethernet / 1 Gb Ethernet / Fibre Channel Switch bays seven to ten are optional high-speed switch bays. These switch bays are used for additional Ethernet connectivity by installing the IBM Virtual Fabric 10Gb Switch Module into the appropriate high-speed switch bays and an optional CFF-h high speed Ethernet daughter card on the blade server.
1.7 Connectors, cables, and options In this section, we describe the most common cables and connectors that you might need to use in your implementation of the IBM 10 Gb Network infrastructure. Figure 1-26 shows the different types of connectors available for use in the IBM System Networking RackSwitches and IBM Virtual Fabric 10Gb Switch Module for IBM BladeCenter.
Description Part number Feature code 10 m Intel Connects Optical Cable 46D0156 3853 30 m Intel Connects Optical Cable 46D0159 3854 . Note: The IBM Virtual Fabric 10Gb Switch Module and IBM RackSwitches can use DAC cables that are MSA compliant. To assist you in selecting the different types of cables and connectors, Figure 1-27 shows the distances that each cable and connector combination can reach.
BLADE Network Technologies (before being acquired by IBM) commissioned Tolly to evaluate the functionality of its RackSwitch G8000, RackSwitch G8124, and IBM Virtual Fabric 10G Switch Module for the IBM BladeCenter against a Cisco Nexus 5010 switch. Functionality tests focused on auto-negotiation, 10 GbE LAN PHY support, IEEE 802.
2 Chapter 2. IBM System Networking Switch 10Gb Ethernet switch features In Chapter 1, “Introduction to IBM System Networking 10Gb Ethernet products” on page 1, we provided an overview of the various features that are available on the different IBM System Networking 10Gb Ethernet switches. In this chapter, we describe those features in detail. When planning a network, you must decide how the network needs to function.
2.1 Virtual Local Area Networks This section describes network design and topology considerations for using Virtual Local Area Networks (VLANs). VLANs commonly are used to split up groups of network users into manageable broadcast domains, to create logical segmentation of workgroups, and to enforce security policies among logical segments. 2.1.1 VLANs overview Setting up VLANs is a way to segment networks, which increases network flexibility without changing the physical network topology.
Because tagging changes the format of frames transmitted on a tagged port, you must plan network designs to prevent tagged frames from being transmitted to devices that do not support 802.1Q VLAN tags, or devices where tagging is not enabled. Important terms used with the 802.1Q tagging feature are: VLAN identifier (VID): The 12-bit portion of the VLAN tag in the frame header that identifies an explicit VLAN.
The default configuration settings for IBM System Networking switches have all ports set as untagged members of VLAN 1 with all ports configured as PVID = 1. In the default configuration example shown in Figure 2-1, all incoming packets are assigned to VLAN 1 by the default port VLAN identifier (PVID =1). Figure 2-1 Default VLAN settings When a VLAN is configured, ports are added as members of the VLAN, and the ports are defined as either tagged or untagged (see Figure 2-2 through Figure 2-5 on page 56).
As shown in Figure 2-3, the untagged packet is marked (tagged) as it leaves the switch through port 5, which is configured as a tagged member of VLAN 2. The untagged packet remains unchanged as it leaves the switch through port 7, which is configured as an untagged member of VLAN 2. Figure 2-3 802.1Q tagging (after port-based VLAN assignment) In Figure 2-4, tagged incoming packets are assigned directly to VLAN 2 because of the tag assignment in the packet.
As shown in Figure 2-5, the tagged packet remains unchanged as it leaves the switch through port 5, which is configured as a tagged member of VLAN 2. However, the tagged packet is stripped (untagged) as it leaves the switch through port 7, which is configured as an untagged member of VLAN 2. Figure 2-5 802.1Q tagging (after 802.1Q tag assignment) 2.1.3 Protocol-based VLANs You can use protocol-based VLANs (PVLANs) to segment network traffic according to the network protocols in use.
Ethernet type: Consists of a 4-digit (16 bit) hex value that defines the Ethernet type. You can use common Ethernet protocol values, or define your own values. Here are examples of common Ethernet protocol values: – IPv4 = 0800 – IPv6 = 86dd – ARP = 0806 Port-based versus protocol-based VLANs Each VLAN supports both port-based and protocol-based association, as follows: The default VLAN configuration is port-based. All data ports are members of VLAN 1, with no PVLAN association.
2.2 Spanning Tree Protocol In high-availability environments, a redundant design is often introduced to minimize any network downtime. The redundancy is implemented on many layers, from physical cabling to redundant switches, to ensure continuous operations. For more information about network availability protocols and technologies, see 2.7, “High availability” on page 77. In a redundant multi-path network, Ethernet broadcast and unknown unicast flooding mechanisms can lead to forwarding loops.
The details of the operations of STP are not covered in this book. What is important to remember is that STP works in Layer 2 by detecting forwarding loops and logically disabling the link that is part of the loop. STP operates by transmitting and receiving Bridge Protocol Data Units (BPDUs). For more information about STP, see the originalIEEE 802.1d specification at the following website: http://standards.ieee.org/getieee802/download/802.1D-2004.
2.3 IP routing IBM System Networking switches use a combination of configurable IP switch interfaces and IP routing options. The switch IP routing capabilities provide the following benefits: Connects the server IP subnets to the rest of the backbone network. Provides routing of IP traffic between multiple VLANs configured on the switch. The physical layout of most corporate networks has evolved over time.
2.3.1 Static routes You can use static routes to forward an IP packet based on a manually configured entry. The entry specifies a network and an IP address of a gateway, or next-hop, to that network. 2.3.2 Equal-Cost Multi-Path static routes Equal-Cost Multi-Path (ECMP) is a forwarding mechanism that routes packets along multiple paths of equal cost. ECMP provides equally distributed link load sharing across the paths. The hashing algorithm used is based on the source IP address.
When a router receives a routing update that includes changes to an entry, it updates its routing table to reflect the new route. The metric value for the path is increased by 1, and the sender is indicated as the next hop. RIP routers maintain only the best route (the route with the lowest metric value) to a destination. RIPv1 RIP version 1 use broadcast User Datagram Protocol (UDP) data packets for the regular routing updates.
OSPF defines the following types of areas (shown in Figure 2-8): Stub area: An area that is connected to only one other area. External route information is not distributed into stub areas. Not-So-Stubby-Area (NSSA): Similar to a stub area with additional capabilities. Routes originating from within the NSSA can be propagated to adjacent transit and backbone areas. External routes from outside the AS can be advertised within the NSSA but are not distributed into other areas.
OSPF router types As shown in Figure 2-9 OSPF uses the following types of routing devices: Internal Router (IR): A router that has all of its interfaces within the same area. IRs maintain LSDBs identical to the LSDBs of other routing devices within the local area. Area Border Router (ABR): A router that has interfaces in multiple areas. ABRs maintain one LSDB for each connected area and disseminate routing information between areas.
Each routing device transmits a Link-State Advertisement (LSA) on each of its active interfaces. LSAs are entered into the LSDB of each routing device. OSPF uses flooding to distribute LSAs between routing devices. Interfaces may also be passive. Passive interfaces send LSAs to active interfaces, but do not receive LSAs, hello packets, or any other OSPF protocol information from active interfaces.
2.3.5 Border Gateway Protocol Border Gateway Protocol (BGP) is an Internet protocol that enables routers on an IPv4 network to share and advertise routing information with each other about the segments of the IPv4 address space they can access within their network and with routers on external networks.
Forming BGP peer routers Two BGP routers become peers or neighbors after you establish a TCP connection between them. For each new route, if a peer is interested in that route (if a peer would like to receive your static routes and the new route is static), an update message is sent to the peer that contains the new route. For each route removed from the route table, if the route has already been sent to a peer, an update message that contains the route to withdraw is sent to that peer.
IGMP Snooping conserves bandwidth. With IGMP Snooping, the switch learns which ports are interested in receiving multicast data, and forwards multicast data only to those ports. In this way, other ports are not burdened with unwanted multicast traffic. The switch can sense IGMP Membership Reports from attached clients and act as a proxy to set up a dedicated path between the requesting host and a local IPv4 Multicast router.
IGMP Querier IGMP Querier allows the switch to perform the multicast router (Mrouter) role and provide Mrouter discovery when the network or virtual LAN (VLAN) does not have a router. When IGMP Querier is enabled on a VLAN, the switch acts as an IGMP querier in a Layer 2 network environment. The IGMP querier periodically broadcasts IGMP Queries and listens for hosts to respond with IGMP Reports indicating their IGMP group memberships.
Instead of sending a separate copy of content to each receiver, a multicast derives efficiency by sending only a single copy of content toward its intended receivers. This single copy becomes duplicated only when it reaches the target domain that includes multiple receivers, or when it reaches a necessary bifurcation point leading to different receiver domains.
PIM Sparse Mode The behavior of PIM Sparse Mode is opposite of Dense Mode. The default behavior is to not to flood the multicast traffic unless the downstream routers signal, by sending a PIM Join message, that there are receivers on their directly connected networks interested in receiving the multicast traffic. For more information about PIM Sparse Mode, see RFC 4601, found at: http://www.ietf.org/rfc/rfc4601.txt 2.
Most IBM Networking OS features permit IP addresses to be configured using either IPv4 or IPv6 address formats. Throughout this manual, IP address is used in places where either an IPv4 or IPv6 address is allowed. In places where only one type of address is allowed, the type (IPv4 or IPv6) is specified. 2.5.2 IPv6 address types IPv6 supports three types of addresses: unicast (one-to-one), multicast (one-to-many), and anycast (one-to-nearest). Multicast addresses replace the use of broadcast addresses.
Anycast addresses are allocated from the unicast address space, using any of the defined unicast address formats. Thus, anycast addresses are syntactically indistinguishable from unicast addresses. When a unicast address is assigned to more than one interface, thus turning it into an anycast address, the nodes to which the address is assigned must be configured to know that it is an anycast address. 2.5.
Host versus router Each IPv6 interface can be configured as a router node or a host node, as follows: A router node’s IP address is configured manually. Router nodes can send Router Advertisements. A host node’s IP address is auto-configured. Host nodes listen for Router Advertisements that convey information about devices on the network. IP forwarding: When IP forwarding is turned on. All IPv6 interfaces configured on the switch can forward packets.
For example, an intrusion detection system (IDS) server or other traffic sniffer device or analyzer can be connected to the monitoring port to detect intruders that attack the network. IBM System Networking switches support a “many to one” mirroring model. As shown in Figure 2-11, selected traffic for ports 1 and 2 is being monitored by port 3. In the example, both ingress traffic and egress traffic on port 2 are copied and forwarded to the monitor.
When polling is enabled, at the end of each configured polling interval, the switch reports general port statistics and port Ethernet statistics. sFlow network sampling In addition to statistical counters, IBM System Networking switches can be configured to collect periodic samples of the traffic data received on each port. For each sample, 128 bytes are copied, UDP-encapsulated, and sent to the configured sFlow analyzer.
2.7 High availability Internet traffic consists of myriad services and applications that use the Internet Protocol (IP) for data delivery. However, IP is not optimized for all the various applications. High availability goes beyond IP and makes intelligent switching decisions to provide redundant network configurations. 2.7.
Link Aggregation Control Protocol Link Aggregation Control Protocol (LACP) is an IEEE 802.3ad standard for grouping several physical ports into one logical port (known as a dynamic trunk group or Link Aggregation group) with any device that supports the standard. See the IEEE 802.3ad-2002 specification for a full description of the standard. The 802.3ad standard allows standard Ethernet links to form a single Layer 2 link by using the Link Aggregation Control Protocol (LACP).
2.7.2 Virtual Link Aggregation Groups In Figure 2-13, we show a typical data center design environment with access and aggregation layers. VLAG peers Aggregation Layer XX X STP blocks implicit loops VLAGs Links remain active Access Layer ... Servers ... Figure 2-13 Spanning Tree Protocoal versus Virtual Link Aggregation Groups As shown in Figure 2-13, a switch in the access layer may be connected to more than one switch in the aggregation layer to provide network redundancy.
You may select a physical port, static trunk, or an LACP adminkey as a Hot Link interface. Forward Delay The Forward Delay timer allows Hot Links to monitor the Master and Backup interfaces for link stability before selecting one interface to change to the active state. Before the transition occurs, the interface must maintain a stable link for the duration of the Forward Delay interval.
Monitoring trunk links Layer 2 Failover can be enabled on any trunk group in IBM System Networking switches, including LACP trunks. Trunks can be added to failover trigger groups. Then, if some specified number of monitor links fail, the switch disables all the control ports in the switch. When the control ports are disabled, it causes the NIC team on the affected servers to fail over from the primary to the backup NIC. This process is called a failover event.
Static trunks When you add a portchannel (static trunk group) to a failover trigger, any ports in that trunk become members of the trigger. You can add up to 64 static trunks to a failover trigger, using manual monitoring. Link Aggregation Control Protocol LACP allows the switch to form dynamic trunks. You can use the admin key to add up to two LACP trunks to a failover trigger by using automatic monitoring.
Virtual router MAC address The VRID is used to build the virtual router MAC Address. The five highest-order octets of the virtual router MAC Address are the standard MAC prefix (00-00-5E-00-01) defined in RFC 2338. The VRID is used to form the lowest-order octet. Owners and renters Only one of the VRRP routers in a virtual router may be configured as the IPv4 address owner. This router has the virtual router’s IPv4 address as its real interface address.
The master periodically sends advertisements to an IPv4 Multicast address. IF the backups receive these advertisements, they remain in the backup state. If a backup does not receive an advertisement for three advertisement intervals, it initiates a bidding process to determine which VRRP router has the highest priority and takes over as master.
A virtual router group has the following characteristics: When enabled, all virtual routers behave as one entity, and all group settings override any individual virtual router settings. All individual virtual routers, after the VRRP group is enabled, assume the group’s tracking and priority. When one member of a VRRP group fails, the priority of the group decreases, and the state of the entire switch changes from Master to Standby. Each VRRP advertisement can include up to 16 addresses.
VRRP high availability with VLAGs VRRP can be used in conjunction with VLAGs and LACP-capable servers and switches to provide seamless redundancy (Figure 2-17). Figure 2-17 Active-active configuration using VRRP and VLAGs 2.7.7 Active Multipath Protocol You can use Active MultiPath Protocol (AMP) to connect three switches in a loop topology, and load-balance traffic across all uplinks (no blocking). When an AMP link fails, upstream communication continues over the remaining AMP link.
Figure 2-18 shows a typical AMP topology, with two aggregators that support a number of AMP groups. Figure 2-18 Active Multipath Protocol topology Each AMP group requires two links on each switch. Each AMP link consists of a single port, a static trunk group, or an LACP trunk group. Local non-AMP ports can communicate through local Layer 2 switching without passing traffic through the AMP links. No two switches in the AMP loop can have another active connection between them through a non-AMP switch.
FDB flush When an AMP port/trunk is the blocking state, FDB flush is performed on that port/trunk. Any time there is a change in the data path for an AMP group, the FDB entries associated with the ports in the AMP group are flushed. This situation ensures that communication is not blocked while obsolete FDB entries are aged out. FDB flush is performed when an AMP link goes down, and when an AMP link comes up. 2.7.
The preferred stacking topology is a bidirectional ring (Figure 2-20). To achieve this topology, two external 10Gb Ethernet ports on each switch must be reserved for stacking. By default, the first two 10Gb Ethernet ports are used. Figure 2-20 Stacking connection The cables used for connecting the switches in a stack carry low-level, inter-switch communications and cross-stack data traffic critical to shared switching functions.
Private VLAN ports Private VLAN ports are defined as follows: Promiscuous: A promiscuous port is an external port that belongs to the primary VLAN. The promiscuous port can communicate with all the interfaces, including ports in the secondary VLANs (Isolated VLAN and Community VLANs). Each promiscuous port can belong to only one Private VLAN. Isolated: An isolated port is a host port that belongs to an isolated VLAN.
User access control IBM System Networking switch allows an administrator to define user accounts that permit users to perform operation tasks through the switch CLI commands. After user accounts are configured and enabled, the switch requires user name and password authentication. For example, an administrator can assign a user, who can then log on to the switch and perform operational commands (effective only until the next switch reboot).
RADIUS authentication and authorization IBM System Networking switch supports the RADIUS (Remote Authentication Dial-in User Service) method to authenticate and authorize remote administrators for managing the switch. This method is based on a client/server model. The Remote Access Server (RAS), the switch, is a client to the back-end database server. A remote user (the remote administrator) interacts only with the RAS, not the back-end server and database.
Supports a user-configurable RADIUS application port. The default is UDP port 1645. UDP port 1812, based on RFC 2138, is also supported. Allows network administrator to define privileges for one or more specific users to access the switch at the RADIUS user database. Switch user accounts The user accounts listed in Table 2-1 can be defined in the RADIUS server dictionary file.
TACACS+ authentication IBM Networking OS supports authentication, authorization, and accounting with networks using the Cisco Systems TACACS+ protocol. The switch functions as the Network Access Server (NAS) by interacting with the remote client and initiating authentication and authorization sessions with the TACACS+ access server. The remote user is defined as someone that requires management access to the VFSM either through a data or management port.
If the remote user is successfully authenticated by the authentication server, the switch verifies the privileges of the remote user and authorizes the appropriate access. The administrator may allow secure back door access through Telnet/SSH. Secure back door provides switch access when the TACACS+ servers cannot be reached. Accounting Accounting is the action of recording a user's activities on the device for the purposes of billing and security. It follows the authentication and authorization actions.
The 802.1X standard describes port-based network access control by using Extensible Authentication Protocol over LAN (EAPoL). EAPoL provides a means of authenticating and authorizing devices attached to a LAN port that has point-to-point connection characteristics and of preventing access to that port in cases of authentication and authorization failures.
Figure 2-21 shows a typical message exchange initiated by the client. Figure 2-21 Authenticating a port by using EAPoL 2.8.6 Access control lists Access control lists (ACLs) are filters that permit or deny traffic for security purposes. They can also be used with QoS to classify and segment traffic to provide different levels of service to different traffic types. Each filter defines the conditions that must match for inclusion in the filter, and also the actions that are performed when a match is made.
Summary of packet classifiers You can use ACLs to classify packets according to various content in the packet header (such as the source address, destination address, source port number, destination port number, and others). Once classified, packet flows can be identified for more processing.
Port Application Port Application Port Application 37 time 119 NNTP 520 rip 42 name 123 NTP 554 rtsp 43 whois 143 IMAP 1645/1812 RADIUS 53 domain 144 news 1813 RADIUS accounting 69 TFTP 161 SNMP 1985 hsrp 70 gopher 162 snmptrap TCP flag value, as shown in Table 2-6 Table 2-6 TCP flag values Flag Value URG 0x0020 ACK 0x0010 PSH 0x0008 RST 0x0004 SYN 0x0002 FIN 0x0001 Packet format (for regular ACLs and VMaps only) – Ethernet format (eth2, SNAP, LLC) –
2.8.7 VLAN maps A VLAN map (VMAP) is an ACL that can be assigned to a VLAN or VM group rather than to a switch port, as with regular ACLs. A VMAP is useful in a virtualized environment where traffic filtering and metering policies must follow virtual machines (VMs) as they migrate between hypervisors.
Figure 2-22 shows the basic QoS model used by the switch: Figure 2-22 QoS model The basic QoS model works as follows: Classify traffic: – Read DSCP value. – Read 802.1p priority value. – Match ACL filter parameters. Perform actions: – – – – – – Define bandwidth and burst parameters. Select actions to perform on in-profile and out-of-profile traffic. Deny packets. Permit packets. Mark DSCP or 802.1p priority. Set COS queue (with or without re-marking).
Re-mark the 802.1p field. Set the COS queue. 2.9.4 ACL metering and re-marking You can define a profile for the aggregate traffic that flows through the switch by configuring a QoS meter (if wanted) and assigning ACLs to ports. Actions taken by an ACL are called In-Profile actions. You can configure additional In-Profile and Out-of-Profile actions on a port.
Differentiated Services concepts To differentiate between traffic flows, packets can be classified by their DSCP value. The Differentiated Services (DS) field in the IP header is an octet, and the first 6 bits, called the DS Code Point (DSCP), can provide QoS functions. Each packet carries its own QoS state in the DSCP. There are 64 possible DSCP values (0-63). Figure 2-23 IPv4 packet with DSCP field The switch can perform the following actions on the DSCP: Read the DSCP value of ingress packets.
Class Selector (CS): This PHB has eight priority classes, with CS7 representing the highest priority, and CS0 representing the lowest priority, as shown in Table 2-8. CS PHB is described in RFC 2474, found at: http://www.ietf.org/rfc/rfc2474.
2.9.6 QoS 802.1p IBM Networking OS provides Quality of Service functions based on the priority bits in a packet’s VLAN header. (The priority bits are defined by the 802.1p standard within the IEEE 802.1q VLAN header.) The 802.1p bits, if present in the packet, specify the priority that should be given to packets during forwarding. Packets with a numerically higher (non-zero) priority are given forwarding preference over packets with lower priority bit value. The IEEE 802.
A scheduling weight of 0 (zero) indicates strict priority. Traffic in the strict priority queue has precedence over other all queues. If more than one queue is assigned a weight of 0, the strict queue with highest queue number is served first. After all traffic in strict queues is delivered, any remaining bandwidth is allocated to the WRR queues, divided according to their weight values. Strict scheduling: Use caution when assigning strict scheduling to queues.
3 Chapter 3. Reference architectures This chapter presents the network architecture used in this book to implement a 10 Gb Ethernet solution by using IBM System Networking switches. The design is based on preferred practices and the experience of the authors and is meant to provide support for the implementation chapters later in this book.
3.1 Overview of the reference architectures The reference architectures describe a mixed environment of both stand-alone and IBM BladeCenter embedded switches that are integrated in a fully functional network. These switches are able to provide end-to-end communication in a data center, for servers that run different operating systems (Windows and Linux) and IP protocol versions (IPv4 and IPv6).
Figure 3-1 shows an overview of the Top-of-Rack architecture. Top-of-Rack Architecture Aggregation Layer RackSwitch G8264 Access Layer RackSwitch G8124 Server IBM System x3550 M3 Figure 3-1 Top-of-Rack architecture overview 3.2.1 Layer 1 architecture The Layer 1, or physical layer, architecture includes the hardware components, physical cabling, connectors, interfaces, and host names. Hardware components The list of equipment used in this topology is shown in Table 3-1.
Table 3-2 shows the switch interfaces used for connecting the devices. The port numbering assumes that aggregation (RackSwitch G8264) is in QSFP+ 40GbE mode.
Figure 3-2 shows a Layer 1 architecture with the host names and interfaces used for inter-switch connections. Aggregation Layer AGG-1 port1 port1 port5 port5 AGG-2 port19 port19 port20 port17 port18 port1 port2 port20 port3 Access Layer port18 port1 port2 port3 port4 ACC-1 port17 port4 port5 port5 port6 port6 ACC-2 port7 port7 NIC0 NIC1 10 Gbps 40 Gbps Server IBM System x3550 M3 SRV-1 Figure 3-2 Top-of-Rack - Layer 1 architecture 3.2.
VLAN number VLAN name Description Member ports VLAN101 ACC-1 - AGG-1 ACC-1 AGG1 p2p network ACC-1, port1 (untagged) ACC-1, port2 (untagged) AGG-1, port17 (untagged) AGG-1, port18 (untagged) VLAN102 ACC-1 - AGG-2 ACC-1 - AGG-2 p2p network ACC-1, port3 (untagged) ACC-1, port4 (untagged) AGG-2, port19 (untagged) AGG-2, port20 (untagged) VLAN103 ACC-2 - AGG-1 ACC-2 - AGG-1 p2p network ACC-2, port3 (untagged) ACC-2, port4 (untagged) AGG-1, port19 (untagged) AGG-1, port20 (untagged) VLAN104 ACC-2
Trunks We group the inter-switch connection into trunks, as shown in Table 3-5.
VLAN IPv4 IPv6 Description VLAN104 10.0.104.0/30 FC14::0/64 Point-to-point link subnet between ACC-2 and AGG-2 VLAN100 10.0.100.0/30 FC00::0/64 Point-to-point link subnet between AGG-1 and AGG-2 In Table 3-7, we show the management IPv4 addresses, that is, addresses assigned to the management interfaces. Table 3-7 Management IP addresses Switch Management IP address Management interface ACC-1 172.25.101.122 port25 (MGTA) ACC-2 172.25.101.123 port25 (MGTA) AGG-1 172.25.101.
VLAN IPv4 interface IPv4 address IPv6 interface IPv6 address Device VLAN102 102 10.0.102.2 112 FC12::2/64 AGG-2 VLAN100 100 10.0.100.2 110 FC00::2/64 AGG-2 Loopback interfaces are used for static router-ID assignment to use with OSPFv2 and OSPFv3. Table 3-9 shows the loopback interfaces IP addresses assignment Table 3-9 Loopback interfaces IP address Loopback Device 1.1.1.1/32 Loopback 1 AGG-1 1.1.1.2/32 Loopback 1 AGG-2 2.2.2.1/32 Loopback 1 ACC-1 2.2.2.
Figure 3-3 shows the Layer 3 architecture of our Top-of-Rack design. AGG-1 AGG-2 OSPF Area 0 VLAN100 10.0.100.0/30 FC00::0/64 .1 ::1 .1 ::1 .2 ::2 ha nn el2 portchannel1 p c or t VLAN102 10.0.102.0/30 FC12::0/64 l2 ne an .1 ::1 VLAN103 10.0.103.0/30 FC13::0/64 .2 ::2 h rtc po portchannel1 VLAN 101 10.0.101.0/30 FC11::0/64 .2 ::2 portchannel3 .2 ::2 .1 ::1 .1 ::1 VLAN104 10.0.104.0/30 FC14::0/64 .2 ::2 portchannel3 ACC-1 .2 / .1 ::2 / ::1 VLAN10 10.0.10.0/24 FC10::0/64 .3 / .
Figure 3-4 shows the components of our BladeCenter architecture. BladeCenter Architecture RackSwitch G8264 Aggregation Layer Stack IBM Virtual Fabric 10Gb Switch Module for BladeCenter Access Layer Blade Server Server BladeCenter Chassis Figure 3-4 BladeCenter - architecture overview 3.3.1 Layer 1 architecture Layer 1, or physical layer, architecture includes the hardware components, physical cabling, connectors, interfaces, and host names.
Figure 3-5 shows the chassis of the IBM BladeCenter H used in the access layer of our BladeCenter architecture. IBM Virtual Fabric 10Gb Switch Module Bay 7 Bay 8 Server Bays 1 2 3 4 5 6 7 8 9 10 11 12 13 14 IBM Blade Center HS22 IBM Virtual Fabric 10Gb Switch Module Bay 9 Bay 10 Figure 3-5 IBM BladeCenter H chassis used in the access layer of our BladeCenter architecture The physical connections details used for the reference architecture installation are summarized in Table 3-12.
Device #1 Interface device #1 Device #2 Interface device #2 Interface type Connector/Cable type AGG-2 port1 AGG-1 port1 40 Gbps DAC copper (QSFP+) AGG-2 port5 AGG-1 port5 40 Gbps DAC copper (QSFP+) Figure 3-6 shows the Layer 1 architecture based on the information in Table 3-12 on page 118. Figure 3-6 BladeCenter - Layer 1 architecture 3.3.2 Layer 2 architecture Access layer switches (ACC-3 and ACC-4) are stacked using a pair of connections on external ports 9 and 10 (port25 and port26).
VLANs Table 3-13 shows VLANs used in the topology and the member ports of those VLANs.
IPv4 and IPv6 addressing Table 3-15 shows the IPv4 and IPv6 address spaces assigned for VLAN 30. Table 3-15 IP address ranges VLAN IPv4 IPv6 Description 30 10.0.30.0/24 FC30::0/64 VLAN30 Table 3-16 shows the IP addresses assigned to devices that have IP interfaces in VLAN30 and VLAN40. Table 3-16 IPv4 and IPv6 addresses assignment VLAN IPv4 interface IPv4 address IPv6 interface IPv6 address Device VLAN30 Bonding interface 10.0.30.30 Bonding interface FC30::30/64 SRV-3 VLAN30 30 10.0.
3.4 Final architecture Figure 3-7 shows the final network topology, with the connected hosts that are able to communicate with each other on IPv4 and IPv6. BladeCenter Chassis SRV-3 10.0.30.30 FC30::30 VFSM Stack ACC-3 VRRP Group 30 10.0.30.1 .2 / .1 ::2 / ::1 .1 ::1 .2 ::2 r nn el2 VLAN102 10.0.102.0/30 FC12::0/64 portchannel1 po a t ch AGG-2 .2 ::2 2 el nn ha .1 ::1 VLAN103 10.0.103.0/30 FC13::0/64 .2 ::2 rtc po portchannel1 VLAN 101 10.0.101.0/30 FC11::0/64 .3 / .
4 Chapter 4. Initial configuration: IBM System Networking 10Gb Ethernet switches In this chapter, we describe the steps to be performed for the initial configuration of the IBM System Networking 10Gb Ethernet switches, both in the Top-of-Rack (TOR) RackSwitch G8264 and embedded switch IBM 10Gb Virtual Fabric Module for BladeCenter versions. We also provide an introduction to the IBM System Networking Element Manager (Previously know as BLADE Harmony Manager) software.
4.1 Overview of the initial setup The steps cover the following elements for the hardware: Terminal connection Setting up the IP address of the switch Configuring the date and time Security For the IBM System Networking Element Manager software, we describe how to: Install it Configure the basic options Switch initial setup Every time we receive a new switch and we want to install it in our network, there is a set of basic initial configuration tasks that should be done.
4.2.1 Console, Telnet, and Secure Shell (SSH) The IBM Networking OS CLI provides a simple and direct method for switch administration. Using a basic terminal, you have an organized hierarchy of menus, each with logically related submenus and commands. You can use these items to view detailed information and statistics about the switch, and to perform any necessary configuration and switch software maintenance.
After successfully logging on, the Switch Dashboard is displayed, as shown in Figure 4-2. Figure 4-2 Switch Dashboard For more details about the web interface, see following documents: IBM System Networking RackSwitch G8264 Browser-Based Interface Quick Guide: https://www-304.ibm.com/support/docview.wss?uid=isg3T7000296&aid=1 IBM System Networking RackSwitch G8124 Browser-Based Interface Quick Guide: https://www-304.ibm.com/support/docview.
4.3 First boot of the RackSwitch G8264 switch When using a TOR switch, before you first boot the switch, connect a console to it to be able to see the boot POST messages and log on to it. For a TOR model, you need to connect the console cable to a serial port of your computer and to the console port of the switch. This serial console is the only available method for you to connect to the switch in the first stage.
Terminal connection: Depending on your guest operating system, the terminal might look different, and the connection options for the serial port might also change. See your operating system help for troubleshooting any issues with the terminal connection to the switches. After your terminal console is ready, you can plug in the power cables of the switch. The switch automatically powers on.
If you connect by using SSH, the first time you connect to the switch, you must exchange encryption keys with the switch to establish the connection. Your SSH client handles this exchange automatically. In our case, the SSH client displayed a window to confirm the key exchange, as shown in Figure 4-5. Figure 4-5 SSH key exchange message When you first log on to the switch, and if there no other user logged on to the switch, you are prompted to choose the CLI that you use.
4.3.2 Global Configuration mode In this scenario, we use the IBM Networking OS CLI and its commands. If we use the industry standard CLI, we note that we are using it where appropriate. After you log on, enter the Global Configuration mode by running enable and then configure terminal, as shown in Example 4-1. Example 4-1 Enabling the Global Configuration mode RSG8264> enable Enable privilege granted. RSG8264# configure terminal Enter configuration commands, one per line.
4.3.3 Setup tool The IBM Networking OS includes a setup utility to complete the initial configuration of your switch. The setup utility prompts you to enter all the necessary information for the basic configuration of the switch. Whenever you log on as the system administrator under the factory default configuration, you are prompted whether you want to run the setup utility. Setup can also be activated manually from the CLI any time after you log on by running /cfg/setup (Example 4-4).
System clock set to 18:55:36 Fri Jul 22, 2011. System Time: Enter hour in 24-hour format [18]: Enter minutes [55]: Enter seconds [37]: System clock set to 18:55:36 Fri Jul 22, 2011. Spanning Tree: Current Spanning Tree Group 1 setting: ON Turn Spanning Tree Group 1 OFF? [y/n] The setup tool prompts you to configure VLANs and VLAN tagging for the ports (Example 4-6). If you want to change settings for VLANs, enter y, or enter n to skip VLAN configuration.
Depending on whether you answered yes or no to the VLAN configuration, you then define the different characteristics of the VLANS. If you selected to configure VLANs in Example 4-6 on page 132, you can enable or disable VLAN tagging for the port (Example 4-9).
After the VLANs are configured, you must configure the Spanning Tree Group membership for the VLAN. Follow the prompts from the setup tool shown in the Example 4-14. Example 4-14 Spanning Tree membership Spanning Tree Group membership: Enter new Spanning Tree Group index [1-127]: The system prompts you to configure the next VLAN (Example 4-15). If you want to configure another VLAN, enter the number.
For the specified IP interface, enter the IP address in IPv4 dotted decimal notation (Example 4-17). Example 4-17 IP address configuration Current IP address: 172.25.101.120 Enter new IP address: To keep the current setting, press Enter. At the prompt, enter the IPv4 subnet mask in dotted decimal notation (Example 4-18). Example 4-18 Subnet mask configuration Current subnet mask: 255.255.0.0 Enter new subnet mask: To keep the current setting, press Enter. In our case, we used the 255.255.0.0 subnet.
At the prompt, enter the IPv4 address for the selected default gateway (Example 4-23). Example 4-23 Default gateway IP Current IP address: 0.0.0.0 Enter new IP address: Enter the IPv4 address in dotted decimal notation, or press without specifying an address to accept the current setting. At the prompt, enter y to enable the default gateway, or n to leave it disabled (Example 4-24).
When prompted, decide whether you want to review the configuration changes (Example 4-28). Example 4-28 Review changes Review the changes made? [y/n] Enter y to review the changes made during this session of the setup utility. Enter n to continue without reviewing the changes. Next, decide whether to apply the changes (Example 4-29). Example 4-29 Apply changes Apply the changes? [y/n] Enter y to apply the changes, or n to continue without applying. Changes are normally applied.
You should change the default passwords for your administrator users. To accomplish this task, you must perform the commands shown in Example 4-32. Example 4-32 Change password procedure for admin user >> RS8264 - Main# /cfg/sys/access >> RS8264 - System Access# user/admpw Changing ADMINISTRATOR password; validation required: Enter current admin password: Enter new admin password (max 128 characters): Re-enter new admin password: New admin password accepted.
User Account Description and tasks performed Default password Operator The Operator manages all functions of the switch. The Operator can reset ports, except for the management port. operator Administrator The super-user Administrator has complete access to all commands, information, and configuration commands on the switch, including the ability to change both the user and administrator passwords. admin To confirm that the creation of the user is done correctly, run show access (Example 4-35).
You must configure the RADIUS secret by running the commands shown in Example 4-39. Example 4-39 RADIUS secret RS8264(config)# radius-server primary-host 10.10.1.1 key <1-32 character secret> RS8264(config)# radius-server secondary-host 10.10.1.2 key <1-32 character secret> You may change the default UDP port number used to listen to RADIUS (Example 4-40). The known port for RADIUS is 1812.
As shown in Example 4-45, configure the IPv4 addresses of the Primary and Secondary TACACS+ servers, and enable TACACS authentication. Specify the interface port (optional). Example 4-45 Primary and Secondary TACACS+ servers RS8264(config)# RS8264(config)# RS8264(config)# RS8264(config)# RS8264(config)# tacacs-server tacacs-server tacacs-server tacacs-server tacacs-server primary-host 10.10.1.1 primary-host mgt-port secondary-host 10.10.1.
You may change the default TCP port number used to listen to LDAP (optional) (Example 4-51). The known port for LDAP is 389. Example 4-51 LDAP port >> # ldap-server port <1-65000> Configure the number of retry attempts for contacting the LDAP server, and the timeout period (Example 4-52). Example 4-52 LDAP retry and timeout >> # ldap-server retransmit 3 >> # ldap-server timeout 10 4.
4.4.1 Basic options Log on to the AMM by using your user name and password. Once in the web interface, expand I/O Module tasks and click Admin/Power/Restart. The window shown in Figure 4-7 opens. Here you can configure a fast POST process and enable the external ports, which accelerate the boot time of the switch and provides access to the external ports of it. Use the settings shown in the figure. Figure 4-7 AMM web interface Chapter 4.
After you complete the POST process and enable the ports, click Configuration, and the window shown in Figure 4-8 opens. Figure 4-8 I/O Module tasks menu on the AMM From this menu, you can identify the switch on one of the high speed bays. In our case, the High Speed Switch (HSS) was installed in bay 7 of the BladeCenter-H chassis. 4.4.
4.4.3 Advanced options If you click Advanced Configuration in Figure 4-8 on page 144, the window shown in Figure 4-9 opens.
In the Advanced Configuration window, you can define the management ports to be visible from the external ports. Depending on your needs, this action might be necessary to access the switches. Typically, you enable this feature, unless you want to handle the switches only from the AMM interface. We do not describe how to perform this procedure in this book. 4.4.4 Telnet access You can start a telnet session from the window shown in Figure 4-9 on page 145. This session is similar to any TOR switch.
4.4.5 Web access You can access the web interface of the switch directly from the window shown in Figure 4-9 on page 145. Click Start Web Session, and a separate window on your web browser opens. From this window, you log in with your user name and password (Figure 4-11). Figure 4-11 Web interface web login window After a successful login, you see the IBM Networking OS web interface dashboard (Figure 4-12). Figure 4-12 IBM Networking OS web interface dashboard Chapter 4.
From this interface, you can perform different configuration tasks on the switch. For more details and configuration options, see the appropriate documentation: IBM RackSwitch G8264 Application Guide: http://www-01.ibm.com/support/docview.wss?rs=1126&context=HW500&dc=DA400&uid=is g3T7000326&loc=en_US&cs=utf-8&lang=en IBM Virtual Fabric 10Gb Switch Module for IBM BladeCenter - Installation Guide: http://publib.boulder.ibm.com/infocenter/bladectr/documentation/topic/com.ibm.b ladecenter.io_39Y9267.
4.4.7 Firmware upgrade from the AMM web interface The I/O Module Firmware Update option of the AMM web interface is currently not supported for the IBM 10Gb Switches. If you try to update the firmware using this option, you receive the message shown in Figure 4-14. Figure 4-14 Firmware upgrade from the AMM web interface is not possible 4.4.8 Working with users and passwords You can see users and passwords from the web interface.
Embedded switch specifics In a TOR switch, you can add a user by clicking the Add User button. You cannot perform this function from the BladeCenter embedded switch module. If you try to do it, you receive an error message, because in the BladeCenter chassis, user creation is handled by the AMM, and you cannot use the web interface. You can change passwords and work with users from the CLI, as described in “User management” on page 137. 4.
The various software components of SNEM are shown in Figure 4-16. Figure 4-16 SNEM solution architecture IBM System Networking Element Manager SNEM provides a single point for management that allows automation of basic network tasks, including remote monitoring and management of Ethernet switches. The benefits of SNEM include: Improves network visibility and drive reliability and performance.
Tivoli Network Manager can display network events, perform root-cause analysis of network events, and enrich network events with topology and other network data. Tivoli Network Manager integrates with other IBM products, such as IBM Tivoli Business Service Manager, Tivoli Application Dependency Discovery Manager, and IBM Systems Director. Using Tivoli Network Manager, you can perform the following tasks: Manage complex networks. View the network in multiple ways.
IBM System Networking Element Manager Solution Device Support List (6.1): http://www-01.ibm.com/support/docview.wss?uid=isg3T7000474 Quick Start Guide for installing and running KVM: http://publib.boulder.ibm.com/infocenter/lnxinfo/v3r0m0/topic/liaai/kvminstall/ kvminstall_pdf.pdf Chapter 4.
154 Implementing IBM System Networking 10Gb Ethernet Switches
5 Chapter 5. IBM System Networking RackSwitch implementation This chapter provides information and instructions for implementing the IBM System Networking 10Gb Top-of-Rack family switches, RackSwitch G8264R/F and RackSwitch G8124. Using the reference architecture described in Chapter 3, “Reference architectures” on page 107, this chapter presents a step by step guide for implementing and configuring the most important functions implemented in IBM Networking OS.
5.1 Layer 1 implementation This section describes Layer 1 related configuration and verification information for the implemented reference architecture. We describe the following topics: Network topology for Layer 1 configuration Configuration of the port settings 5.1.1 Network topology for Layer 1 configuration This section describes the Layer 1 implementation of the reference architecture.
5.1.2 Port settings configuration The physical connections details are provided in the Chapter 3, “Reference architectures” on page 107. Most Layer 1 aspects do not involve any configuration. In this section, we provide some useful commands for ports verification, that is, to make sure all the links are up and running before proceeding to upper layer configuration. Additional commands and details for Layer 1 configuration can be found in the technical documentation listed in 5.
Configuring QSFP+ The G8264 RackSwitch is equipped with QFSP+ ports that can operate either in 10 GbE or 40 GbE mode. Important: Changing the QSFP+ operation mode requires a reboot.
QSFP ports saved configuration: Port 1 - 40G Mode Port 5 - 40G Mode Port 9, 10, 11, 12 - 10G Mode Port 13, 14, 15, 16 - 10G Mode AGG1# 4. Reset the RackSwitch by running the command shown in Example 5-4. Example 5-4 Reset the switch AGG1#reload 5. Verify the current operation mode of the QSFP+ ports, as shown in Example 5-5.
11 12 13 14 15 16 17 Q10G Q10G Q10G Q10G Q10G Q10G SFP+ 18 SFP+ 19 SFP+ 20 SFP+ 21 SFP+ 22 SFP+ 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 160 SFP+ SFP+ SFP+ SFP+ SFP+ SFP+ SFP+ SFP+ SFP+ SFP+ SFP+ SFP+ SFP+ SFP+ SFP+ SFP+ SFP+ SFP+ SFP+ SFP+ SFP+ SFP+ SFP+ SFP+ SFP+ SFP+ SFP+ SFP+ SFP+ SFP+ SFP+ SFP+ SFP+ SFP+ SFP+ SFP+ SFP+ 3.C 3.D 4.A 4.B 4.C 4.
60 SFP+ 61 SFP+ 62 SFP+ 63 SFP+ 64 SFP+ AGG-1# 44 45 46 47 48 < < < < < NO NO NO NO NO Device Device Device Device Device Installed Installed Installed Installed Installed > > > > > Run show interface status or show interface link to display information about link, duplex, speed, and flow control. Example 5-8 shows the commands’ output.
41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 MGT AGG-1# 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 1G/10G 1G/10G 1G/10G 1G/10G 1G/10G 1G/10G 1G/10G 1G/10G 1G/10G 1G/10G 1G/10G 1G/10G 1G/10G 1G/10G 1G/10G 1G/10G 1G/10G 1G/10G 1G/10G 1G/10G 1G/10G 1G/10G 1G/10G 1G/10G 1000 full full full full full full full full full full full full full full full full full full full full full full full full full no no no no no no no no no no no no no no no no no n
Octets: UcastPkts: BroadcastPkts: MulticastPkts: FlowCtrlPkts: PriFlowCtrlPkts: Discards: Errors: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Ingress Discard reasons for port 2: VLAN Discards: 0 Empty Egress Portmap: 0 Filter Discards: 0 Policy Discards: 0 Non-Forwarding State: 0 IBP/CBP Discards: 0 -----------------------------------------------------------------Press q to quit, any other key to continue... 5.
The configuration topics described in this section are: VLANs and Port VLAN ID Numbers VLAN Tagging Protocol-based VLANs Private VLANs VLANs and Port VLAN ID numbers Here we show some basic switching configuration, such as configuring a VLAN, assigning a port to a VLAN, and configuring protocol-based VLANs and private VLANs. The RackSwitch G8264 and RackSwitch G8124 switches support up to 1024 VLANs per switch.
Important: Example 5-10 on page 164 is a template. Complete the configuration of the VLANs for all the reference architecture switches according to the details provided in the Chapter 3, “Reference architectures” on page 107. To show the VLAN configuration on the switch, run show vlan [information] (Example 5-11).
The resulting PVID assignment configuration is shown in Example 5-13.
Important: Example 5-13 on page 166 is a template. Complete the configuration of the ports and VLAN assignment for all the reference architecture switches according to the details provided in Chapter 3, “Reference architectures” on page 107. Run show vlan to show the VLANs member ports (Example 5-14).
To allow communication over a tagging enabled connection, the end ports of the switch must be declared members of the required VLANs to be transported over the link. Ports 5 and 6 on ACC-1 and ACC-2 are configured members of VLAN 10 and VLAN 20: ACC-2#configure terminal Enter configuration commands, one per line. ACC-2(config)#vlan 10,20 ACC-2(config-vlan)#member 5,6 ACC-2(config-vlan)#^Z ACC-2# End with Ctrl/Z. Important: This example is a template.
Protocol-based VLANs This feature is not part of our reference architecture implementation. However, for completeness, we provide a summary of commands used for configuration and verification in this section. For more information about the protocols-based VLANs concept, see Chapter 2, “IBM System Networking Switch 10Gb Ethernet switch features” on page 51. Important: This feature is supported only on RackSwitch G8264.
Run no protocol-vlan <1-8> at the VLAN level to delete the selected protocol configuration from the VLAN. Run show protocol-vlan <1-8> at the VLAN level to display current parameters for the selected PVLAN. Private VLANs This feature is not part of our reference architecture implementation. However, for completeness, we give a summary of the commands used for configuration and verification in this section.
Two trunk types are available: Static trunk groups (portchannel) Dynamic LACP trunk groups Regarding the two RackSwitch types: RackSwitch G8264 A RackSwitch G8264 switch supports up to 64 trunk groups (static and LACP). Each type can contain up to 16 member ports. RackSwitch G8124 A RackSwitch G8124 switch supports up to 24 trunk groups on the switch (static and LACP). Each type can contain up to eight member ports.
1. Add physical ports to a trunk group. Run [no] portchannel port from the global configuration mode to add or remove ports to or from a trunk group (Example 5-17). Example 5-17 Static trunk configuration AGG-1#configure terminal Enter configuration commands, one per line. AGG-1(config)#portchannel 1 port 17-18 AGG-1(config)#portchannel 2 port 19,20 AGG-1(config)#portchannel 3 port 1,5 AGG-1(config)#^Z AGG-1# End with Ctrl/Z. 2. Enable the trunk group.
Syntax: There are command syntax differences between the RackSwitch G8264 and RackSwitch G8124 switches. Both sets of commands are presented in Table 5-2 on page 173.
Run show portchannel hash to verify the global hash parameters shown in Example 5-20. Example 5-20 Display trunk hashing parameters AGG-1#show portchannel hash Current L2 trunk hash settings: smac Current L3 trunk hash settings: sip dip Current ingress port hash: enabled Current L4 port hash: disabled AGG-1# ACC-1#show portchannel hash Current Trunk Hash settings: smac ACC-1# IP has: Source IP and destination IP hash is enabled by default on RackSwitch G8264 switches. 4.
Verify the trunk group parameters by running show portchannel (Example 5-22). Example 5-22 Display trunk group parameters AGG-1#show portchannel 1 Protocol - Static Current settings: enabled ports: 17, 18 Current L2 trunk hash settings: smac Current L3 trunk hash settings: sip dip Current ingress port hash: enabled Current L4 port hash: disabled AGG-1# Dynamic trunks Link Aggregation Control Protocol (LACP) is an IEEE 802.
Important: When the system is initialized, all ports by default are in LACP off mode and are assigned unique admin keys. A dynamic trunk is configured in the reference architecture on the link between ACC-1 and ACC-2 Follow these steps to configure and activate LACP trunks: 1. Configure the global LACP parameters by running the following commands: a. Run lacp system-priority <1-65535> to define the priority value for the switch. Lower numbers provide higher priority. The default value is 32768.
c. Set the admin key for the selected ports by running lacp key <1-65535>. Only ports with the same adminkey and operkey (the operational state is generated internally) can form a LACP trunk group. The LACP admin key configuration is shown in Example 5-25. Example 5-25 LACP admin key configuration ACC-1#configure terminal Enter configuration commands, one per line. ACC-1(config)#interface port 5,6 ACC-1(config-if)#lacp key 3 ACC-1(config-if)#^Z ACC-1# End with Ctrl/Z. 3.
22 23 24 ACC-1# off off off 22 23 24 22 23 24 no no no 32768 32768 32768 ---- ---- ---- 1 1 1 c. Run show lacp aggregator <1-24> to show aggregation information for the selected admin key (Example 5-28).
5.2.3 Spanning Tree Protocol The Spanning Tree Protocol used for the reference architecture is Per-VLAN Rapid Spanning Tree (PVRST). PVRST mode is based on RSTP, which provides rapid Spanning Tree convergence, and allows for multiple Spanning Tree Groups (STGs), with STGs on a per-VLAN basis. PVRST mode is compatible with Cisco R-PVST/R-PVST+ mode. To simplify switch configuration, VLAN Automatic STG Assignment (VASA) can be used in SPT/PVST+ or PVRST modes.
According to the reference architecture described in Chapter 3, “Reference architectures” on page 107, the links between AGG1 and AGG2, AGG1 and ACC1, AGG1 and ACC2, AGG2 and ACC1, and AGG2 and ACC2 are point-to-point Layer 3 links. The corresponding VLANs do not span across the network, so no loops are formed. The STP runs on the default configuration. A configuration other than the default configuration is applied only for VLAN 10 and VLAN 20, which contains the hosts in our topology.
To verify the STP configuration and operation, run the following command with the output from the reference architecture. The spanning-tree stp [bridge|information]command shows detailed information about the STP operation. If you do not provide an STG number, the command output displays the STP information for all groups, as shown in Example 5-32. If you want to narrow the output to a specific STG, add the optional parameters.
Parameters: Priority 4116 Hello 2 MaxAge 20 FwdDel 15 Aging 300 Topology Change Counts 10 Port Prio Cost State Role Designated Bridge Des Port Type ------------- ---- ---------- ----- ---- ---------------------- -------- ---5 (pc15) 128 1990!+ FWD ROOT 0014-08:17:f4:34:4c:00 8027 P2P ! = Automatic path cost. + = Portchannel cost, not the individual port cost.
In Example 5-32 on page 181, you can see that the current switch (ACC-1) is the root for VLAN 10 and the ACC-2 switch is the root for VLAN 20, by checking the MAC addresses of the root bridge for each STG. You can find the MAC address of the current root by running show system: ACC-1#show system | include MAC MAC address: 08:17:f4:34:4d:00 IP (If 10) address: 10.0.10.
Run [no] qos dscp re-marking to turn on/off DSCP re-marking globally. Run show qos dscp to display the current DSCP parameters. Control plane protection Important: This feature is supported only on RackSwitch G8264 switches. You can use the following commands to limit the number of selected protocol packets received by the control plane (CP) of the switch. These limits help protect the CP from receiving too many protocol packets in a time period.
Run no qos protocol-packet-control rate-limit-packetqueue to clear the packet rate configured for the selected packet queue. Run show qos protocol-packet-control information protocol to display the mapping of protocol packet types to each packet queue number. The status indicates whether the protocol is running or not running. Run show qos protocol-packet-control information queue to display the packet rate configured for each packet queue.
5.3 Layer 3 This section provides configuration background information for using the RackSwitch G8124 and RackSwitch G8264 switches to perform IP routing functions. Any differences between the models regarding the availability of certain software features or command syntax are highlighted. All the configurations presented in this chapter are implemented by using the IBM Networking OS V6.8software installed in the reference architecture switches.
2. Complete the IP interface configuration. Complete the following steps to create and activate an IP interface (use the commands shown in Example 5-33): a. Create the IP interface. b. Define an IP address. c. Map the IP interface to a VLAN. d. Enable the IP interface. Example 5-33 IP interface configuration AGG-1#configure terminal Enter configuration commands, one per line. End with Ctrl/Z. AGG-1(config)#interface ip 100 (create the interface) AGG-1(config-ip-if)#ip address 10.0.100.1 255.255.255.
Default gateway Management IP addresses and gateways should already be configured during the initial setup. However, configuration guidelines are presented here for completeness. Important: This feature is used only for management interfaces in our reference architecture. Static routes are not used in the reference architecture; instead, we use a dynamic routing protocol (OSPF).
172.25.1.1: #2 172.25.1.1: #3 172.25.1.1: #4 172.25.1.1: #5 Ping finished. AGG-1# ok, ok, ok, ok, RTT RTT RTT RTT 1 1 1 1 msec. msec. msec. msec. Important: The default ping destination is management port unless otherwise specified. To test IP addresses reachability through the data ports, use the data-port option with the ping command as follows: AGG-1#ping 172.25.1.
Run no ip route [] to remove a static route. The destination address of the route to remove must be specified by using dotted decimal notation. Run no ip route destination-address to clear all static routes with the specified destination. Run no ip route gateway to clear all static routes that use the specified gateway.
Use the following steps to configure ECMP static routes. 1. Define multiple static routes for the same destination but with different gateways. Run ip route [] to define a static route. Enter all addresses by using dotted decimal notation, as shown in Example 5-39. Example 5-39 ECMP static routes configuration ACC-1#configure terminal Enter configuration commands, one per line. End with Ctrl/Z. ACC-1(config)#ip route 10.0.0.0 255.255.0.0 10.0.101.
Run [no] ip route healthcheck to enable or disable static route health checks. The default setting is disabled. 3. Verify the ECMP static routes operation. Run show ip route static to display the static routes information. You can see the difference between simple static routes and ECMP static routes in Example 5-41.
Important: For a complete list of the supported features, see the IBM Networking OS 6.8 Features Summary at: http://www.ibm.com/support/docview.wss?uid=isg3T7000470 At the time of writing, the following IPv6 features were not supported in IBM Networking OS V6.8: Dynamic Host Control Protocol for IPv6 (DHCPv6) Border Gateway Protocol for IPv6 (BGP) Routing Information Protocol for IPv6 (RIPng) You can use IBM Networking OS V6.
You cannot configure an IPv4 address on an IPv6 interface. Each interface can be configured with only one address type, either IPv4 or IPv6, but not both. When changing between IPv4 and IPv6 address formats, the prior address settings for the interface are discarded. Each IPv6 interface can belong to only one VLAN. Each VLAN can support only one IPv6 interface. Each VLAN can support multiple IPv4 interfaces.
IPv6 interfaces and VLAN: The IPv6 interfaces are mapped to the same VLAN as their IPv4 pairs: AGG-1# interface ip 101 ip address 10.0.101.2 255.255.255.252 vlan 101 enable exit interface ip 111 ipv6 address fc11:0:0:0:0:0:0:2 64 vlan 101 enable ip6host exit AGG-1# 2. Configure the router nodes interfaces (Example 5-43). Run ipv6 address [anycast|enable] to configure a primary unicast IPv6 address on an interface.
Run ipv6 nd prefix to define the prefix to be advertised (Example 5-44). Example 5-44 Neighbor Discover protocol configuration ACC-1#configure terminal Enter configuration commands, one per line. End with Ctrl/Z. ACC-1(config)#interface ip 106 ACC-1(config-ip-if)#no ipv6 nd suppress-ra ACC-1(config-ip-if)#ipv6 nd prefix fc10:0:0:0:0:0:0:0 64 ACC-1(config-ip-if)#^Z ACC-1# 3. Host configuration. We now define the host part of the configuration for Windows and Linux.
IPv4 and IPv6 are configured on the same server interface (Teaming Adapter). There is no need for additional configuration of the switch port, as shown in Example 5-45. Example 5-45 Switch port configuration for a dual TCP/IP stack host ACC-1#show interface information 7 Alias Port Tag RMON Lrn Fld PVID NAME ------- ---- --- ---- --- --- ----- -------------------------------------------7 7 n d e e 10 SRV1 10 VLAN(s) * = PVID is tagged.
IPv4 and IPv6 addresses are configured by using the input information from Chapter 3, “Reference architectures” on page 107. The static IPv4 and IPv6 configuration on the interface is shown in Figure 5-4. Figure 5-4 Windows IPv4 and IPv6 static addresses configuration If the Router Advertisements protocol is enabled on the switch (see step 2 on page 195), then the IPv6 interface on the host auto-configures itself using the advertised prefix, as shown in Figure 5-5.
Both default gateways (IPv4 and IPv6) are reachable, as shown in Figure 5-6. Figure 5-6 IPv4 and IPv6 gateway reachability – Configure Linux hosts. For the Embedded switch reference architecture (see Chapter 6, “IBM Virtual Fabric 10Gb Switch Module implementation” on page 239), we used a blade server that runs the Red Hat Enterprise Linux operating system connected internally to the two 10GbE Virtual Fabric Switches in the Blade Center chassis.
ACC-3#show running-config | begin 1:14 interface port 1:14 name "SRV-3" pvid 30 exit ACC-3#show running-config | begin 2:14 interface port 2:14 name "SRV-3" pvid 30 exit ACC-3# We start with the bonding interface already created but with no IP address, as shown in Figure 5-7.
Because the Router Advertisement protocol is enabled on the switch, the IPv6 address is auto-configured on the bond interface, as shown in Figure 5-8. Figure 5-8 Linux host IPv6 autoconfiguration Chapter 5.
For our reference architecture, we use static IPv6 configuration. For a static IPv4 and IPv6 configuration example, see Figure 5-9. Figure 5-9 Linux static IPv4 and IPv6 configuration Important: This method is the only method for configuring IP addresses and IP routes on Linux. This configuration is also a temporary configuration that is discarded at reboot. For a permanent IP configuration in Linux, see the appropriate Red Hat Linux documentation.
Note: The following guidelines and limitations apply when configuring IPv4 and IPv6 interfaces: You cannot configure an IPv4 address on an IPv6 interface. Each interface can be configured with only one address type, either IPv4 or IPv6, but not both. Each IPv6 interface can belong to only one VLAN. Each VLAN can support only one IPv6 interface. Each VLAN can support multiple IPv4 interfaces.
For SRV-1 to reach SRV-3 on the other side of the network, it must rely on the routing mechanisms in the middle. OSPF must redistribute the directly connected networks of SRV-1 and SRV-3 to the other routers on the network. The task is considered complete when SRV-1 is able to reach SRV-3 on both IPv4 and IPv6. Reachability: The reachability test between SRV-1 and SRV-3 assumes that the 10Gb Virtual Fabric Switch configuration is complete and operational.
AGG-2(config-ip-loopback)#ip address 1.1.1.2 255.255.255.255 AGG-2(config-ip-loopback)#ip router-id 1.1.1.2 AGG-2(config)#^Z AGG-2# ACC-1#configure terminal Enter configuration commands, one per line. End with Ctrl/Z. ACC-1(config)#interface loopback 1 ACC-1(config-ip-loopback)#ip address 2.2.2.1 255.255.255.255 ACC-1(config-ip-loopback)#ip router-id 2.2.2.1 ACC-1(config)#^Z ACC-1# ACC-2#configure terminal Enter configuration commands, one per line. End with Ctrl/Z.
Important: The area option is an arbitrary index used only on the switch and does not represent the actual OSPF area number. The actual OSPF area number is defined in the area-id. The area option is an arbitrary index (0 - 5) used only by the switch. This index number does not necessarily represent the OSPF area number, though for configuration simplicity, it should do so where possible. The area index and area ID configuration is shown in Example 5-49.
5. Attach an area to a network. After an OSPF area is defined, it must be associated with a network. To attach the area to a network, you must assign the OSPF area index to an IP interface that participates in the area. Run ip ospf area and ip ospf enable under the interface configuration mode, as shown in Example 5-51. If authentication with MD5 is used, assign an MD5 key ID to OSPF interfaces by running ip ospf message-digest-key .
ACC-2#conf t Enter configuration commands, one per line. End with Ctrl/Z. ACC-2(config)#router ospf ACC-2(config-router-ospf)#redistribute fixed export 10 1 ACC-2(config-router-ospf)#^Z ACC-2# 7. Verify OSPF. Run the following commands to verify OSPF operation: – – – – – show show show show show ip ip ip ip ip ospf ospf neighbor ospf database database ospf routes route See the following examples from ACC-1 switch for these commands’ output.
102 1.1.1.2 ACC-1# 1 Full 10.0.102.2 Show the OSPF database information by running show ip ospf database database (Example 5-55). Example 5-55 show ip ospf database database command output ACC-1#show ip ospf database AS External LSAs Link ID 10.0.10.0 10.0.10.0 10.0.30.0 10.0.30.0 1.1.1.2 1.1.1.1 2.2.2.2 2.2.2.1 10.0.20.0 10.0.20.0 ADV Router 2.2.2.1 2.2.2.2 1.1.1.1 1.1.1.2 1.1.1.2 1.1.1.1 2.2.2.2 2.2.2.1 2.2.2.1 2.2.2.
*E1 2.2.2.2/32 via 10.0.102.2 E1 10.0.10.0/24 via 10.0.101.2 E1 10.0.10.0/24 via 10.0.102.2 E1 10.0.20.0/24 via 10.0.101.2 E1 10.0.20.0/24 via 10.0.102.2 *E1 10.0.30.0/24 via 10.0.101.2 * 10.0.100.0/30 via 10.0.101.2 * 10.0.100.0/30 via 10.0.102.2 * 10.0.103.0/30 via 10.0.101.2 * 10.0.104.0/30 via 10.0.102.2 ACC-1# Show the global routing table by running show ip route (Example 5-57).
* 224.0.0.5 * 224.0.0.6 * 224.0.0.18 * 255.255.255.255 ACC-1# 255.255.255.255 255.255.255.255 255.255.255.255 255.255.255.255 0.0.0.0 0.0.0.0 0.0.0.0 255.255.255.255 multicast multicast multicast broadcast addr addr addr broadcast OSPFv3 configuration OSPF version 3 is based on OSPF version 2, but is modified to support IPv6 addressing.
2. Enable OSPF. Run ipv6 router ospf to enter the protocol configuration mode (Example 5-58). Example 5-58 OSPF activation ACC-1#configure terminal Enter configuration commands, one per line. End with Ctrl/Z. ACC-1(config)#ipv6 router ospf ACC-1(config-router-ospf3)#router-id 2.2.2.1 ACC-1(config-router-ospf3)#enable ACC-1(config-router-ospf3)#^Z ACC-1# Important: This operation is performed for all four Layer 3 switches (AGG-1, AGG-2, ACC-1, and ACC-2) 3. Define areas.
4. Attach an area to a network. After an OSPF area is defined, it must be associated with a network. To attach the area to a network, you must assign the OSPF area index to an IP interface that participates in the area. Run ipv6 ospf area and ipv6 ospf enable in interface configuration mode (Example 5-51 on page 207). Example 5-59 Attach an area to a network ACC-1#configure terminal Enter configuration commands, one per line.
ACC-2(config-router-ospf3)#^Z ACC-2# 6. Verify OSPF. Run the following commands to verify OSPF operation: – – – – – show show show show show ipv6 ipv6 ipv6 ipv6 ipv6 ospf ospf neighbor ospf database ospf routes route See the following examples for the ACC-1 switch for these commands’ output. Show the current OSPFv3 configuration settings by running show ipv6 ospf (Example 5-61). Example 5-61 Show ipv6 ospf command output ACC-1#show ipv6 ospf Current OSPFv3 settings: ON Router ID: 2.2.2.
Show information about OSPFv3-formed adjacencies by running show ipv6 ospf neighbor (Example 5-62). Example 5-62 show ipv6 ospf neighbor command output ACC-1#show ipv6 ospf neighbor ID Pri State 1.1.1.1 1 FULL/DR fe80::a17:f4ff:fe32:c46e 1.1.1.2 1 FULL/DR fe80::fecf:62ff:fe9d:9a6f ACC-1# DeadTime 33 Address 38 Show OSPFv3 database information by running show ipv6 ospf database (Example 5-63). Example 5-63 show ipv6 ospf database command output ACC-1#show ipv6 ospf database Router LSAs (Area 0.0.0.
RtrId 1.1.1.1 1.1.1.1 1.1.1.1 1.1.1.1 1.1.1.2 1.1.1.2 1.1.1.2 1.1.1.2 2.2.2.1 2.2.2.1 2.2.2.1 2.2.2.2 2.2.2.2 2.2.2.
via fe80::fecf:62ff:fe9d:9a6f, Interface 112 fc10::/64 [1/1] via ::, Interface 106 C fc11::/64 [1/1] via ::, Interface 111 C fc12::/64 [1/1] via ::, Interface 112 O fc13::/64 [2/110] via fe80::a17:f4ff:fe32:c46e, Interface 111 O fc14::/64 [2/110] via fe80::fecf:62ff:fe9d:9a6f, Interface 112 O fc30::/64 [3/110] via fe80::a17:f4ff:fe32:c46e, Interface 111 C fe80::a17:f4ff:fe34:4d69/128 [1/1] via ::, Interface 106 C fe80::a17:f4ff:fe34:4d6e/128 [1/1] via ::, Interface 111 C fe80::a17:f4ff:fe34:4d6f/128 [1/1] v
Windows host verification Figure 5-11 shows that the Windows host is able to ping the Linux host by using both IPv4 and IPv6.
Linux host verification Figure 5-12 shows the output from an ICMP test from a Linux host to a Windows host. Figure 5-12 Linux host to Windows host verification 5.3.3 Border Gateway Protocol Border Gateway Protocol (BGP) is not in the scope of the reference architecture implementation of this publication. However, a summary of commands used for configuration and verification is presented in this section. For more information about BGP, see 2.3.5, “Border Gateway Protocol” on page 66.
You can use BGP commands to configure the switch to receive routes and to advertise static routes, fixed routes, and virtual server IP addresses with other internal and external routers. In the current IBM Networking OS implementation, the RackSwitch G8264 switch does not advertise BGP routes that are learned from one iBGP speaker to another iBGP speaker. Important: BGP is turned off by default.
Run neighbor route-origination-interval <1-65535> at the BGP level to set the minimum time between route originations, in seconds. The default value is 15 seconds. Run neighbor time-to-live <1-255> at the BGP level to configure the TTL value for a specified peer. Time-to-live (TTL) is a value in an IP packet that tells a network router whether the packet has been in the network too long and should be discarded.
– Redistribute: Default routes are either configured through default gateway or learned through other protocols and redistributed to peers. If the routes are learned from default gateway configuration, you must enable static routes, because the routes from default gateway are static routes. Similarly, if the routes are learned from a certain routing protocol, you must enable that protocol. Note: No routes are configured in our configuration.
5.4.1 Virtual Router Redundancy Protocol The RackSwitch G8264 and RackSwitch G8124 switches support IPv4 high-availability network topologies through an enhanced implementation of VRRP. VRRP enables redundant router configurations within a LAN, providing alternative router paths for a host to eliminate single points of failure within a network. Each participating VRRP-capable routing device is configured with the same virtual router IPv4 address and ID number.
ACC-1(config)#router vrrp ACC-1(config-vrrp)#virtual-router 1 virtual-router-id 10 ACC-1(config-vrrp)# ACC-2#configure terminal Enter configuration commands, one per line. End with Ctrl/Z. ACC-2(config)#router vrrp ACC-2(config-vrrp)#virtual-router 1 virtual-router-id 10 ACC-2(config-vrrp)# b. Configure the Virtual router IP address (Example 5-68). Run [no] virtual-router <1-15> address to define an IP address for this virtual router, using dotted decimal notation.
d. Define the election priority (Example 5-70). Run virtual-router <1-15> priority <1-254> to define the election priority bias for this virtual server. The priority value can be any integer 1 - 254. The default value is 100. During the master router election process, the routing device with the highest virtual router priority number wins. If there is a tie, the device with the highest IP interface address wins.
Run [no] virtual-router <1-128> fast-advertise to enable or disable fast advertisements. When enabled, the VRRP master advertisements interval is calculated in units of centiseconds, instead of seconds. For example, if advertisement is set to 1 and fast advertisement is enabled, master advertisements are sent every .01 second. When you disable fast advertisement, the advertisement interval is set to the default value of 1 second. To support Fast Advertisements, set the interval to 20 - 100 centiseconds.
ACC-1#configure terminal Enter configuration commands, one per line. End with Ctrl/Z. ACC-1(config)#router vrrp ACC-1(config-vrrp)#virtual-router 1 enable ACC-1(config-vrrp)# Aug 15 23:45:18 ACC-1 NOTICE Aug 15 23:45:23 ACC-1 NOTICE vrrp: virtual router vrrp: virtual router 10.0.10.1 is now BACKUP 10.0.10.1 is now MASTER. ACC-1(config-vrrp)#^Z ACC-1# 3. Configure tracking.
– Run tracking-priority-increment ports <0-254> to define the priority increment value for active ports on the virtual router’s VLAN. The default value is 2. Tracking configuration is shown in Example 5-74. Example 5-74 Configure tracking ACC-1#configure terminal Enter configuration commands, one per line. End with Ctrl/Z.
track nothing Current VRRP virtual router settings: 1: vrid 10, 10.0.10.1, if 10, prio 100, adver 2, enabled preem enabled, predelay 5, fast-advertisement disabled track ports ACC-2# Run show ip vrrp virtual-router <1-15> to show the current VRRP parameters of the selected virtual router (Example 5-76). Example 5-76 Verify the selected virtual router current parameters ACC-1#show ip vrrp virtual-router 1 Current VRRP virtual router 1: vrid 10, 10.0.10.
track ports ACC-1# 5.4.2 Layer 2 Failover The primary application for Layer 2 Failover is to support Network Adapter Teaming. With Network Adapter Teaming, all the NICs on each server share an IP address, and are configured into a team. One NIC is the primary link, and the other is a standby link. For more details, see the documentation for your Ethernet adapter. Link limits: Only two links per server can be used for Layer 2 Failover (one primary and one backup).
2. Configure the Failover Manual Monitor Port. Run [no] failover trigger <1-8> mmon monitor portchannel command to add or remove the selected trunk group to the Manual Monitor Port configuration (Example 5-81). These ports are the whose states the switch monitors to control the server port link. Example 5-81 Failover Manual Monitor Port configuration ACC-1#configure terminal Enter configuration commands, one per line. End with Ctrl/Z.
4. Configure the Failover Trigger limit. Run failover trigger <1-8> limit <0-1024> to configure the minimum number of operational links allowed within each trigger before the trigger initiates a failover event (Example 5-83). If you enter a value of zero (0), the switch triggers a failover event only when no links in the trigger are operational. Example 5-83 Failover Trigger limit configuration ACC-1#configure terminal Enter configuration commands, one per line.
6. Verify the failover configuration. Run show failover trigger <1-8> to show the current Failover Trigger settings (Example 5-85). Example 5-85 Display current fAilover Trigger settings ACC-1#show failover trigger 1 Current Trigger 1 setting: enabled limit 2 Manual Monitor settings: trunks 1 2 Manual Control settings: ports 7 ACC-1# ACC-2#show failover trigger 1 Current Trigger 1 setting: enabled limit 2 Manual Monitor settings: trunks 1 2 Manual Control settings: ports 7 ACC-2# 7.
Aug 11 12:59:21 ACC-1 NOTICE link: link down on port 7 ACC-1(config-if)#^Z ACC-1# b. Verify the Failover Trigger status. Run show failover trigger <1-8> information to verify the failover trigger status (Example 5-87).
15 16 17 18 19 20 21 22 23 24 MGTA MGTB ACC-1# 15 16 17 18 19 20 21 22 23 24 25 26 1G/10G 10000 1G/10G 1G/10G 1G/10G 1G/10G 1G/10G 1G/10G 1G/10G 10000 1000 any full full full full full full full full full full full any no no no no no no no no no no yes yes no no no no no no no no no no yes yes down up down down down down down down down up up up d. Enable the two uplink ports. Run no shutdown to manually enable the ports and see the messages related to the failover operation (Example 5-89).
2 Operational PortChannel 2 3 Operational 4 Operational Control State: Auto Controlled Member Status ------------------7 Operational ACC-1# f. Verify the interface status. Run show interface status to verify that the control port (port 7) is also disabled along with the manually disabled uplink ports (1,2) (Example 5-91).
For detailed information about trunking, see 5.2.2, “Ports and trunking” on page 170. 5.4.4 Hot Links Important: The Hot Links function was not used in the reference architecture implementation. The following section is just a short outline of the feature. For more information, see 5.5, “More information” on page 238. For network topologies that require STP to be turned off, the Hot Links function provides basic link redundancy with fast recovery. Hot Links consists of up to 25 triggers.
5.5 More information For more information about the topics described in this chapter, see the following documentation Configuration guides: – IBM RackSwitch G8264 Application Guide (6.8): http://www-01.ibm.com/support/docview.wss?uid=isg3T7000464 – IBM RackSwitch G8124/G8124-E Application Guide (6.8): http://www-01.ibm.com/support/docview.wss?uid=isg3T7000465 Command reference guides: – IBM RackSwitch G8264 Menu-based CLI Reference (6.8): http://www-01.ibm.com/support/docview.
6 Chapter 6. IBM Virtual Fabric 10Gb Switch Module implementation This chapter provides information and instructions for implementing the IBM Virtual Fabric 10Gb Switch Module for IBM BladeCenter. Using the reference architecture described in Chapter 3, “Reference architectures” on page 107, this chapter presents a step by step guide for implementing and configuring the most important functions available in IBM Networking OS. It is not meant to cover all the features in the operating system.
6.1 Purpose of this implementation This implementation shows how a mixed environment of both stand-alone and embedded switches provide end-to-end communication in a data center, for servers that run different operating systems and IP protocol versions (IPv4 and IPv6). At the conclusion of this implementation, the Linux host installed in the BladeCenter H chassis is able to communicate with the Windows host connected to the G8124 switches of a Top-of-Rack implementation.
6.2.1 Stacking overview A stack is a group of up to eight Virtual Fabric 10Gb Switch Module switches with IBM Networking OS that work together as a unified system. A stack has the following properties, regardless of the number of switches included: The network views the stack as a single entity. The stack can be accessed and managed as a whole using standard switch IP interfaces configured with IPv4 addresses.
Port flood blocking Protocol-based VLANs RIP Router IDs Route maps sFlow port monitoring Static MAC address addition Static multicast Uni-Directional Link Detection (UDLD) Virtual NICs Virtual Router Redundancy Protocol (VRRP) Important: In stacking mode, switch menus and commands for unsupported features might be unavailable, or might have no effect on switch operation. 6.2.
If multiple stack links or stack Member switches fail, and separate the Master and Backup into separate substacks, the Backup automatically becomes an active Master for the partial stack in which it is. Later, if the topology failures are corrected, the partial stacks merge, and the two active Masters come into contact.
The active Master is rebooted with the boot configuration set to factory defaults (clearing the Backup setting). Master failover When the Master switch is present, it controls the operation of the stack and pushes configuration information to the other switches in the stack. If the active Master fails, then the designated Backup (if one is defined in the Master’s configuration) becomes the new acting Master and the stack continues to operate normally.
3. Configure the same stacking VLAN for all switches in the stack. 4. Configure the stacking interlinks. 5. Configure an external IP interface on the Master (if external management is wanted). 6. Bind Member switches to the Master. 7. Assign a Backup switch. These tasks are covered in detail in the following sections. Preferred configuration practices Here are guidelines for building an effective switch stack: Always connect the stack switches in a complete ring topology.
2. Set the stacking membership mode. On each switch, set the stacking membership mode by running boot stack mode {master|member} (Example 6-2). By default, each switch is set to Member mode. However, one switch must be set to Master mode.
5. Physically connect the stack trunks. Connect the stacking links (Figure 6-1) and verify that the link is up. Figure 6-1 Stack links physical connections 6. Verify the stacking configuration. Verify the saved stacking configuration by running show boot stack (Example 6-5).
Switch Mode : Member Stack Trunk Ports: EXT9 EXT10 Stack VLAN : 4090 ACC-4# 7. Reboot the switches and verify the stack operation. Important: After the switches restart, the stack is formed but it is not yet operational. Some observations can be made: The switches loaded the default configuration (see Example 6-6). The member switch is attached to the stack but is not bound to it (see Example 6-6). The member switches interfaces are not operational. Their link status is dettached (see Example 6-7).
asnum UUID Bay MAC csnum State -----------------------------------------------------------------------------A1 05e9050bcd92450f903d7e60c581e4a4 7 00:25:03:6e:77:00 C1 IN_STACK A2 05e9050bcd92450f903d7e60c581e4a4 9 fc:cf:62:0a:49:00 ATTACH Router# Run show interface status to verify the stack members port status (Example 6-7).
2:15 2:16 2:17 2:18 2:19 2:20 2:21 2:22 2:23 2:24 2:25 2:26 2:27 3:1 3:2 3:3 3:4 3:5 3:6 3:7 3:8 3:9 3:10 3:11 3:12 3:13 3:14 3:15 3:16 3:17 3:18 3:19 3:20 3:21 3:22 3:23 3:24 3:25 3:26 3:27 4:1 4:2 4:3 4:4 4:5 4:6 4:7 4:8 4:9 4:10 4:11 4:12 4:13 4:14 4:15 250 79 80 81 82 83 84 85 86 87 88 89 90 91 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 100 100 1G/10G 1G/10G 1G/10G 1G/10G 1G/10
4:16 4:17 4:18 4:19 4:20 4:21 4:22 4:23 4:24 4:25 4:26 4:27 5:1 5:2 5:3 5:4 5:5 5:6 5:7 5:8 5:9 5:10 5:11 5:12 5:13 5:14 5:15 5:16 5:17 5:18 5:19 5:20 5:21 5:22 5:23 5:24 5:25 5:26 5:27 6:1 6:2 6:3 6:4 6:5 6:6 6:7 6:8 6:9 6:10 6:11 6:12 6:13 6:14 6:15 6:16 208 209 210 211 212 213 214 215 216 217 218 219 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 100 1G/10G 1G/10G 1G/10G 1G/10G 1
6:17 6:18 6:19 6:20 6:21 6:22 6:23 6:24 6:25 6:26 6:27 7:1 7:2 7:3 7:4 7:5 7:6 7:7 7:8 7:9 7:10 7:11 7:12 7:13 7:14 7:15 7:16 7:17 7:18 7:19 7:20 7:21 7:22 7:23 7:24 7:25 7:26 7:27 8:1 8:2 8:3 8:4 8:5 8:6 8:7 8:8 8:9 8:10 8:11 8:12 8:13 8:14 8:15 8:16 8:17 252 337 338 339 340 341 342 343 344 345 346 347 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 1G/10G 1G/10G 1G/10G 1G/10G
8:18 8:19 8:20 8:21 8:22 8:23 8:24 8:25 8:26 8:27 Router# 466 467 468 469 470 471 472 473 474 475 1G/10G 1G/10G 1G/10G 1G/10G 1G/10G 1G/10G 1G/10G 1G/10G 1G/10G any full full full full full full full full full any no no no no no no no no no no no no no no no no no no no no detached detached detached detached detached detached detached detached detached detached 8. Bind Members to the stack. Important: To fully activate the Member switches, you must bind them to the stack.
----------------------------------------------------------------------csnum UUID Bay MAC asnum ----------------------------------------------------------------------C1 05e9050bcd92450f903d7e60c581e4a4 7 00:25:03:6e:77:00 A1 C2 05e9050bcd92450f903d7e60c581e4a4 9 fc:cf:62:0a:49:00 A2 Attached Switches in Stack: -----------------------------------------------------------------------------asnum UUID Bay MAC csnum State -----------------------------------------------------------------------------A1 05e9050bcd924
Master switch: csnum MAC UUID Bay Number - 1 00:25:03:6e:77:00 05e9050bcd92450f903d7e60c581e4a4 7 Backup switch: csnum MAC UUID Bay Number - 2 fc:cf:62:0a:49:00 05e9050bcd92450f903d7e60c581e4a4 9 Configured Switches: ----------------------------------------------------------------------csnum UUID Bay MAC asnum ----------------------------------------------------------------------C1 05e9050bcd92450f903d7e60c581e4a4 7 00:25:03:6e:77:00 A1 C2 05e9050bcd92450f903d7e60c581e4a4 9 fc:cf:62:0a:49:00 A2 Attach
6.3.1 Network topology for Layer 1 configuration This section presents the Layer 1 implementation of the reference architecture. Figure 6-2shows the physical connections of the lab equipment that is used to demonstrate the examples in this chapter. Figure 6-2 Physical topology All the equipment used in the reference architecture uses IBM Networking OS V6.8, and the configuration, statistics, and information commands were tested on the switches. 6.3.
Additional commands and details for Layer 1 configuration can be found in the technical documentation listed in 6.7, “More information” on page 284 Port link configuration IBM switches include a factory default configuration that enables interfaces with the following link settings: In the copper Gigabit Ethernet interfaces: – Auto-negotiation is set. – The speed for 10/100/1000 RJ45 (copper) Gigabit Ethernet interfaces is set to auto, so that the interface can operate at 10 Mbps, 100 Mbps, or 1 Gbps.
Run show interface transceiver to show the installed transceivers type, part number, serial number, laser type, and status. Example 6-12 shows the command’s output.
1:21 1:22 1:23 1:24 1:25 1:26 1:27 2:1 2:2 2:3 2:4 2:5 2:6 2:7 2:8 2:9 2:10 2:11 2:12 2:13 2:14 2:15 2:16 2:17 2:18 2:19 2:20 2:21 2:22 2:23 2:24 2:25 2:26 2:27 ACC-3# 21 22 23 24 25 26 27 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 1000 1G/10G 1G/10G 1G/10G 10000 10000 any 1G/10G 10000 1G/10G 10000 10000 1G/10G 1G/10G 10000 1G/10G 1G/10G 1G/10G 1G/10G 1G/10G 1G/10G 100 100 10000 10000 1G/10G 1G/10G 1G/10G 1G/10G 1G/10G 1G/10G 10000 10000 any full full full full full
Ingress Discard reasons for port 1:17: VLAN Discards: 0 Empty Egress Portmap: 0 Filter Discards: 0 Policy Discards: 0 Non-Forwarding State: 0 IBP/CBP Discards: 0 -----------------------------------------------------------------Interface statistics for port 1:18: ifHCIn Counters ifHCOut Counters Octets: 49414808 4537173 UcastPkts: 0 0 BroadcastPkts: 1845 0 MulticastPkts: 712449 15594 FlowCtrlPkts: 0 0 PriFlowCtrlPkts: 0 0 Discards: 462838 462644 Errors: 0 0 Ingress Discard reasons for port 1:18: VLAN Discard
6.4.1 VLANs The VLAN-related configuration applied to the reference architecture switches is described in Chapter 3, “Reference architectures” on page 107. The configuration topics described in this section are: VLANs and port VLAN ID numbers VLAN tagging Private VLANs VLANs and port VLAN ID numbers This section shows some basic switching configuration, such as configuring a VLAN, assigning a port to a VLAN, and configuring private VLANs.
Port 1:17 is an UNTAGGED Port 1:18 is an UNTAGGED Port 2:14 is an UNTAGGED Port 2:17 is an UNTAGGED Port 2:18 is an UNTAGGED ACC-3(config-vlan)#^Z ACC-3# port port port port port and and and and and its its its its its PVID PVID PVID PVID PVID is is is is is changed changed changed changed changed from from from from from 1 1 1 1 1 to to to to to 30 30 30 30 30 3. To verify the VLANs, show the VLAN configuration on the switch by running show vlan [information], as shown in Example 6-17.
To enable tagging for an untagged port, run tagging in interface configuration mode (Example 6-18). Example 6-18 Tagging configuration ACC-3#configure terminal ACC-3(config-if)#interface port 1:1 ACC-3(config-if)#tagging ACC-3(config-if)#^Z ACC-3# To allow communication over a tagging enabled connection, the end ports of the switch must be declared members of the required VLANs to be transported over the link.
2:7 71 y Internal 2:8 72 y Internal 2:9 73 y Internal 2:10 74 y Internal 2:11 75 y Internal 2:12 76 y Internal 2:13 77 y Internal 2:14 78 y Internal 2:15 79 y RemoteMgmt 2:16 80 y RemoteMgmt 2:17 81 n External 2:18 82 n External 2:19 83 n External 2:20 84 n External 2:21 85 n External 2:22 86 n External 2:23 87 n External 2:24 88 n External 2:25 89 n Stacking 2:26 90 n Stacking 2:27 91 n External * = PVID is tagged.
6.4.2 Ports and trunking When using port trunk groups between two switches, you can create a virtual link between the switches, operating with combined throughput levels that depend on how many physical ports are included. Two trunk types are available: Static trunk groups (portchannel), and dynamic LACP trunk groups. Up to 18 trunks of each type are supported in stand-alone (non-stacking) mode, and 64 trunks of each type are supported in stacking mode, depending of the number and type of available ports.
When a trunk is enabled, the trunk Spanning Tree participation setting takes precedence over any trunk member. 802.1X authentication is not supported on ISL ports or on any port that is part of a trunk. You cannot configure a trunk member as a monitor port in a port-mirroring configuration. Trunks cannot be monitored by a monitor port; however, trunk members can be monitored. All ports in static trunks must have the same link configuration (speed, duplex, and flow control).
Trunk hash parameters are set globally. You can enable one or two parameters to configure any of the following valid combinations: – – – – – – SMAC (source MAC only) DMAC (destination MAC only) SIP (source IP only) DIP (destination IP only) SIP + DIP (source IP and destination IP) SMAC + DMAC (source MAC and destination MAC) For trunk hashing configuration commands, see Table 6-1.
Current L4 port hash: enabled ACC-3# 4. Verify the trunk group configuration. Run the following commands to verify the trunk configuration and status. To verify the trunk group status, run show portchannel [information] or show portchannel [information] (Example 6-24).
6.4.3 Spanning Tree Protocol The STP used for the reference architecture is Per-VLAN Rapid Spanning Tree (PVRST). PVRST mode is based on RSTP, which provides rapid Spanning Tree convergence, but allows for multiple Spanning Tree Groups (STGs), with STGs on a per-VLAN basis. PVRST mode is compatible with Cisco R-PVST/R-PVST+ mode. To simplify the switch configuration, VLAN Automatic STG Assignment (VASA) can be used in SPT/PVST+ or PVRST modes.
Run spanning-tree stp bridge priority <0-65535> to configure the bridge priority for VLAN 30, as shown in Example 6-27. Example 6-27 Bridge priority configuration example AGG-2#conf t Enter configuration commands, one per line. End with Ctrl/Z.
Port 1:17 : Priority 128, Path Cost 0, auto Port 1:18 : Priority 128, Path Cost 0, auto Port 2:14 : Priority 128, Path Cost 0, auto, edge, Spanning Tree turned OFF Port 2:17 : Priority 128, Path Cost 0, auto Port 2:18 : Priority 128, Path Cost 0, auto ACC-3# ACC-3# ACC-3#show spanning-tree stp 30 information ----------------------------------------------------------------------------------------------------------------------------------Spanning Tree Group 30: On (PVRST) VLANs: 30 Current Root: 001e 08:17:f4
To configure the 802.1p parameters, run the following commands: Run qos transmit-queue mapping to map the 802.1p priority to the Class of Service queue (COSq) priority. Enter the 802.1p priority value (0 - 7), followed by the Class of Service queue that handles the matching traffic. Run qos transmit-queue weight-cos to configure the weight of the selected Class of Service queue (COSq).
6.5.2 Layer 2 Failover The primary application for Layer 2 Failover is to support Network Adapter Teaming. With Network Adapter Teaming, all the NICs on each server share an IP address, and are configured into a team. One NIC is the primary link, and the other is a standby link. For more details, see the documentation for your Ethernet adapter. Important: Only two links per server can be used for Layer 2 Trunk Failover (one primary and one backup).
Hot Links has up to 25 triggers. A trigger is a pair of Layer 2 interfaces, each containing an individual port, trunk, or LACP adminkey. One interface is the Master, and the other is a Backup. While the Master interface is set to the active state and forwards traffic, the Backup interface is set to the standby state and blocks traffic until the Master interface fails. If the Master interface fails, the Backup interface is set to active and forwards traffic.
To enable, configure, and verify VRRP in IBM Networking OS switches, complete the following steps: 1. Enable VRRP. Run router vrrp to enter the VRRP configuration mode and enable to activate the protocol on both AGG-1 and AGG-2 switches (Example 6-29). Example 6-29 Enable VRRP AGG-1#configure terminal Enter configuration commands, one per line. AGG-1(config)#router vrrp AGG-1(config-vrrp)#enable AGG-2#configure terminal Enter configuration commands, one per line.
AGG-1(config-vrrp)#virtual-router 3 address 10.0.30.1 AGG-1(config-vrrp)# AGG-2#configure terminal Enter configuration commands, one per line. End with Ctrl/Z. AGG-2(config)#router vrrp AGG-2(config-vrrp)#virtual-router 3 address 10.0.30.1 AGG-2(config-vrrp)# c. Select a switch IP interface (Example 6-32). Run virtual-router <1-15> interface to select a switch IP interface.
e. Configure preemption (Example 6-34). Run [no] virtual-router <1-15> preemption to enable or disable master preemption. When enabled, if this virtual router is in backup mode but has a higher priority than the current Master, this virtual router pre-empts the lower priority Master and assumes control. Even when preemption is disabled, this virtual router always preempts any other Master if this switch is the owner (the IP interface address and virtual router addr are the same).
AGG-2(config-vrrp)#virtual-router 3 timers advertise 2 AGG-2(config-vrrp)#virtual-router 3 timers preempt-delay-time 5 AGG-2(config-vrrp)# g. Enable the configured virtual router. Run [no] virtual-router <1-15> enable to enable or disable this virtual router. Run no virtual-router <1-15> to delete this virtual router from the switch configuration. Note: We enabled the AGG-2 (Backup) router first to show the preemption operation. In Example 6-36, note the log messages that show the role status change.
Some tracking criteria apply to standard virtual routers called virtual interface routers. A virtual server router is defined as any virtual router whose IP address is the same as any configured virtual server IP address. Run [no] virtual-router <1-15> track virtual-routers to allow the priority for this virtual router to be increased for each virtual router in Master mode on this switch.
4. Verify the VRRP operation Run the commands listed in this step to verify the VRRP operation on the switch. Run show ip vrrp to display the current VRRP parameters (Example 6-38).
Current VRRP virtual router 3: vrid 30, 10.0.30.1, if 30, prio 100, adver 2, enabled preem enabled, predelay 5, fast-advertisement disabled track ports AGG-2# Run show ip vrrp counters to display VRRP statistics (Example 6-40).
Figure 6-3 shows the complete architecture with the IP addressing details. SRV-1 is connected to ACC-1 and ACC-2 and is able to ping the SRV-3 connected to the ACC-3 stack. BladeCenter Chassis SRV-3 10.0.30.30 FC30::30 VFSM Stack VLAN30 10.0.30.0/24 FC30::0/64 AGG-1 .3 / .1 ::3 / ::1 OSPF Area 0 AGG-2 VLAN100 10.0.100.0/30 FC00::0/64 .1 .2 portchannel3 .1 .2 .1 portchannel1 ACC-1 el2 nn ha .1 el2 nn ha c t r po VLAN102 10.0.102.0/30 FC12::0/64 rt c po portchannel1 VLAN 101 10.0.101.
Windows host verification Figure 6-4 shows that the Windows host is able to ping the Linux host using both IPv4 and IPv6. Figure 6-4 Windows host to Linux host verification Chapter 6.
Linux host verification Figure 6-5 shows the output from an ICMP test from a Linux host to a Windows host. Figure 6-5 Linux host to Windows host verification 6.7 More information For detailed information about Layer 2 configuration, see the following documents: Configuration guides: IBM Virtual Fabric 10Gb Switch Module for IBM BladeCenter Application Guide (6.8): http://www-01.ibm.com/support/docview.
7 Chapter 7. Maintenance and troubleshooting In this chapter, we describe some elements that can help you with the maintenance and troubleshooting of IBM System Networking 10Gb switches. © Copyright IBM Corp. 2012. All rights reserved.
7.1 Configuration management This section describes how to manage configuration files, save and restore a configuration in the switch, perform a firmware upgrade, and identify some problems by checking the system logs and descriptions. 7.1.1 Configuration files The switch stores its configuration in two files: startup-config is the configuration the switch uses when it is reloaded. running-config is the configuration that reflects all the changes you made from the CLI.
Managing the configuration through SNMP This section describes how to use MIB calls to work with switch images and configuration files. You can use a standard SNMP tool to perform the actions, using the MIBs listed in Table 7-1. For information about how to set up your switch to use SNMP, see 7.3.2, “SNMP” on page 297. Table 7-1 SNMP MIBs for managing switch configuration and firmware MIB name MIB OID agTransferServer 1.3.6.1.4.1872.2.5.1.1.7.1.0 agTransferImage 1.3.6.1.4.1872.2.5.1.1.7.2.
Loading a saved configuration To load a saved switch configuration with the name MyRunningConfig.cfg into the switch, complete the following steps. This example shows a TFTP server at IPv4 address 172.25.101.200, although IPv6 is also supported, where the configuration previously saved is available to download. 1. Set the FTP/TFTP server address where the switch Configuration File is located: Set agTransferServer.0 "172.25.101.200" 2. Set the name of the configuration file: Set agTransferCfgFileName.
4. If you are using an FTP server, enter a password: Set agTransferPassword.0 "MyPassword" 5. Initiate the transfer. To save a dump file, enter 5. Set agTransferAction.0 "5" 7.1.4 Factory defaults To reset the switch to the factory defaults, you need to perform one of the following procedures. Resetting with access to the terminal If you have access to the switch’s terminal, and you would like to reset the switch to the factory defaults, complete the following steps: 1.
5. You see the initial menu once again. Enter 4 to exit and reset the switch with the default configuration: Boot Management Menu 1 - Change booting image 2 - Change configuration block 3 - Xmodem download 4 - Exit Please choose your menu option:4 The switch resets to the factory default configuration. 7.1.5 Password recovery To perform password recovery, you need to set the switch to the factory default by using one of the procedures described in 7.1.4, “Factory defaults” on page 289.
Current FLASH software: image1: version 6.3.2, downloaded 7:36:34 Tue Jan 3, 2000 image2: version 6.8.0.3, downloaded 11:38:34 Fri Jan 20, 2000 boot kernel: version 6.8.0.3 Currently scheduled reboot time: none In Example 7-1 on page 290, you can see that the system has two OS images: image1: Version 6.3.2 image2: Version 6.8.0.3 The boot image version is 6.8.0.3. We want to make sure that the switch uses the same version for boot image and OS image.
Important: When connecting to the switch with the serial port, you should use the following parameters, which do not change, except for the speed, that changes when we recover from a faulty upgrade: Speed: 9600 bps Data Bits: 8 Stop Bits: 1 Parity: None Flow Control: None 7.2.3 Loading the new firmware In this section, we show how to load the new firmware on the switch by using both Menu-Based CLI and ISCLI.
7. The system then informs you which software image (image1 or image2) is currently set to be loaded at the next reset, and prompts you to enter a new choice: Currently set to use switch software "image1" on next reset. Specify new image to use on next reset ["image1"/"image2"]: Specify the image that contains the newly loaded software. 8. Reboot the switch to run the new software by running reset: Boot Options# reset The system prompts you to confirm your request.
Important: The procedure described in this section might also be useful when you boot the switch and the boot and OS versions are not equal. Then, power on the switch and you see some boot messages. From your terminal window, press Shift + B while the Memory tests are processing and dots are showing the progress. A menu opens. Select 3 for Xmodem download. Change the serial connection properties as follows: ## Switch baudrate to 115200 bps and press ENTER ...
**** Switch OS **** Please choose the Switch OS Image to upgrade [1|2|n] : You are prompted to the select the image space in the switch you want to upgrade. If you are performing a recovery, select 1. You see a screen similar to the one in Example 7-5. Example 7-5 Upgrading the OS image Switch OS Image 1 ... Un-Protected 27 sectors Erasing Flash.............................. done Writing to Flash..............................
Each syslog message has a criticality level associated with it, included in text form as a prefix to the log message. One of eight different prefixes is used, depending on the condition that the administrator is being notified of: Level 0 - EMERG: Indicates that the system is unusable. Level 1 - ALERT: Indicates that action should be taken immediately. Level 2 - CRIT: Indicates critical conditions. Level 3 - ERR: Indicates error conditions or operations in error.
Logging destinations You can set up to two destinations for reporting. A destination of 0.0.0.0 means logs are stored locally on the switch. Another instance of a log destination host can be a remote logging server. In this case, the logs are sent to the server through Syslog. For each of the two destinations, you can define many parameters, including the severity of logs to be sent to that particular destination.
The SNMP manager should be able to reach the management interface or any of the IP interfaces on the switch.
Enter current admin password: Enter new privacy password: Re-enter new privacy password: New privacy password accepted. 2. Configure a user access group, along with the views the group may access, by running the commands shown in Example 7-10. Use the access table to configure the group’s access level.
4. Specify the IPv4 address and other trap parameters in the targetAddr and targetParam tables. Use the commands shown in Example 7-15 to specify the user name associated with the targetParam table.
RS8264(config)#snmp-server target-parameters 11 name v3param RS8264(config)#snmp-server target-parameters 11 user-name v3trap RS8264(config)#snmp-server target-parameters 11 level authNoPriv 7.3.3 Remote Monitoring (RMON) The IBM switches provide a Remote Monitoring (RMON) interface that allows network devices to exchange network monitoring data. RMON allows the switch to perform the following functions: Track events and trigger alarms when a threshold is reached.
etherStatsBroadcastPkts: 4380 etherStatsMulticastPkts: 6612 etherStatsCRCAlignErrors: 22 etherStatsUndersizePkts: 0 etherStatsOversizePkts: 0 etherStatsFragments: 2 etherStatsJabbers: 0 etherStatsCollisions: 0 etherStatsPkts64Octets: 27445 etherStatsPkts65to127Octets: 12253 etherStatsPkts128to255Octets: 1046 etherStatsPkts256to511Octets: 619 etherStatsPkts512to1023Octets: 7283 etherStatsPkts1024to1518Octets: 38 RMON Group 2: History You can use the RMON History Group to sample and archive Ethernet statisti
IBM System Networking Element Manager IBM System Networking Element Manager (SNEM) is an application for remote monitoring and management of Ethernet switches from IBM. It is designed to simplify and centralize the management of your BladeCenter or blade server and Top-of-Rack Ethernet switches. IBM System Networking Element Manager is a simple yet powerful tool that you can use to easily set up a logging and reporting environment to monitor the devices from a central point.
Health Status Summary pane The Health Status Summary pane shows the individual count of devices discovered that are Down (red), Critical (orange), Non-Critical (yellow), and Up (green). It also provides a pie chart that indicates the percentages of Down/Critical/Non-Critical/Up devices. A sample Health Status Summary pane is shown in Figure 7-2.
Viewing Health Status The Health Status page shows processor and Memory Utilization, ARP and Routing Table Utilization, Power Supply Status, Panic Dump Status, Temperature Sensors reading, and Fan Speed. A sample Health Status window is shown in Figure 7-3. Figure 7-3 Health Status window Viewing reports You can view various reports associated with all the discovered switches by choosing the items under the Reports menu in SNEM.
For more information about SNEM, see the following publications: IBM SNEM 6.1 Solution Getting Started Guide: http://www-01.ibm.com/support/docview.wss?uid=isg3T7000471&aid=1 IBM SNEM 6.1 User Guide: http://www-01.ibm.com/support/docview.wss?uid=isg3T7000473&aid=1 IBM SNEM 6.1 Release Notes Changes: http://www-01.ibm.com/support/docview.wss?uid=isg3T7000474&aid=1 IBM System Networking Element Manager Solution Device Support List (6.1): http://www-01.ibm.com/support/docview.
Solution 1: Check the port configuration in the software (see the Command Reference for your switch). If the port is configured with a specific speed or duplex mode, check the other device to verify that it is set to the same configuration. If the switch port is set to autonegotiate, verify that the other device is set to autonegotiate. Solution 2: Check the cables that connect the port to the other device. Make sure that they are connected. Verify that you are using the correct cable type.
Connecting via DATA port. [host 10.0.100.1, max tries 5, delay 1000 msec , length 0] 10.0.100.1: #1 ok, RTT 0 msec. 10.0.100.1: #2 ok, RTT 0 msec. 10.0.100.1: #3 ok, RTT 1 msec. 10.0.100.1: #4 ok, RTT 0 msec. 10.0.100.1: #5 ok, RTT 0 msec. You can see in the output that all five ICMP Echo requests received the replies. There is also additional information about the Round Trip Time (RTT), that is, the time it took for the ACC-2 to receive the response from AGG-1.
A Appendix A. Configuration files This appendix provides the final working configuration of the equipment used for the reference architecture. The configuration files are from the following equipment: AGG-1: Aggregation switch (RackSwitch G8264) AGG-2: Aggregation switch (RackSwitch G8264) ACC-1: Access switch (RackSwitch G8124) ACC-2: Access switch (RackSwitch G8124) ACC-3: Access switch (Virtual Fabric 10Gb Switch Module stack) © Copyright IBM Corp. 2012. All rights reserved.
AGG-1: Aggregation switch (RackSwitch G8264) Example A-1 shows the final configuration of the AGG-1 aggregation switch. Example A-1 Final configuration of the AGG-1 aggregation switch AGG-1# ! version "6.8.0.
interface port 21 name "AGG1-ACC3" pvid 30 exit ! interface port 22 name "AGG1-AGG4" pvid 30 exit ! vlan 1 member 2-4,6-16,23-64 no member 1,5,17-22 ! vlan 30 enable name "SRV3" member 1,5,21-22 ! vlan 100 enable name "IPv4_AGG1-AGG2" member 1,5 ! vlan 101 enable name "IPv4_AGG1-ACC1" member 17-18 ! vlan 103 enable name "IPv4_AGG1-ACC2" member 19-20 ! portchannel 1 port 17 portchannel 1 port 18 portchannel 1 enable ! portchannel 2 port 19 portchannel 2 port 20 portchannel 2 enable ! portchannel 3 port 1 por
spanning-tree stp 30 bridge priority 0 spanning-tree stp 30 vlan 30 spanning-tree stp 100 bridge priority 0 spanning-tree stp 100 vlan 100 spanning-tree stp 101 bridge priority 0 spanning-tree stp 101 vlan 101 spanning-tree stp 103 bridge priority 0 spanning-tree stp 103 vlan 103 ! logging host 2 address 10.10.53.219 MGT ! lldp enable ! ip router-id 1.1.1.1 ! interface ip 30 ip address 10.0.30.2 255.255.255.
enable ip6host exit ! interface ip 111 ipv6 address fc11:0:0:0:0:0:0:2 64 vlan 101 enable ip6host exit ! interface ip 113 ipv6 address fc13:0:0:0:0:0:0:1 64 vlan 103 enable ip6host exit ! interface ip 128 ip address 172.25.101.120 enable exit ! interface loopback 1 ip address 1.1.1.1 255.255.255.255 enable exit ! ip gateway 4 address 172.25.1.
! redistribute fixed export 2 1 ! message-digest-key 1 md5-ekey ea28046b4028002ab376e7a28398a3d86e6f385d01eb3bc9926a86856348faa4bcfb83951d8c8d2fee 026eccb994eb6189d271a8be987bb684edb91152d0b937 ! interface ip 100 ip ospf enable ip ospf message-digest-key 1 ! interface ip 101 ip ospf enable ip ospf message-digest-key 1 ! interface ip 103 ip ospf enable ip ospf message-digest-key 1 ! ipv6 router ospf router-id 1.1.1.1 enable ! area 0 area-id 0.0.0.
ipv6 ipv6 ipv6 ipv6 ipv6 ipv6 ipv6 ospf ospf ospf ospf ospf ospf ospf retransmit-interval 5 transmit-delay 1 priority 1 hello-interval 10 dead-interval 40 cost 1 enable ! end AGG-2: Aggregation switch (RackSwitch G8264) Example A-2 shows the final configuration of the AGG-2 aggregation switch. Example A-2 Final configuration of the AGG-2 aggregation switch version "6.8.0.
name "AGG2-ACC1" pvid 102 exit ! interface port 20 name "AGG2-ACC1" pvid 102 exit ! interface port 21 name "AGG2-ACC3" pvid 30 exit ! interface port 22 name "AGG2-AGG4" pvid 30 exit ! vlan 1 member 2-4,6-16,23-64 no member 1,5,17-22 ! vlan 30 enable name "IPv4_SRV3" member 1,5,21-22 ! vlan 100 enable name "AGG1-AGG2" member 1,5 ! vlan 102 enable name "AGG2-ACC1" member 19-20 ! vlan 104 enable name "AGG2-ACC2" member 17-18 ! portchannel 1 port 17 portchannel 1 port 18 portchannel 1 enable ! portchannel 2 por
portchannel 4 port 21 portchannel 4 port 22 portchannel 4 enable ! portchannel thash l2thash l2-source-mac-address ! portchannel thash ingress ! ! spanning-tree stp 30 bridge priority 4096 spanning-tree stp 30 vlan 30 spanning-tree stp 100 vlan 100 spanning-tree stp 102 vlan 102 spanning-tree stp 104 vlan 104 ! ! lldp enable ! ip router-id 1.1.1.2 ! interface ip 30 ip address 10.0.30.3 255.255.255.
interface ip 110 ipv6 address fc00:0:0:0:0:0:0:2 64 vlan 100 enable ip6host exit ! interface ip 112 ipv6 address fc12:0:0:0:0:0:0:2 64 vlan 102 enable ip6host exit ! interface ip 114 ipv6 address fc14:0:0:0:0:0:0:1 64 vlan 104 enable ip6host exit ! interface ip 128 ip address 172.25.101.121 255.255.255.0 enable exit ! interface loopback 1 ip address 1.1.1.2 255.255.255.255 enable exit ! ip gateway 4 address 172.25.101.
area 0 authentication-type md5 area 0 enable ! redistribute fixed export 5 1 ! message-digest-key 1 md5-ekey 2d6914f4002000a024a0f7b7c390a3528f55837641b160226c2c717671830ebc1d2e3de38fc850c1e0 280022ea6d7f87350fe239c7568803d9090116b2482cc0 ! interface ip 100 ip ospf enable ip ospf message-digest-key 1 ! interface ip 102 ip ospf enable ip ospf message-digest-key 1 ! interface ip 104 ip ospf enable ip ospf message-digest-key 1 ! ipv6 router ospf router-id 1.1.1.2 enable ! area 0 area-id 0.0.0.
interface ip ipv6 ipv6 ipv6 ipv6 ipv6 ipv6 ipv6 ipv6 ! end 114 ospf ospf ospf ospf ospf ospf ospf ospf area 0 retransmit-interval 5 transmit-delay 1 priority 1 hello-interval 10 dead-interval 40 cost 1 enable ACC-1: Access switch (RackSwitch G8124) Example A-3 shows the final configuration of the ACC-1 access switch. Example A-3 Final configuration of the ACC-1 access switch version "6.8.0.
name "ACC1-ACC2" tagging pvid 10 exit ! interface port 6 name "ACC1-ACC2" tagging pvid 10 exit ! interface port 7 name "SRV1" pvid 10 exit ! vlan 1 member 8-24 no member 1-7 ! vlan 10 enable name "IPv4_SRV1" member 5-7 ! vlan 101 enable name "IPv4_ACC1-AGG1" member 1-2 ! vlan 102 enable name "IPv4_ACC1-AGG2" member 3-4 ! portchannel 1 port 1 portchannel 1 port 2 portchannel 1 enable ! portchannel 2 port 3 portchannel 2 port 4 portchannel 2 enable ! portchannel hash source-mac-address ! ! spanning-tree stp 1
lacp mode active lacp priority 16384 lacp key 3 ! interface port 6 lacp mode active lacp key 3 ! failover enable failover trigger 1 limit 2 failover trigger 1 mmon monitor PortChannel 1 failover trigger 1 mmon monitor PortChannel 2 failover trigger 1 mmon control member 7 failover trigger 1 enable ! ! ! ! ! ! ! ! ! lldp enable ! ip router-id 2.2.2.1 ! interface ip 10 ip address 10.0.10.2 255.255.255.0 vlan 10 enable exit ! interface ip 101 ip address 10.0.101.1 255.255.255.
vlan 101 enable ip6host exit ! interface ip 112 ipv6 address fc12:0:0:0:0:0:0:1 64 vlan 102 enable ip6host exit ! interface ip 127 ip address 172.25.101.122 enable exit ! interface loopback 1 ip address 2.2.2.1 255.255.255.255 enable exit ! ip gateway 3 address 172.25.1.
message-digest-key 1 md5-ekey 28e2cc3e0862882af1a3a7f7cbd22bd8f4046e5ad324465d9b1565b6fce72e2e7feafe9495d94195ae cb3a9ef27493713d8790e864829b90bc64ce5ffaaef852 ! interface ip 101 ip ospf enable ip ospf message-digest-key 1 ! interface ip 102 ip ospf enable ip ospf message-digest-key 1 ! ipv6 router ospf router-id 2.2.2.1 enable ! area 0 area-id 0.0.0.
ACC-2: Access switch (RackSwitch G8124) Example A-4 shows the final configuration of the ACC-2 access switch. Example A-4 Final configuration of the ACC-2 access switch version "6.8.0.
pvid 10 exit ! vlan 1 member 8-24 no member 1-7 ! vlan 10 enable name "IPv4_SRV1" member 5-7 ! vlan 103 enable name "IPv4_ACC2-AGG1" member 3-4 ! vlan 104 enable name "IPv4_ACC2-AGG2" member 1-2 ! portchannel 1 port 1 portchannel 1 port 2 portchannel 1 enable ! portchannel 2 port 3 portchannel 2 port 4 portchannel 2 enable ! portchannel hash source-mac-address ! ! spanning-tree stp 10 bridge priority 4096 spanning-tree stp 10 vlan 10 spanning-tree stp 103 vlan 103 spanning-tree stp 104 vlan 104 ! interface
failover trigger 1 enable ! ! ! ! ! ! ! ! ! lldp enable ! ip router-id 2.2.2.2 ! interface ip 10 ip address 10.0.10.3 255.255.255.0 vlan 10 enable exit ! interface ip 103 ip address 10.0.103.2 255.255.255.252 vlan 103 enable exit ! interface ip 104 ip address 10.0.104.2 255.255.255.
enable exit ! interface loopback 1 ip address 2.2.2.2 255.255.255.255 enable exit ! ip gateway 3 address 172.25.1.1 ip gateway 3 enable ! ! router vrrp enable ! tracking-priority-increment ports 50 ! virtual-router 1 virtual-router-id 10 virtual-router 1 interface 10 virtual-router 1 address 10.0.10.
area 0 enable ! redistribute connected export 10 1 ! interface ip 113 ipv6 ospf area 0 ipv6 ospf retransmit-interval 5 ipv6 ospf transmit-delay 1 ipv6 ospf priority 1 ipv6 ospf hello-interval 10 ipv6 ospf dead-interval 40 ipv6 ospf cost 1 ipv6 ospf enable ! interface ip 114 ipv6 ospf area 0 ipv6 ospf retransmit-interval 5 ipv6 ospf transmit-delay 1 ipv6 ospf priority 1 ipv6 ospf hello-interval 10 ipv6 ospf dead-interval 40 ipv6 ospf cost 1 ipv6 ospf enable ! end ACC-3: Access switch (Virtual Fabric 10G Swi
interface port 1:17 name "AGG1-ACC3" pvid 30 exit ! interface port 1:18 name "AGG2-ACC3" pvid 30 exit ! interface port 2:14 name "SRV-3" pvid 30 exit ! interface port 2:17 name "AGG1-ACC3" pvid 30 exit ! interface port 2:18 name "AGG2-ACC3" pvid 30 exit ! vlan 1 member 1:1-1:13,1:19-1:27 no member 1:14,1:17-1:18 member 2:1-2:13,2:19-2:27 no member 2:14,2:17-2:18 member 3:1-3:14,3:17-3:27 member 4:1-4:14,4:17-4:27 member 5:1-5:14,5:17-5:27 member 6:1-6:14,6:17-6:27 member 7:1-7:14,7:17-7:27 member 8:1-8:14,8
portchannel 1 portchannel 1 portchannel 1 ! portchannel 2 portchannel 2 portchannel 2 ! ! spanning-tree port 1:17 port 2:17 enable port 1:18 port 2:18 enable stp 30 vlan 30 ! snmp-server name "ACC-3" ! ! ! ! ! ! ! ! lldp enable ! ! ! ! end Appendix A.
332 Implementing IBM System Networking 10Gb Ethernet Switches
Related publications The publications listed in this section are considered particularly suitable for a more detailed discussion of the topics covered in this book. Locating the web material The web material associated with this book is available in softcopy on the Internet from the IBM Redbooks web server. Point your web browser at: ftp://www.redbooks.ibm.com/redbooks/SG247960 Alternatively, you can go to the IBM Redbooks website at: ibm.
IBM 1/10Gb Uplink Ethernet Switch Module for IBM BladeCenter Command Reference: http://www-947.ibm.com/systems/support/supportsite.wss/docdisplay?brandind=5000 008&lndocid=MIGR-5076525 IBM 1/10Gb Uplink Ethernet Switch Module for IBM BladeCenter Installation Guide: ftp://ftp.software.ibm.com/systems/support/system_x_pdf/dw1gymst.pdf IBM 1/10Gb Uplink Ethernet Switch Module for IBM BladeCenter ISCLI Reference: http://www-947.ibm.com/systems/support/supportsite.
IBM RackSwitch G8264 Blade OS Application Guide: http://www-01.ibm.com/support/docview.wss?uid=isg3T7000326 IBM RackSwitch G8264 Browser-Based Interface Quick Guide: http://www-01.ibm.com/support/docview.wss?uid=isg3T7000342 IBM RackSwitch G8264 Installation Guide: http://www.bladenetwork.net/userfiles/file/G8264_install.pdf IBM RackSwitch G8264 ISCLI Command Reference: http://www-01.ibm.com/support/docview.wss?uid=isg3T7000329 IBM RackSwitch G8264 Menu-Based CLI Reference Guide: http://www-01.
Online resources These websites are also relevant as further information sources: IBM 1/10Gb Uplink Ethernet Switch Module Announcement Letter: http://www.ibm.com/common/ssi/rep_ca/5/872/ENUSAG08-0365/ENUSAG080365.PDF IBM BladeCenter H Announcement Letter: http://www-01.ibm.com/common/ssi/cgi-bin/ssialias?infotype=an&subtype=ca&appnam e=gpateam&supplier=897&letternum=ENUS109-438 IBM BladeCenter HT Announcement Letter: http://www-01.ibm.
Index Numerics 10 Gb uplink ports 7 10Gb SFP+ 48 10GbE SFP+ ports 8 1Gb SFP+ 48 802.1p configuration 183 802.1p priority value 271 802.1Q VLAN tagging 167, 262 802.1Q VLAN tags 53 802.
CHAP 94 chassis UUID 253 CIO-v 46 Cisco STP packets 184 cKVM 39 class of service 138 Class of Service (COS) 105 Class Selector (CS) 104 CLI 129, 286 CLI commands 91 Client 96 Combination I/O Vertical (CIO-v) 41 Combo Form Factor Horizontal (CFF-h) 42 Community 90 Community VLAN 89 community VLAN 264 compact flash (CF) 44 concurrent KVM (cKVM) 39 Configuration blocks 286 Configuration files 286 Configure authentication 206 Configure hashing 172, 266 Configured Switch Number (csnum) 244 Configuring date and t
Failover Methods 84 failover mode 193 Failover Trigger 232 failover trigger 233 Failover Trigger limit 232 failover trigger status 234–235 Fast Uplink Convergence 80 FastLeave 68 FCoE 22, 28, 31 FCoE Initialization Protocol (FIP) 31 FDB 80 FDB Flush 88 FDB Update 80 Fibre Channel over Ethernet (FCoE) 27, 50, 241 Fibre Channel over Ethernet (FCoE) 17 FIP Snooping 17, 24 firmware 292–293 Firmware files 290 Firmware Management 290 Firmware upgrade 149 Flow control 127 flow control 18, 131, 157, 161, 171, 257 F
Intel Connects Optical Cable 48 interface status 234, 236 internal BGP (iBGP) 66, 219 Internal Router (IR) 64 internal routing 65 Internet Group Management Protocol (IGMP) 67, 193 Internet Protocol version 6 (IPv6) 71 inter-switch connection 111 IP address 130, 144 IP Addressing 113 IP Configuration 134 IP filtering 35 IP forwarding 30, 35, 136 IP interface 186–187 IP multicast 67 IP Routing 60, 136 IP routing 155 IP subnets 60, 136 IPv4 108, 121, 134, 186, 192 IPv4 packets 184 IPv4 static routes 189 IPv6 1
Multiple Spanning Tree 24 Multiple Spanning Tree Protocol (MSTP) 59 Multiple STP (MSTP) 30, 34, 36 Myrinet 45 N negotiation mode 131 Neighbor Advertisements 73 Neighbor Discovery 73 Neighbor Discovery protocol (ND) 73 Neighbor Solicitations 73 neighbors 64 Netboot 18 Network Access Server (NAS) 94 Network Adapter Teaming 273 Network Equipment Provider (NEP) 43 Network Time Protocol (NTP) 35 network topology 107 NIC Teaming 80 NIC teaming 5, 193 No Backup 244 Non-blocking architecture 34 Not-So-Stubby-Area
Quality of service 10, 17 Quality of Service (QoS) 35, 100, 271 Querier 67 query-response-interval 68 R RackSwitch G8052 6–7 RackSwitch G8124 12, 155 RackSwitch G8264 20 RADIUS 9, 16, 24, 91–92, 139, 193 Radius 30, 35 Radius authentication 139 RADIUS server 139 Rapid Spanning Tree 5, 24 Rapid Spanning Tree Protocol 50 Rapid Spanning-Tree Protocol (RSTP) 59 Rapid STP (RSTP) 30, 34, 36 RAS 92 Redbooks website 333 Contact us xiv Redirect messages 73 Redundant midplane 44 Reference architectures 108 Re-Maring
Spanning-Tree mode 120 Spanning-Tree Protocol (STP) 58 Speed 131 speed 161, 171, 257 sport 191 SSH 17, 24, 91, 129 SSM 68 stack 88 stack links 242 Stack Member Identification 244 stack members 249 Stack Membership 242 stack status 248 Stacking 3, 88, 272 stacking 240 stacking configuration 247 stacking links 246 stacking membership mode 246 stacking VLAN 246 Stateful address configuration 73 Stateless address configuration 73 stateless auto-configuration 194 Static MAC address adding 242 Static multicast 24
untagged port 169 Uplink Failure Detection 81 Uplink failure detection 10, 17 Uplink Failure Detection (UFD) 7, 14 USB port 8 User 138 User Account 138 user accounts 137 User Datagram Protocol (UDP) 62 V VASA 179, 269 VFSM 96, 239, 274 VID 53 Virtual Fabric 14, 21 Virtual Fabric 10Gb Switch Module 239 Virtual Interface Router 83 virtual link 170 Virtual Link Aggregation Groups (VLAGs) 79 Virtual Local Area Networks (VLANs) 52, 60 Virtual local area networks (VLANs) 111 Virtual NICs 242 virtual NICs 13 virt
Implementing IBM System Networking 10Gb Ethernet Switches Implementing IBM System Networking 10Gb Ethernet Switches Implementing IBM System Networking 10Gb Ethernet Switches Implementing IBM System Networking 10Gb Ethernet Switches (0.5” spine) 0.475”<->0.
Implementing IBM System Networking 10Gb Ethernet Switches Implementing IBM System Networking 10Gb Ethernet Switches
Back cover ® Implementing IBM System Networking 10Gb Ethernet Switches ® Introduction to IBM System Networking RackSwitch hardware Sample network design and implementation Switch troubleshooting and maintenance In today’s infrastructure, it is common to build networks based on 10 Gb Ethernet technology. The IBM portfolio of 10 Gb systems networking products includes Top-of-Rack switches, and the embedded switches in the IBM BladeCenter family.
