IBM Front cover Cisco Systems Intelligent Gigabit Ethernet Switch Module for IBM Eserver BladeCenter Copper Ethernet switching technology integrated into the BladeCenter chassis Helpful configurations and troubleshooting techniques Configuration examples using CMS and CLI Rufus Credle Yoko Fujiwara Matt Slavin Kenichi Tanaka Mark Welch ibm.
International Technical Support Organization Cisco Systems Intelligent Gigabit Ethernet Switch Module for IBM Eserver BladeCenter April 2005
Note: Before using this information and the product it supports, read the information in “Notices” on page vii. First Edition (April 2005) This edition applies to Cisco Systems Intelligent Gigabit Ethernet Switch Module for the IBM Eserver BladeCenter. © Copyright International Business Machines Corporation 2004, 2005. All rights reserved. Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.
Contents Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii Trademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix The team that wrote this Redpaper . . . . . . . . . . . . . . . . . . . . . . . .
5.2.6 CiscoView . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2.7 IBM Director and Remote Deployment Manager . . . . . . . . . . . . . . . . . . . . . . . . . 5.3 In-depth management path discussions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.1 Introduction to this in-depth management discussion . . . . . . . . . . . . . . . . . . . . . . 5.3.2 Why was this in-depth section created? . . . . . . . . . . . . . .
8.1.1 Basic rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.1.2 Basic symptoms and possible solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2 Introduction to troubleshooting the IGESM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2.1 General comments on troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2.2 Information useful to technical support . . . . . . . . .
vi Cisco Systems Intelligent Gigabit Ethernet Switch Module
Notices This information was developed for products and services offered in the U.S.A. IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used.
Trademarks The following terms are trademarks of the International Business Machines Corporation in the United States, other countries, or both: AIX® BladeCenter™ Domino® Electronic Service Agent™ Enterprise Storage Server® Eserver® Eserver® eServer™ HelpCenter® HelpWare® ibm.
Preface This IBM® Redpaper positions the Cisco Systems Intelligent Gigabit Ethernet Switch Module for the IBM Eserver® BladeCenter™ and describes how it enhances the BladeCenter value proposition by seamlessly interfacing into a customer’s existing data network. This paper helps you plan, install, and configure the Cisco Systems Intelligent Gigabit Ethernet Switch Module for several network topologies.
Kenichi Tanaka is an I/T Specialist in Network Systems for IBM Japan Systems Engineering in IBM Makuhari, Japan. He has three years of experience in networking. He provides technical support for network products, design, and implementation. His areas of expertise include Cisco networking products and F5 Networks load balancer. His industry certification includes Cisco Certified Network Professional. He holds a degree in electronic and information engineering from Tokyo Metropolitan University.
Edward Suffern, BladeCenter Ethernet Switching IBM RTP Pritesh Patel, Manager, Software Development, EAG Desktop Switch Business Unit Cisco Systems San Jose Amit Sanyal, Product Marketing Manager, EAG Desktop Switch Business Unit Cisco Systems San Jose Damon West, IBM PC Institute IBM RTP Chris Durham, BladeCenter Development, IBM Systems Group IBM RTP Khalid Ansari, Storage Networking Support, LAN/ATM Switch Support, BISC Team IBM RTP Robert Jakes, BISC Team - (Blade Infrastructure Solutions Center) IBM RT
Comments welcome Your comments are important to us! We want our papers to be as helpful as possible. Send us your comments about this Redpaper or other Redbooks in one of the following ways: Use the online Contact us review redbook form found at: ibm.com/redbooks Send your comments in an e-mail to: redbook@us.ibm.com Mail your comments to: IBM Corporation, International Technical Support Organization Dept. HQ7 Building 662 P.O.
1 Chapter 1. Executive summary IBM and Cisco have committed to a strategic alliance to address customer requirements regarding the integration of server and networking technology. The Cisco Systems Intelligent Gigabit Ethernet Switch Module for IBM Eserver BladeCenter (Cisco Systems IGESM) represents an important initial step in this alliance. This BladeCenter switch module offers BladeCenter customers Cisco’s world-class copper Ethernet switching technology integrated into the BladeCenter chassis.
Each Cisco Systems IGESM provides one Gigabit/sec Ethernet (GbE) connectivity to each of the 14 blade slots and four GbE uplink interfaces external to the BladeCenter. The customer can install as few as one Cisco Systems IGESM or as many as four Cisco Systems IGESMs in one BladeCenter. With four Cisco Systems IGESMs installed, the customer can obtain 16 GbE uplink interfaces, as well as 56 GbE internal switching capability.
2 Chapter 2. IBM eServer BladeCenter overview The IBM Eserver BladeCenter innovative modular technology, leadership density, and availability was designed to help solve a multitude of real-world problems. For organizations seeking server consolidation, the BladeCenter centralizes servers for increased flexibility, ease of maintenance, reduced cost, and streamlined human resources.
Figure 2-1 shows the BladeCenter chassis, HS40, HS20, and JS20: IBM eServer BladeCenter chassis The BladeCenter is a high-density blade solution that provides maximum performance, availability, and manageability for application serving, storage flexibility, and long-life investment protection. HS40 HS40 is a four-way blade server for high-performance enterprise applications requiring four-processor SMP capability.
IBM eServer BladeCenter storage solutions IBM delivers a wide range of easy-to-install, high-capacity, tested storage products for the BladeCenter to meet your demanding business needs.
2.1 IBM eServer BladeCenter architecture In this section, we look into the architectural design of the BladeCenter chassis and components. 2.1.1 The midplane In Figure 2-2, we discuss the BladeCenter midplane. The midplane has two similar sections (upper and lower) that provide redundant functionality. The processor blades (blade servers) plug into the front of the midplane. All other major components plug into the rear of the midplane.
Management Module 1 Ethernet 3 → Management Module 1 Ethernet 4 → Management Module 2 Ethernet 1 → Management Module 2 Ethernet 2 → Management Module 2 Ethernet 3 → Management Module 2 Ethernet 4 → Expansion Switch Module 3 Ethernet 15 Expansion Switch Module 4 Ethernet 15 Switch Module 1 Ethernet 16 Switch Module 2 Ethernet 16 Expansion Switch Module 3 Ethernet 16 Expansion Switch Module 4 Ethernet 16 IBM ^ BladeCenter™ Management Module Ethernet Interface Switch Module Switch Module Power
only to be used when a daughter card is installed. Unless a daughter card is installed in one or more processor blades, there is no need for Switch Modules 3 and 4. Further, the switch modules have to be compatible with the LAN interface generated by the processor blade.
The CSB5 provides the interface to: One PCI Bus used to connect to the ATI Rage XL video controller with 8 MB of memory Two Low Pin Count (LPC) Buses used to connect to the 4 MB EEPROM (holding the POST/BIOS code) and to the SIO (SuperI/O) chip Two IDE channels supporting the internal storage Four USB Buses for redundant connections to FDD/CDROM and keyboard/video This system uses a H8S2148 IBM Integrated System Management Processor that is wired to the I2C buses.
10/100 Mb internal link that connects the Management Module and the Ethernet Switch Modules through the BladeCenter backplane are exploited (notice that the internal network interface of the Management Module has a default static IP address of 192.168.70.126). These more complete tools can also be accessed by pointing your Web browser or a Telnet client to the IP of the Ethernet Switch Module itself (default for a module plugged in Rear Bay 1 is 192.168.70.127, but DHCP-based addressing can be configured).
3 Chapter 3. Cisco Systems Intelligent Gigabit Ethernet Switch Module In this chapter, we discuss the features included in the Cisco Systems Intelligent Gigabit Ethernet Switch Module that offers BladeCenter customers Cisco’s world-class Ethernet switching technology integrated within the IBM Eserver BladeCenter. © Copyright IBM Corp. 2004, 2005. All rights reserved.
3.1 Product description The Cisco Systems Intelligent Gigabit Ethernet Switch Module (Figure 3-1) provides layer 2 switching functions for the BladeCenter server chassis. It provides up to 250 virtual LANs for assigning different users to VLANs associated with network resources, traffic patterns, and bandwidth. It also supports trunking protocols (IEEE 802.1Q) and Link Aggregation for the automatic creation of EtherChannel links.
Integrates industry-leading Cisco networking capabilities to reduce data center complexity and increases networking manageability. Leverages the leadership capabilities our BladeCenter Alliance Partners to provide customers the most technological choices.
3.3.2 Port features These items are the ports of the Cisco Systems Intelligent Gigabit Ethernet Switch Module: Four external 1000BASE-T connectors for making 10/100/1000 Mbps connections to a backbone, end stations, and servers Fourteen internal full-duplex Gigabit ports, one connected to each of the blade servers in the BladeCenter unit Two internal full-duplex 100 Mbps ports connected to the Management Modules 3.3.
3.3.5 VLAN support The switch supports 250 port-based VLANs for assigning users to VLANs associated with the applicable network resources, traffic patterns, and bandwidth. VLAN support highlights: The switch supports up to 4094 VLAN IDs to allow service provider networks to support the number of VLANs allowed by the IEEE 802.1Q standard. IEEE 802.
IEEE 802.1X port-based authentication with guest VLAN to provide limited services to non-802.1X-compliant users Standard and extended IP access control lists (ACLs) for defining security policies 3.3.7 Quality of Service (QoS) and Class of Service (CoS) This list represent the Quality of Service (QoS) and Class of Service (CoS) of the Cisco Systems Intelligent Gigabit Ethernet Switch Module: Classification – IEEE 802.
3.3.
18 Cisco Systems Intelligent Gigabit Ethernet Switch Module
4 Chapter 4. Cisco Systems Intelligent Gigabit Ethernet Switch Module architecture In this section, we look at a system overview of the Cisco Systems Intelligent Gigabit Ethernet Switch Module (Cisco Systems IGESM) for the IBM Eserver BladeCenter. First, we focus on the Cisco Systems IGESM itself. The switch is a layer 2 switch with visibility into layers 2 through 4. Figure 4-1 shows the architecture overview of the Cisco Systems Intelligent Gigabit Ethernet Switch Module.
14 ports 1000 Mbps Internal links to BladeServers Blade1 Blade2 Blade3 Blade4 Blade5 Blade6 Blade7 Blade8 Blade9 Blade10 Blade11 Blade12 Blade13 Blade14 G0/1 G0/2 G0/3 G0/4 G0/5 G0/6 G0/7 G0/8 G0/9 G0/10 G0/11 G0/12 G0/13 G0/14 2 ports -100 Mbps MM1 Internal links to the MM2 Management Modules G0/15 G0/16 G0/17 G0/18 G0/19 G0/20 Serial CIGESM Cisco Systems ModuleModule IGESM 17 18 19 20 4 ports 10/100/1000 Mbps RJ45 links for external network connections 1 port RJ45 Service port Serial console con
Internal layer 2 traffic flow in the Cisco Systems IGESM Figure 4-3 shows the internal layer 2 traffic flow in the Cisco Systems IGESM. The hard coded filter in the Cisco Systems IGESM blocks all traffic between the external ports and the Management Module ports. Two Cisco Systems IGESMs in the same BladeCenter chassis exchange layer 2 frames across the Management Module. However, the Cisco Systems IGESM blocks BPDUs which are switched by the Management Module.
4.1 Cisco Systems Intelligent Gigabit Ethernet Switch Module block diagram Figure 4-4 shows the block diagram of the Cisco Systems Intelligent Gigabit Ethernet Switch Module. The Cisco Systems IGESM has two ASICs for switching. It has 1 MB on chip cache for packet buffers and supports 12 Gigabit Ethernet ports. The two ASICs are interconnected with 10 Gigabit link, which is shown as the 10 Gigabit Ethernet connection in Figure 4-4.
5 Chapter 5. Cisco Systems IGESM management and user orientation In this chapter, we discuss tools and applications that help with management and deployment of the Cisco Systems Intelligent Gigabit Ethernet Switch Module (IGESM) in an IBM Eserver BladeCenter. We also discuss the management paths and rules for connecting to and accessing the IGESM. As noted elsewhere in this document, the information herein applies to the 4-port copper-based IGESM running a 12.1(14) version of IOS.
5.1 Cisco Systems IGESM user interface This section discusses the management interface of the switch module and what each task represents. To configure and manage the switch module we can use the following interfaces: Command-line interface (CLI) You can configure and monitor the switch and switch cluster members from the CLI, which is accessible through Telnet or SSH from a remote management station.
Table 5-1 summarizes each interface’s characteristics. Table 5-1 Management interfaces Command-line interface (CLI) Cluster Management Suite (CMS) Interface type Text-based Graphical Advantage Detailed and controlled Intuitive and easier to start Interface for access Telnet, SSH Web browser with Java plug-in Port used Telnet: 23 SSH: 22 HTTP 80 (default) Can be modified to 0 to 65535, but well-known ports (1-1023) should be excluded.
CLI command modes The Cisco IOS user interface has many different modes. Which commands are available depend on which mode you are currently in. Table 5-2 describes these modes: Main command modes Functions in this mode Display prompt according to mode How to access How to exit the mode The examples in the table use the host name Switch. Table 5-2 CLI modes Mode Functions Prompt How to start How to exit User EXEC Limited privilege Switch> Default of user with privilege level 14 or lower.
Purpose Keystrokes Move one word forward ESC+F Delete one word Ctrl+W Move to the beginning of the line Ctrl+A Move to the end of the line Ctrl+E Delete from cursor to the beginning Ctrl+U Delete from cursor to the end Ctrl+K Getting help Use the commands shown in Table 5-4 to display a list of commands that are available for each command mode or a list of associated keywords and arguments for any command.
Useful commands In this section, we demonstrate some typical commands that are used in configuration and troubleshooting. All commands listed here can be run in privileged EXEC mode. Verifying current configuration and system status The commands shown in Table 5-5 are helpful for verifying your current settings and status. Table 5-5 Checking current configuration Command Purpose show version Check the software version, system uptime, and so forth.
The main pane displays the switch IP address, MAC address, and other information supporting switch management, such as host name, serial number, IOS version, and uptime. From the left menu, you can also launch the Cluster Management Suite, run diagnostics and monitoring tools, and access the help resources. Figure 5-2 Cisco Systems Intelligent Gigabit Ethernet Switch Module Home (12.1(14) IOS) 5.1.
Note: The Java 1.4 Plug-in is required for this session. You will be given the option to download and install the plug-in if necessary. The system we used to access CMS was running the Java 1.4.2_03 Plug-in and a dialog box indicated that it was unsupported. We clicked Continue and were allowed to access the interface.
Administration (Figure 5-4 on page 30) – IP Addresses – SNMP – System Time – HTTP Port – Users and Passwords – Console Baud Rate – MAC Addresses – ARP – Save Configuration – Restore Configuration – Software Upgrade – System Reload – Event Notification Figure 5-5 Administration menu Cluster (Figure 5-6) – Create Cluster Figure 5-6 Cluster menu Chapter 5.
Device (Figure 5-7) – Host Name – STP – IGMP Snooping – ACL (guide mode available in read-write mode) – Security Wizard – QoS – AVVID Wizards Figure 5-7 Device menu Port (Figure 5-8) – Port Settings – Port Search – Port Security – EtherChannels – SPAN – Protected Port – Flooding Control Figure 5-8 Port menu 32 Cisco Systems Intelligent Gigabit Ethernet Switch Module
VLAN (Figure 5-9) – VLAN (guide mode available in read-write mode) – Management VLAN – VMPS – Voice VLAN Figure 5-9 VLAN menu Reports (Figure 5-10) – Inventory – Port Statistics – Bandwidth Graphs – Link Graphs – Link Reports – Multicast – Resource Monitor – System Messages Figure 5-10 Reports menu Chapter 5.
Tools (Figure 5-11) – Ping and Trace Figure 5-11 Tools menu View (Figure 5-12) – Refresh – Front Panel Figure 5-12 View menu Window (Figure 5-13) – Front Panel View Figure 5-13 Window menu 34 Cisco Systems Intelligent Gigabit Ethernet Switch Module
Help (Figure 5-14) – Overview – What’s New? – Help For Active Window – Contents – Legend – About Figure 5-14 Help menu When you right-click a switch module, a device pop-up menu similar to the one shown in Figure 5-15 opens. The selected switch module will be surrounded by a yellow line.
Figure 5-17 Device pop-up menu: Bandwidth Graphs Figure 5-18 Device pop-up menu: Device Properties 36 Cisco Systems Intelligent Gigabit Ethernet Switch Module
When you right-click a port icon similar to the one shown in Figure 5-19, a port pop-up menu opens. You can select multiple ports by using Shift or Ctrl and configure them at the same time. Selecting all ports is also possible using Select All Ports from the pop-up menu. Selected ports will be surrounded by a yellow line.
Figure 5-21 Port pop-up menu: VLAN Figure 5-22 Port pop-up menu: Port Security Figure 5-23 Port pop-up menu: Link Graphs 38 Cisco Systems Intelligent Gigabit Ethernet Switch Module
5.1.4 Cisco Systems Intelligent Gigabit Ethernet Switch Module Tools Click Tools on the Switch Management Home page and a window similar to the one shown in Figure 5-24 opens. This window enables you to start a Telnet session to the switch or obtain monitoring and troubleshooting information. The Tools window has the following menu: Telnet Opens a Telnet session to the switch module. Extended Ping Opens a ping dialog, in which you can issue extended ping.
5.1.5 Cisco Systems Intelligent Gigabit Ethernet Switch Module Help Resources When accessing the Help Resources menu (Figure 5-25), you are provided links to other help resources and product documentation. Figure 5-25 Cisco Systems Intelligent Gigabit Ethernet Switch Module Help Resources 5.2 Systems management considerations In this section, we look at some of the system management functions of the Cisco Systems IGESM as well as system management tools.
and the Management Module itself. The IGESM can be managed via this path or over its own external uplinks. (See 5.3, “In-depth management path discussions” on page 55 for a more in-depth discussion for the rules for these management paths.) By default, the Ethernet internal switch management ports are placed in VLAN 1. Typically, Cisco’s recommendation is not to use VLAN 1 for security reasons, but it is still common to use VLAN 1 for management purposes.
Legend Ethernet Ethernet path Management Workstation I2C Interface 2 I2C path RJ45 Serial 1 3 RJ45 Serial path Routed Production Network Management Network 4 3 5 2 External Ethernet Interface Web Interface External Ports Internal Ethernet Interface G0/15 or 16 1A Cisco Systems IGESM Service Port I2C Interface Management Module 1B 4 Blade Server Figure 5-26 Management paths to the Cisco Systems IGESM Path 1 details Path 1 is from any Management Workstation, over the Management Network,
essentially on path 2, because the Management Module is now acting simply as a pass-through from the external Management Network into the Cisco Systems IGESM, through the Ethernet connection between the Management Module and the Cisco Systems IGESM. Note that for this path to function, the management IP address on the Cisco Systems IGESM must be in the same IP subnet that is in use on both the internal and external Ethernet interfaces of the Management Modules. See 5.3.
the Cisco Systems IGESM) are on the same IP subnet, the Management Module will still attempt to proxy for the Cisco Systems IGESMs, which could result in confusion in the network. We also recommend that you change the management VLAN on the Cisco Systems IGESM to something other than the default VLAN1, and it will have be in a VLAN that contains the IP subnet that will be used to access the Cisco Systems IGESM through its external connections.
advanced way, by attaching it to a terminal server that is connected to an IP network (for the purposes of remote management through the service port on the Cisco Systems IGESM). See “Possible issues with Hyperterm when using the console port” on page 237 to ensure successfully utilizing this connection. Consequences of configuring for a particular management path Configuring for the use of a specific path as previously noted requires conscious choices that can affect the availability of the other paths.
Clustering servers can be achieved with clustering software on the servers, and high availability of data paths can be achieved with proper network design and the use of such features as Trunk Failover and NIC Teaming. Building a new cluster To build a new cluster, perform these steps: 1. Assign an IP address to the switch on the management subnet. This can be performed using the Management Module. See Figure 6-4 on page 83. 2. Connect this switch to other switches that run clustering software.
Adding to a cluster To add additional devices to the cluster, perform the following steps: 1. Choose Cluster → Add To Cluster from the menu bar (Figure 5-30). Figure 5-30 Cisco Cluster Management Suite window 2. In the Current Candidates list (Figure 5-31), select the switches that you want to add to the cluster. To select all the switches in the list, click Select All. Figure 5-31 Add To Cluster window 3. Click Add to move your selections to the Add To Cluster list.
6. From the menu bar, choose Administration → Save Configuration to save your changes to nonvolatile memory. Allow approximately one minute for changes to be saved to nonvolatile memory before resetting or turning off the switch. When the device has been added successfully, its label turns green (Figure 5-32). Figure 5-32 Front Panel View 7. Click the box of the device you want to display on the Front Panel View (Figure 5-33). CMS queries the device and displays it.
8. Any device in the cluster can now be managed either by clicking the switch graphic and then the menu bar task, or by choosing the task on the menu bar and selecting the device’s host name (Figure 5-34). In our example, we chose VLAN. Figure 5-34 VLAN Devices window Cluster topology view After a cluster is created, it can be viewed in a graphical representation. Click the Topology icon on the menu bar to display the topology (see Figure 5-35).
Clicking the Topology icon opens a window similar to the one shown in Figure 5-36. CMD Topology Options icon Figure 5-36 Topology View window This shows a network map of the command switch and cluster members; the command switch is labeled CMD. The view can also include any cluster candidates, neighboring devices, neighboring clusters, and node and link information. Figure 5-36 is a snapshot of one of our setups: BladeCenter1Bay1 is the command switch (CMD).
This opens a window similar to the one shown in Figure 5-37. Note that in this figure, all available filter options are enabled. Figure 5-37 Topology View window Chapter 5.
Right-click a device, and a pop-up window with management selections opens (Figure 5-38). Figure 5-38 Topology View Right-click a link and a pop-up window with management selections opens (Figure 5-39). Figure 5-39 Topology View window 5.2.5 CiscoWorks LAN Management Solution CiscoWorks LAN Management Solution (LMS) provides a foundation of basic and advanced device management applications that help network operators manage their networks.
For more information about CiscoWorks LAN Management Solution (LMS), visit: http://www.cisco.com/en/US/products/sw/cscowork/ps2425/index.html 5.2.6 CiscoView CiscoView is a Web-based device management application providing dynamic status, monitoring, and configuration information for the broad range of Cisco internetworking products. CiscoView displays a physical view of a device chassis, with color-coding of modules and ports for at-a-glance status.
CiscoWorks Campus Manager Designed for operational use, Campus Manager provides layer 2 tools for configuring, managing, and understanding complex physical and logical infrastructures. Campus Manager enables administrators to more easily change, monitor, and control network relationships, making them more effective in delivering business-critical and advanced networking services to their users and customers. 5.2.
5.3 In-depth management path discussions In this section, we take a closer look at the interactions of the Cisco Systems IGESM with the Management Module and at the rules that are necessary to ensure a stable management connection to the IGESM. 5.3.1 Introduction to this in-depth management discussion This section attempts to clarify the preferred paths, in particular the management paths for the various types of traffic that will be carried to and through the Cisco Systems IGESM in the IBM BladeCenter.
There is another (internal) consequence of the Management Module proxying for its IP subnet: If a blade server is placed on the IP subnet and VLAN that is being used by the IGESM’s management interface, the blade server will almost certainly fail to bring up its IP interface, as one of the first things most OSs do when they bring up an IP interface is to send out an ARP request looking for its own IP address (to make sure someone else is not already using it).
5.3.3 General management path design considerations Previous sections discussed the concept of in-band and out-of-band management. Here we look more closely at a subtle distinction between the Cisco Systems IGESM and traditional stand-alone Cisco switches with regard to these paths.
As noted, an important aspect of these two network-based management paths is that the selection of which to use is an either/or proposition. It is necessary to configure specifically to manage the IGESM via the Management Modules uplink (as shown in scenarios 1, 2, and possibly 7) or its own uplinks (as demonstrated in scenarios 3 and 4). Attempting to configure IGESM management for both paths at the same time usually creates intermittent connectivity issues when attempting to connect to the IGESM.
Avoid carrying management traffic and data traffic in the same VLAN. Limit the use of any VLAN used for management to only those ports that have to use that VLAN. Prune or otherwise block it from non-necessary links. Only carry VLANs on a trunk that are needed on the other side of the trunk, and prune or block all other VLANs. More about the reasoning behind these recommendations can be found in the “Virtual LAN Security Best Practices” document at: http://www.cisco.
Management Module first, then connect to the IGESM, but that you can actually point a Telnet or browser session directly at the IP address of the IGESM and directly attach (via a path through the Management Module) into the IGESM as you would any other Cisco switch.
There is one possible exception to this rule: Often during an IGESM’s initial evaluation period, a simple single VLAN network is set up for test purposes. Scenario 7 in this section discusses the possibilities and ramifications of this approach.
5. Ensure proper IP subnet selection. Make sure that the IP subnet that is used by the IGESM is different from the subnet that is defined on the Management Module for its own IP addresses. As with step 3, it is important to isolate the IGESM management path to the Management Module because we are now going to manage the IGESM via its own uplinks. Different IP subnets between the Management Module and the IGESM will complete this isolation.
the management VLAN on ports g0/15 and g0/16 the native VLAN, having different management VLANs results in each IGESM complaining about a native VLAN mismatch in their respective logs and on their console ports.
5.3.7 Scenario 1 (recommended) IGESM management using Management Module uplink Physically isolated management and data networks Management Network Data Network 802.1Q Trunk(s) or Mode Access Mode Access VLAN A, B, C... VLAN X VLAN X should be blocked Must be same IP subnet Sw Bay - external management over all ports disabled ETH 0 Management Module MGMT Interface VLAN X ETH 1 Bay 1 - 4 To blade servers BladeCenter Flow Name Data traffic MM and IGESM traffic IGESM Flow Line VLAN A, B, C.
It is not imperative in this environment of physically isolated management and data networks to block the management VLAN on the IGESM uplinks, but doing so prevents issues if the two upstream networks are ever physically merged. Note that this Scenario covers physically isolated networks (different switches and routers for each network).
See 5.3.4, “Considerations: Using the Management Module uplink to manage the IGESM” on page 59 for rules for this scenario. As noted in scenario 1, this is one of the simplest designs to deploy and support because all management traffic utilizes the Management Module’s uplink port and isolates this traffic from the data traffic.
5.3.9 Scenario 3 (recommended) IGESM management using IGESM uplinks IGESM, Management Module, and data traffic in separate VLANs Common Management/Data Network 802.1Q Trunk(s) Mode Access VLAN X VLAN Y VLAN A, B, C... Must be different IP subnets Sw Bay - external management over all ports enabled ETH 0 Management Module MGMT Interface VLAN Y ETH 1 Flow Line Bay 1 - 4 To blade servers BladeCenter Flow Name Data traffic MM traffic IGESM traffic IGESM VLAN A, B, C...
Of primary note is the fact that each of the management paths (for the IGESM and the Management Module) are on separate VLANs, thus separate IP subnets, and that the paths that are used for data traffic into the blade servers does not use either of these VLANs.
5.3.10 Scenario 4 (possible alternative) IGESM management using IGESM uplinks IGESM and data traffic in a common VLAN Common Management/Data Network 802.1Q trunk(s) or mode access Mode access VLAN X Sw Bay - external management over all ports enabled VLAN Y Must be different IP subnets ETH 0 Management Module MGMT Interface VLAN Y ETH 1 IGESM Bay 1 - 4 VLAN Y.
likely have a difficult time connecting to devices on the network. This is because the Management Module will attempt to proxy for any IP ARP requests (coming up from the blade server and over to the Management Module via the internal connection), and the blade server may see a duplicate IP address for itself, or possibly the wrong MAC address for its default gateway, resulting in failure to complete a connection.
In scenario 5, we attempt to utilize the uplink ports on the IGESM to manage the IGESM, and use the uplink port of the Management Module to manage the Management Module, but we place them in the same VLAN and presumably the same IP subnet.
5.3.12 Scenario 6 (not recommended) IGESM management using IGESM uplinks IGESM, Management Module, and Data Traffic all in common VLAN Common Management/Data Network 802.
Blade server sends initial ARP to world requesting info for own IP address (essentially a duplicate IP address check) ARP – “Who has 172.16.1.200?” As a broadcast, this is carried on all ports on the VLAN, including the internal facing ports toward the MM ETH 0 MGMT interface VLAN X Eth1 IP 172.16.1.2 Management ETH 1 Module 2 MM sees ARP request on same IP subnet on internal interface Sends ARP response: “I own 172.16.1.200 !” IGESM Blade server using VLAN X 1 Blade server IP – 172.16.1.
5.3.13 Scenario 7 (possible evaluation test environment) IGESM management using Management Module uplinks Management Module and data traffic all in common VLAN IGESM on internally different VLAN, but shares Management Module uplink VLAN for management Common Management/Data Network 802.
See 5.3.4, “Considerations: Using the Management Module uplink to manage the IGESM” on page 59 for basic rules for this scenario. Keep in mind that this violates some of those rules as explained below, but is still functional for evaluation purposes. As noted earlier, this scenario might prove useful for evaluating the IGESM on a single VLAN test network. But because it shares the same VLAN for all traffic, it is not advised for use in production environments.
interface vlan 4000 Creates the new management interface based on the new VLAN. no shutdown Brings up the new management VLAN. Moves the IP address over from the old VLAN interface. Shuts down the old VLAN interface. Changes the native VLAN on ports g0/15 and 16 to 4000. Adds VLAN 4000 to the VLAN carried list on g0/15 and 16. end Exits configuration mode. write Saves configuration to NVRAM.
Changing the IGESM’s blade server facing ports Based on the stated goals of this scenario, any blade server ports (g0/1 – 14) also must be placed into VLAN 1. The following text shows an example of placing the blade server in front slot 1 into Access VLAN 1: conf t Places IGESM into configuration mode. interface g0/1 Must be performed on any blade server facing port to be used for this test. switchport mode access Sets port for access.
78 Cisco Systems Intelligent Gigabit Ethernet Switch Module
6 Chapter 6. IBM eServer BladeCenter system initial setup This chapter discusses the network topology and the hardware configured to provide you with a tested and working configuration to help implement your Cisco Systems Intelligent Gigabit Ethernet Switch Module (IGESM) for the IBM Eserver BladeCenter. As noted elsewhere in this document, the information herein applies to the 4-port copper-based IGESM running a 12.1(14) version of IOS.
6.1 IBM eServer BladeCenter system In this section, we discuss the stages of our preparing our BladeCenter for operation. 6.1.1 Management Module firmware After the required hardware has been installed in your BladeCenter, you should update the Management Module using IBM eServer BladeCenter - Management Module Firmware Update Version 1.10 or later. Go to the following Web sites to acquire the firmware: http://www.ibm.com/pc/support/site.wss/document.
Establishing a physical connection to the Management Module The only way to manage the Management Module is through the external 10/100 Mbps Ethernet port on the front of the module. To establish the physical connection to the Management Module, use one of the following methods: Use a Category 3, 4, 5, or higher unshielded twisted pair (UTP) straight-through cable to connect the Ethernet port on the Management Module to a switch in a network that has an accessible management station.
Figure 6-2 Management Module External Network Interface window The BladeCenter Management Module defaults to the IP address 192.168.70.125. If you have more than one BladeCenter on your Management Network, you are required to change the external network interface (eth0). If you do not, you will have IP address conflicts that will result in not being able to access your Management Modules.
6.1.3 I/O module management tasks In this section, we set up and configure the Cisco Systems IGESM. IGESM setup and configuration The IGESM can be installed into any of the four BladeCenter switch bays in the rear of the chassis. Bay 1 is attached to one of the Ethernet Network Interfaces Controllers (NIC) on the blade HS20. Bay 2 is attached to the other Ethernet NIC. Each NIC is a Gigabit Full Duplex link to only one of the switches.
Enable IGESM uplink ports through the Management Module In this section, we enable the Ethernet ports of the Cisco Systems Intelligent Gigabit Ethernet Switch Module from the BladeCenter Management Module. In the I/O Module Tasks → Management (Bay 1 Ethernet SM) window shown in Figure 6-4 on page 83, click Advanced Management. If necessary, scroll down to the Advanced Setup section. You must at least set the External ports to Enabled for data to be sent out through the switch (Figure 6-5).
External management over all ports Use this field to enable or disable external configuration management of this module. When this field is set to Disabled, only the Management Module ports can be used to change the configuration on this module (in other words, out-of-band management). When the field is set to Enabled, all ports (including internal, external, and Management Module ports) are enabled for management and you must follow certain rules. See 5.
4. Compare the level of software that you noted from the show version command to the latest level of available software. If the two software levels do not match, download the latest level from the Web and install it on your switch. Upgrading the switch software Switch software is upgraded through a TFTP server application. Typically, this software runs as an application under your operating system.
2. Change the TFTP Root Directory to the location of the Cisco switch firmware to be updated (Figure 6-7). Figure 6-7 Switch firmware location 3. Click the Security tab and change the TFTP Server to Transmit and Receive files. Click OK to save. The TFTP Server is now running.
3. Click Advanced Management in the bay in which the switch resides. 4. To start a Telnet session, click Start Telnet Session. Note: The Java 1.4 Plug-in is required to run this application. If it is not installed, it will be downloaded if an Internet connection is available. If an Internet session is not available, download it and install it separately. Complete these steps to upgrade the switch software: 1. Enter your user ID and password.
Note: The UpdateXpress CD might not always provide the latest level updates at the time of configuration, because updates are released as needed. This applies to both firmware and device drivers. Check the IBM Support Web site for the updates released later than the UpdateXpress CD: http://www.ibm.com/servers/eserver/support/xseries/index.html Getting started Prior to starting IBM UpdateXpress, you should back up your system unless you have a pristine system.
Figure 6-9 BladeCenter Firmware VPD window 6.2.2 Operating systems In this section, we prepare the use of our operating systems for the BladeCenter HS20s. Creating a Microsoft Windows 2000 Server installation CD To install Microsoft Windows 2000 Server or Advanced Server, you must have a shrink-wrapped version integrated with Service Pack 3 or later.
service pack without installing it. Note that from this machine, you should be able to create a new CD-ROM image. We downloaded the image from: http://www.microsoft.com/windows2000/downloads/servicepacks/sp3/sp3lang.asp 2. Copy the i386 directory from the Windows 2000 CD to the machine with the CD writer, to the d:\images\bootcd\i386 directory. 3. Apply the service pack with the /s option to the d:\images\bootcd\ directory. We used the command: d:\images\sp3\update\update /s:d:\images\bootcd 4.
In this section, we update the device drivers on the supported servers, HS20 Type 8832 running Windows Advanced Server 2000, using UpdateXpress V3.3. Complete the following steps to update the device drivers: 1. Start the system. 2. Insert the UpdateXpress CD into the CD-ROM drive. Note: If the CD-ROM does not automatically start UpdateXpress, use DOS to navigate to the UpdateXpress directory on the CD; then, run launch.exe. UpdateXpress displays all of the supported device drivers that it detects.
Red Hat Linux AS 2.1 Broadcom driver installation In this section, we install Red Hat Linux AS 2.1. After loading the operating system, the network drivers worked immediately. However, we downloaded the latest Broadcom device drivers for Linux and performed the instructions in Example 6-1 to install them. The latest Broadcom device drivers were obtained from the following URL: http://www.ibm.com/pc/support/site.wss/document.
-rwxr-xr-x root/root 26544 2004-01-06 12:05:13 ./baspcfg -rw-r--r-- root/root 1921 2004-01-06 12:05:13 ./baspcfg.8.gz -rw-r--r-- root/root 2240 2004-01-06 12:05:13 ./bcmtype.h -rw-r--r-- root/root 6913 2004-01-06 12:05:13 ./blf.c -rw-r--r-- root/root 1312 2004-01-06 12:05:13 ./blfcore.h -rw-r--r-- root/root 53116 2004-01-06 12:05:13 ./blfcore.o -rw-r--r-- root/root 122 2004-01-06 12:05:13 ./blfopt.h -rw-r--r-- root/root 1795 2004-01-06 12:05:13 ./blfver.h -rw-r--r-- root/root 4622 2004-01-06 12:05:14 .
cp -f basp.o /var/tmp/basplnx-buildroot/lib/modules/`uname -r`/kernel/net/basp mkdir -p /var/tmp/basplnx-buildroot/lib/modules/`uname -r`/build/include/linux cp -f nicext.h /var/tmp/basplnx-buildroot/lib/modules/`uname -r`/build/include/linux Processing files: basplnx-6.2.1-1 Executing(%doc): /bin/sh -e /var/tmp/rpm-tmp.67022 Finding Provides: (using /usr/lib/rpm/find-provides)... Finding Requires: (using /usr/lib/rpm/find-requires)... PreReq: /bin/sh /bin/sh /bin/sh rpmlib(PayloadFilesHavePrefix) <= 4.
Figure 6-10 Broadcom selection window 2. Click MANAGEMENT PROGRAMS, and a window similar to Figure 6-11 opens. Figure 6-11 Select Features window 3. Select Control Suite and BASP. Click Next to continue, then click Finish.
6.3 Firmware and device drivers used in this example We applied the following firmware and drivers to our environment: IBM eServer BladeCenter Management Module: – Management Module Firmware Update Version 1.10 BladeCenter HS20(8832) firmware: – BladeCenter HS20 (Type 8832) - Flash BIOS Update Version 1.04 – BladeCenter HS20 (Type 8678,8832) - blade server integrated system management processor firmware update Version 1.04 – Broadcom NetXtreme firmware level 3.
98 Cisco Systems Intelligent Gigabit Ethernet Switch Module
7 Chapter 7. Cisco Systems IGESM configuration and network integration This chapter discusses the configuration of several scenarios that incorporate the IBM Eserver BladeCenter, using the embedded Cisco Systems Intelligent Gigabit Ethernet Switch Module (Cisco Systems IGESM), into a data center type environment. We provide configuration examples using both the Cisco command-line interface (CLI) and the Cluster Management Suite (where appropriate).
7.1 Introduction to configuration and integration The Cisco Systems IGESM module discussed in this document is a standards-based layer 2 switch, with QoS features based on layer 2 through 4 information, using Cisco Systems Internet Operating System (IOS). It contains most of the features and functionality traditionally associated with a Cisco switch, in a hot-pluggable module dedicated for use in a BladeCenter. 7.1.
Top IGESM (Bay 1) 1 2 3 4 5 17 6 7 8 9 10 11 12 13 14 15 16 18 19 20 IGESM External uplinks Eth1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 B S S 1 B S S 2 B S S 3 B S S 4 B S S 5 B S S 6 B S S 7 B S S 8 B S S 9 B S S 1 0 B S S 1 1 B S S 1 2 B S S 1 3 B S S 1 4 2 2 2 2 2 MM1 Uplink Eth0 MM1 MM2 Eth0 2 2 2 2 2 2 2 2 2 MM2 Uplink Eth1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Bottom IGESM (Bay 2) BladeCenter Chassis 16 17 18 19 20 IGESM External uplinks
the port on the Cisco Systems IGESM to be an access port (switchport mode access) and then set the access VLAN to the desired VLAN (switchport access vlan x). You can also change the default VLAN by leaving it trunked and changing the native VLAN to some other value. Portfast and BPDU Filter enabled by default on blade server ports. Portfast and BPDU filters can be disabled by users. All blade server ports are given a default description to match their function.
Ports G0/17 through G0/20: Connects to external ports 17 through 20, respectively: Preset default values for ports going to external connections (includes ports g0/17 through g0/20, shown for port g0/17): description extern1 switchport trunk native vlan 2 These ports default to shutdown when in a new BladeCenter. You must use the Management Module Web interface, under I/O tasks, Advanced settings to set External Ports to Enabled) to bring them up the first time.
See 5.3, “In-depth management path discussions” on page 55 for more details on why this is important. Default Spanning Tree settings: – spanning-tree mode rapid-pvst Rapid-PVST implements 802.1w for quicker recovery from issues in a layer 2 network without having to perform a lot of extra commands that were necessary prior to 802.1w to achieve this same rapid recovery.
7.2 Management network considerations This section discusses an extremely important topic for the BladeCenter: the selection of the management VLAN and its use within the BladeCenter. Although the BladeCenter has some very specific needs regarding its management VLAN, it might help to first generically understand the importance of selecting a suitable VLAN for management traffic, as well as the role the native VLAN can play in this selection.
Because the default VLAN for the management VLAN is 1, under the rule above, do not place blade servers on VLAN 1. If you change the management VLAN to something other than 1 (by creating a new VLAN, creating an interface for that new VLAN, and performing a no shutdown on the new interface), do not put blade servers on this new VLAN.
Note that the current revision of code available at the time of this most recent update to this document is 12.1(14)AY4. It is strongly recommended that this or a newer revision be installed to ensure all recent bug fixes are applied and new features are available. Cisco Catalyst 6500 switch hardware and software Two Cisco Catalyst 6509s each with: IOS version: 12.2(17d)SXB Image name: s72033-jk9sv-mz.122-17d.SXB.
Note: Available features and command syntax can be different with different versions of code. This document was prepared using the features and syntax from the aforementioned revisions of code, and as such, might vary from other revisions. For complete and current lists of available features and commands for these products, visit the IBM or Cisco Web sites. Base configuration options common to all examples Here, we list some configuration options established that are common to all of the examples.
Using the Management Module Web interface Perform the following steps to use the Management Module Web interface to enable the external ports of the Cisco Systems IGESMs for the first time: 1. Point your browser to the external IP address of the Management Module in bay 1 (defaults to 192.168.70.125) and log on using the following credentials: ID= USERID and Password = PASSW0RD (where 0 in password is a numeric zero). 2. On the left side of the window, under I/O module tasks, click Management. 3.
– 6500-1, VLAN 40 address: 10.1.40.251/24 HSRP address: 10.1.40.254/24 – 6500-1, VLAN 45 address: 10.1.45.251/24 HSRP address: 10.1.45.254/24 – 6500-1, VLAN 50 address: 10.1.50.251/24 HSRP address: 10.1.50.254/24 IP Addresses used in the base for 6500-3 are as follows: – 6500-3, VLAN 10 address: 10.1.10.253/24 HSRP address: 10.1.10.254/24 – 6500-3, VLAN 15 address: 10.1.15.253/24 HSRP address: 10.1.15.254/24 – 6500-3, VLAN 20 address: 10.1.20.253/24 HSRP address: 10.1.20.
vtp mode transparent ! spanning-tree mode rapid-pvst spanning-tree vlan 1-2,10,15,20,25,30,35,40,45,50 priority 8192 ! enable password ese ! vlan 2 ! vlan 10 name Web ! vlan 15 name User ! vlan 20 name Application ! vlan 25 name Backup ! vlan 30,35,40,45,50 ! interface Port-channel1 switchport switchport trunk encapsulation dot1q switchport trunk allowed vlan 1,2,10,15,20,25,30,35,40,45,50 switchport mode trunk switchport nonegotiate ! interface TenGigabitEthernet6/1 no ip address switchport switchport trun
standby 1 ip 10.1.10.254 standby 1 timers 1 3 standby 1 priority 110 standby 1 preempt delay minimum 60 standby 1 authentication cisco no shutdown ! interface Vlan15 ip address 10.1.15.251 255.255.255.0 no ip redirects no ip proxy-arp arp timeout 200 standby 1 ip 10.1.15.254 standby 1 timers 1 3 standby 1 priority 110 standby 1 preempt delay minimum 60 standby 1 authentication cisco no shutdown ! interface Vlan20 ip address 10.1.20.251 255.255.255.
standby 1 ip 10.1.35.254 standby 1 timers 1 3 standby 1 priority 110 standby 1 preempt delay minimum 60 standby 1 authentication cisco no shutdown ! interface Vlan40 ip address 10.1.40.251 255.255.255.0 no ip redirects no ip proxy-arp arp timeout 200 standby 1 ip 10.1.40.254 standby 1 timers 1 3 standby 1 priority 110 standby 1 preempt delay minimum 60 standby 1 authentication cisco no shutdown ! interface Vlan45 ip address 10.1.45.251 255.255.255.
spanning-tree mode rapid-pvst spanning-tree vlan 1-2,10,15,20,25,30,35,40,45,50 priority 28672 ! enable password ese ! vlan 2 ! vlan 10 name Web ! vlan 15 name User ! vlan 20 name Application ! vlan 25 name Backup ! vlan 30,35,40,45,50 ! interface Port-channel1 switchport switchport trunk encapsulation dot1q switchport trunk allowed vlan 1,2,10,15,20,25,30,35,40,45,50 switchport mode trunk switchport nonegotiate ! ! interface TenGigabitEthernet6/1 no ip address switchport switchport trunk encapsulation dot1
standby 1 timers 1 3 standby 1 priority 110 standby 1 preempt delay minimum 60 standby 1 authentication cisco no shutdown ! interface Vlan15 ip address 10.1.15.253 255.255.255.0 no ip redirects no ip proxy-arp arp timeout 200 standby 1 ip 10.1.15.254 standby 1 timers 1 3 standby 1 priority 110 standby 1 preempt delay minimum 60 standby 1 authentication cisco no shutdown ! interface Vlan20 ip address 10.1.20.253 255.255.255.0 no ip redirects no ip proxy-arp arp timeout 200 standby 1 ip 10.1.20.
standby 1 ip 10.1.35.254 standby 1 timers 1 3 standby 1 priority 110 standby 1 preempt delay minimum 60 standby 1 authentication cisco no shutdown ! interface Vlan40 ip address 10.1.40.253 255.255.255.0 no ip redirects no ip proxy-arp arp timeout 200 standby 1 ip 10.1.40.254 standby 1 timers 1 3 standby 1 priority 110 standby 1 preempt delay minimum 60 standby 1 authentication cisco no shutdown ! interface Vlan45 ip address 10.1.45.253 255.255.255.
The topologies presented in this chapter discuss attaching the BladeCenter to an external infrastructure made up of Cisco 6500s running in native mode (using IOS for all control of the 6500). The 6500 has other possible code configurations (for example, hybrid mode, where both IOS and CatOS are running). Although there are many possibilities for both platform choice and code choice, the 6500 in native mode was chosen as the best option for use in a data center environment.
Important: Although you can attach the Cisco Systems IGESMs to external switches at 100 Mb speeds, in production environments, we strongly recommend that you use 1000BaseT connections (available on all Cisco platforms suitable for data center environments) to ensure the best possible throughput. Use of the terms trunk and aggregation Some industry terms and acronyms have proven to be a source of confusion. One such term is the word trunk or trunking.
Important: The links connecting the aggregation switches, 6500-1 and 6500-3 in this document, are absolutely critical in the operation and health of the network. Based on this, and as already noted, the links between these two switches should be spread over multiple modules. This will help to ensure that a single module failure in either chassis will not take this entire link down.
involving link aggregation (because the HS40 offers two NICs to each IGESM instead of the HS20’s single NIC to each IGESM), but no testing was done with the HS40 during the creation of this document. IP addressing and MAC addressing When an SLB team is created, IP addresses are configured on the virtual adapter and not on the team member physical NICs. SLB teaming supports both Active/Active and Active/Standby configurations.
Systems IGESM port connected to the NIC as a trunk port and configure VLANs accordingly. Also note that the server should have 64 MB of system memory per 8 VLANs configured on a BASP virtual adapter in order to maintain optimum performance. For more information about BASP NIC teaming, refer to the BACS online help and BCM570X Broadcom NetXtreme Gigabit Ethernet Teaming white paper, which is available at: http://www.broadcom.com/collateral/wp/570X-WP100-R.
– Repeat for the next Cisco Systems IGESM. 4. Configure the blade server ports on the server blades: – Configure any desired teaming or SLB, or both. – Configure any desired VLANs/trunking. – Configure any desired access links. – Configure desired IP address. – Repeat for the next blade server. 5. Re-enable or recable the links that were disabled in step 1 (Table 7-2 on page 123). 6. Confirm the desired operation of the configuration.
Descriptions and comments Via CLI Via Management Module Web interface Via CMS user interface Option 3: Pull connecting cables from either the Cisco Systems IGESM or the external switch. N/A N/A N/A Summary of reconnect procedure to be performed for each example Table 7-2 includes the steps performed after the configuration of both sides of the connection is complete. It should be the reverse of the procedure used from Table 7-1 on page 122.
7.5 Example topologies and their configuration This section provides several topologies and offers reasons for their selection, as well as step-by-step configuration options. 7.5.
To Core Routers Po1 Data Center 6500-1 Aggregation Layer Data Center 6500-3 Mod 6 - 1 Mod 6 - 2 Mod 2 - 25 26 27 28 Mod 6 - 1 Mod 6 - 2 25 26 27 28 - Mod 2 Po2 Po2 Po1 Po1 17 18 19 20 17 18 19 20 CIGESM1 CIGESM2 Trunked VLAN 10 and 15 Access VLAN 10 Trunked VLAN 20 and 25 Management Netw ork Access VLAN 20 M M 1 M M 2 Trunked VLAN 10 and 15 1 Trunked VLAN 20 and 25 2 Access Access 1 2 Management Workstation Blade Server 1 Blade Server 2 BladeCenter Topology 1 between Managemen
Step 1: Taking down the link or links You should disable links before making any configuration changes (Table 7-1 on page 122). Step 2: Configuring the external switches The following assumptions have been made for this example: The bulk of the configuration for the 6500s is included in the base configuration (see “Cat 6500 base configurations” on page 109), because the goal of this document is to show how to configure the BladeCenter components rather than generic Cisco devices.
Step 3: Configuring Cisco Systems IGESMs This section steps through the sequence of actions required to configure the Cisco Systems IGESMs for this example. It has two major sections: one for configuring the Cisco Systems IGESM in bay 1 and one for configuring the Cisco Systems IGESM in bay 2.
Description and comments Actions via IOS CLI for CIGEMS1 Actions via CMS for CIGEMS1 Step 3.1.2: Configure link aggregation toward 6500-1. int range g0/17 - 20 description To-6500-1 channel-group 1 mode active 1. On the top toolbar, click Port → EtherChannels. 2. Click Create. 3. Select the check boxes next to ports Gi0/17 through Gi0/20. 4. Enter 1 in the Group [1-6] field to select the port channel to use. 5. Click OK. 6. Click Apply or OK. This example uses LACP to form the aggregation.
Description and comments Actions via IOS CLI for CIGEMS1 Actions via CMS for CIGEMS1 Step 3.1.6: Save Cisco Systems IGESM config to NVRAM. Failure to perform copy running-config startup-config 1. On the top toolbar, click Administration → Save Configuration. 2. Leave the Source set to Running Configuration. 3. In Destination, select Startup Configuration. 4. Click Save.
Description and comments Actions via IOS CLI for CIGEMS2 Actions via CMS for CIGEMS2 Step 3.2.3: Configure 802.1Q trunking toward 6500-3. int port-channel 1 description EtherChannel-To-6500-3 switchport trunk native vlan 2 switchport trunk allowed vlan 2,20,25 switchport mode trunk 1. In the top menu bar, click VLAN → VLAN. 2. Hold down the Ctrl key and click ports Gi0/17 through Gi0/20. 3. Click Modify. 4. In the Trunk-Allowed VLAN field, enter 2,20,25. 5.
Description and comments Actions via IOS CLI for CIGEMS2 Actions via CMS for CIGEMS2 Step 3.2.6: Save Cisco Systems IGESM config to NVRAM. copy running-config startup-config 1. In the top menu bar, click Administration → Save Configuration. 2. Leave the Source set to Running Configuration. 3. In Destination, select Startup Configuration. 4. Click Save.
Note that the choice to use more than one default gateway (for example, one on each VLAN) is up to the user. See the discussion about default gateways on multihomed systems in Appendix A, “Hints and tips” on page 227. Step-by-step instructions to configure BladeServer1 Table 7-6 shows the step-by-step instructions to configure BladeServer1. Table 7-6 Configuring BladeServer1 for 802.1Q trunks with multiple VLANs Description and comments On BladeServer1 BASP using VLANs on both Ethernet ports Step 4.1.
Description and comments On BladeServer1 BASP using VLANs on both Ethernet ports Step 4.1.4: Save the changes made to BASP. This step creates four new logical interfaces in Windows 2000: ToCIGESM1/VLAN10-WEB ToCIGESM1/VLAN15-USER ToCIGESM2/VLAN20-APPS ToCIGESM2/VLAN25-BACKUP Note: Exiting the BASP program without clicking Apply or OK will result in losing your configuration changes. 1. Click Apply in the main BASP window. 2.
Step 5: Reconnecting the devices This is the final step to bring the connection into full operation. This will be the reverse of whatever procedure was used in step 1. See Table 7-2 on page 123 for details about how to reestablish the links. Step 6: Verifying the configuration This section provides options for verifying the correct and desired operation.
Figure 7-6 Windows 2000 networking showing physical and logical interfaces on BladeServer1 Figure 7-7 Windows 2000 networking showing only physical interfaces on BladeServer2 From the CMD prompt (Start → Run → cmd → OK), execute the ipconfig command and confirm that the desired interfaces have the desired IP configuration on each blade server (make sure they are not reversed, where the IP address you want on the Local Area Connection is not on the connection named Local Area Connection 2, assuming they ha
At this time, you should be able to ping as just described. If you cannot ping to these addresses, and your above configuration checks were okay, proceed to the next section and inspect the other components in this configuration. Note: With this configuration (per this example), you will not be able to directly ping the management VLAN on the Cisco Systems IGESMs. This is because it is on a different VLAN than the blade servers.
Run the command show etherchannel 1 port-channel and check for the desired output (it should be similar on both Cisco Systems IGESMs): Port-channels in the group: ---------------------Port-channel: Po1 (Primary Aggregator) -----------Age of the Port-channel = 01d:05h:15m:50s Logical slot/port = 1/0 Number of ports = 4 HotStandBy port = null Port state = Port-channel Ag-Inuse Protocol = LACP Ports in the Port-channel: Index Load Port EC state No of bits ------+------+------+------------------+----------0 0
There is one case where high availability might still be an issue if NIC Teaming/Trunk Failover is not configured, and that is if both uplinks from a single Cisco Systems IGESM were to go down, but the Cisco Systems IGESM itself did not go down. In that case, the blade server would be unable to detect the upstream failure and issues would arise. Utilizing NIC Teaming and Trunk Failover would ensure that this is not an issue.
BladeServer4: 802.1Q trunk links carrying multiple VLANs on a teamed/SLB connection to the server. This configuration is provided to show how to use multiple NICs to look like a single NIC, but still make use of multiple VLANs on this single logical NIC. It makes use of the teaming drivers to tie the NICs together and create the desired VLANs.
To Core Routers Po1 Data Center 6500-1 Aggregation Layer Mod 2 - 25 26 Mod 6 - 1 Mod 6 - 2 27 28 Po2 Data Center 6500-3 Mod 6 - 1 Mod 6 - 2 25 26 Po3 Po1 Po2 Po2 Po2 17 18 19 20 CIGESM1 Management Netw ork Po3 Po1 17 18 19 20 Trunked VLAN 10 and 15 27 28 - Mod 2 Access Access VLAN 10 VLAN 30 Trunked VLAN 35, 40, 45 and 50 Trunked VLAN 20 and 25 CIGESM2 Access VLAN 20 Access VLAN 30 Trunked VLAN 35, 40, 45 and 50 M M 1 M M 2 Trunked VLAN 10 and 15 1 2 Trunked VLAN 20 and 25 Acces
Step 1: Taking down the link or links It is always advisable to disable the link or links prior to making any configuration changes. See Table 7-1 on page 122 for the needed procedures.
Description and comments On the 6500-1 On the 6500-3 Step 2.2: Configure VLAN and trunking options.
On BladeServer1, both ports will be using trunking (but not load balancing) through the Broadcom BASP software. The first port will be configured for VLANs 10 and 15, the second port will be configured for VLANs 20 and 25. On BladeServer2, both ports will be simple access links and will be placed on VLANs 10 and 20, respectively, through port settings on the Cisco Systems IGESMs.
Description and comments Actions via IOS CLI for CIGEMS1 Actions via CMS for CIGEMS1 Step 3.1.2: Configure Link Aggregation toward the 6500s. int range g0/17 - 18 description To-6500-1 channel-group 1 mode active This creates a logical interface named Port-Channel1 and places the interfaces g0/17 and g0/18 into it. 1. In the top menu bar, click Port → EtherChannels. 2. Click Create. 3. Select the check boxes next to ports Gi0/17 and Gi0/18. 4.
Description and comments Actions via IOS CLI for CIGEMS1 Actions via CMS for CIGEMS1 Step 3.1.5: Configure access links to BladeServer2 and set access VLAN. int g0/2 switchport mode access switchport access vlan 10 1. In the top menu bar, click VLAN → VLAN. 2. Click port Gi0/2. 3. Click Modify. 4. In the Administrative Mode field, select Static Access. 5. In the Static-Access VLAN field, enter 10. 6. Click OK. 7. Click Apply or OK. This places BladeServer2’s first NIC into VLAN 10. Step 3.1.
Step 3.2: Configuring the second Cisco Systems IGESM (CIGESM2) Table 7-10 shows the step-by-step instructions used to configure CIGESM2, showing both CLI and CMS commands. Important: The current version of CMS supported on the Cisco Systems IGESM has a limitation in its ability to completely control VLANs being placed on a given trunk: It always includes VLAN 1 and 1001-1005, even if you do not set them as allowed.
Description and comments Actions via IOS CLI for CIGEMS2 Actions via CMS for CIGEMS2 Step 3.2.3: Configure 802.1Q trunking toward 6500s and add allowed VLANs. int port-channel 1 description EtherChannel-To-6500-1 switchport trunk native vlan 2 switchport trunk allowed vlan 2,20,25,30,35,40,45,50 switchport mode trunk 1. In the top menu bar, click VLAN → VLAN. 2. Click po1. 3. Click Modify. 4. In the Trunk-Allowed VLAN field, enter 2,20,25,30,35,40,45,50. 5. Click OK. 6. Click Apply or OK. 7.
Description and comments Actions via IOS CLI for CIGEMS2 Actions via CMS for CIGEMS2 Step 3.2.6: Configure access links int g0/3 switchport mode access switchport access vlan 30 1. In the top menu bar, click VLAN → VLAN. 2. Click port Gi0/3. 3. Click Modify. 4. In the Administrative Mode field, select Static Access. 5. In the Static-Access VLAN field, enter 30. 6. Click OK. 7. Click Apply or OK. to BladeServer3 and set access VLAN. This places BladeServer3’s second NIC into VLAN 30. Step 3.2.
Commands are being performed in the sequence shown. BladeServer1: Trunk connection to Cisco Systems IGESM. – The Broadcom Advanced Server Program (BASP, also know as the Broadcom Advanced Control Suite) software has been installed on BladeServer1. BladeServer1 will be using the BASP software to create logical interfaces for VLANs 10, 15, 20, and 25, and all IP configuration will be performed on these logical interfaces (not on the physical interfaces).
BladeServer4 (VLANs 35, 40, 4,5 and 50), and all IP configuration will be performed on these four logical interfaces (not on the physical interfaces). – This logical port will connect to both CIGESM1 (port g0/4) and CIGESM2 (port g0/4) and will make use of LANs 35, 40, 45, and 50 through port settings on each Cisco Systems IGESM. – We will be using the following IP addresses (24-bit masks): First port, VLAN 10 to CIGESM1: 10.1.35.4 (default gateway = 10.1.35.254) First port, VLAN 15 to CIGESM1: 10.1.40.
Description and comments On BladeServer1 BASP using VLANs on both Ethernet ports Step 4.1.3b: Create desired VLANs on Team CIGESM2. Create and name VLANs 20 and 25 on the team going to CIGESM2. 1. 2. 3. 4. 5. Step 4.1.4: Save the changes made to BASP.
Step-by-step instructions to configure BladeServer2 Table 7-12 shows the step-by-step instructions used to configure BladeServer2. Table 7-12 Configuring BladeServer2 for standard interface connections Description and comments On BladeServer2 No BASP software, using physical access links on both Ethernet ports Step 4.2.1: Configure IP addresses directly on the desired interfaces. This procedure will be no different from configuring a stand-alone server with two NICs. 1.
Description and comments On BladeServer3 BASP using VLANs on both Ethernet ports for SLB Step 4.3.5: Configure desired IP address on each VLAN. This step assumes that the user knows how to add IP addressing information. Note that the default gateway used is part of the base HSRP config of the 6500s. Also note that on production systems, you would normally configure one or more DNS servers. This was not included as part of this environment but should be included in most production networks.
Description and comments On BladeServer4 BASP using teaming and VLANs on both Ethernet ports Step 4.4.4: Save the changes made to BASP. This step creates four new logical interfaces in Windows 2000: ToBoth-Trunked/VLAN35 ToBoth-Trunked/VLAN40 ToBoth-Trunked/VLAN45 ToBoth-Trunked/VLAN50 Note: Exiting the BASP program without clicking Apply or OK will result in losing your configuration changes. 1. Click Apply at the main BASP window. 2.
Figure 7-9 BladeServer1 BASP configuration Figure 7-10 BladeServer2 BASP configuration (BASP not used of BladeServer2) Figure 7-11 BladeServer3 BASP configuration Chapter 7.
Figure 7-12 BladeServer4 BASP configuration Using Windows 2000 networking tools, review the logical and physical network. The following figures show BladeServers 1, 2, 3, and 4.
Figure 7-15 Windows 2000 networking showing only physical interfaces on BladeServer3 Figure 7-16 Windows 2000 networking showing only physical interfaces on BladeServer4 From the CMD prompt (Start → Run → cmd → OK), execute the ipconfig command and confirm that the desired interfaces have the desired IP configuration on each blade server (make sure that they are not reversed, where the IP address you want on the Local Area Connection is not on the connection named Local Area Connection 2, assuming they ha
Ping from BladeServer4 to 10.1.35.254 (HSRP address on the 6500s) Ping from BladeServer4 to 10.1.40.254 (HSRP address on the 6500s) Ping from BladeServer4 to 10.1.45.254 (HSRP address on the 6500s) Ping from BladeServer4 to 10.1.50.254 (HSRP address on the 6500s) At this time, you should be able to ping as just described. If you cannot ping to these addresses, and your configuration checks above were okay, proceed to the next section and inspect the other components in this configuration.
Gi0/17 Gi0/18 Gi0/19 Gi0/20 on on on on 802.1q 802.1q 802.1q 802.
Verifying correct operation on the external switches (6500-1 and 6500-3) This section includes some commands you can use to verify the desired configuration and operation of the 6500s. Basically, you can run the same set of commands as previously shown for the Cisco Systems IGESMs. Naturally, there will be some differences in the output, but you want to make sure that the proper ports are channeled and trunked and carrying the correct VLANs. Also watch out for any admin down ports.
Important: The RSPAN reflector ports discussed and tested for this topology were one of the four external ports on the Cisco Systems IGESM. It is possible to use an unused blade server port (for example, g0/14) for the role of the reflector port, but extreme caution should be used, because using an internal port that had a blade server attached could lead to unexpected and undesired behavior.
g2/25 g2/26 g0/17 g0/18 g2/2 VLAN500 g0/19 - Reflector-port g0/1 CIGESM1 BladeCenter Data Center 6500-1 Sniffer 1 Blade Server 1 RSPAN source: Port g0/1 on CIGESM1 RSPAN Reflector-port: Port g0/19 on CIGESM1 RSPAN VLAN: VLAN 500 RSPAN destination: Port g2/2 on 6500-1 Figure 7-17 Desired RSPAN data flow for this example Configurations presented for blade server attachment to this topology Important: The blade server configurations offered in this chapter are not part of the topology discussion, but
configuration. From the Cisco Systems IGESM’s perspective, both connections are configured as simple access ports with a static VLAN assigned. – This configuration uses of the Broadcom teaming software to bind and balance the links together. The Cisco Systems IGESMs will establish what VLAN the teamed ports will be placed into (it will need to be the same for both Cisco Systems IGESM ports that go to this server).
To Core Routers Po1 Data Center 6500-1 Aggregation Layer Mod 2 - 25 26 RSPAN Port Mod 6 - 1 Mod 6 - 2 28 Po2 Po1 Po1 17 18 20 19 CIGESM1 20 Access Access VLAN 10 VLAN 30 Trunked VLAN 10 and 15 1 2 Trunked VLAN 20 and 25 Trunked VLAN 35, 40, 45 and 50 Access Access Trunked VLAN 20 Trunked and 25 VLAN 35, Access 40, 45 VLAN 20 and 50 Access VLAN 30 Teamed/Access Teamed/Trunked VLAN 35, 40, 45 and 50 1 2 Management Workstation Blade Server 1 17 18 CIGESM 19 M M 1 M M 2 25 26 - Mod 2
Step 1: Taking down the link or links It is always advisable to disable the link or links prior to making any configuration changes. See Table 7-1 on page 122 for the needed procedures.
Description and comments On the 6500-1 On the 6500-3 Step 2.2: Configure VLAN and trunking options.
Description and comments On the 6500-1 On the 6500-3 Step 2.4: Save config to copy running-config startup-config copy running-config startup-config NVRAM. Note: Failure to save your configuration will result in possible network down conditions if the switch is restarted prior to the save (all changes since last save will be lost). Step 3: Configuring Cisco Systems IGESMs This section steps through the sequence of actions required to configure the Cisco Systems IGESMs for this example.
Table 7-16 Configuring CIGESM1 Description and comments Actions via IOS CLI for CIGEMS1 Actions via CMS for CIGEMS1 Step 3.1.1: Configure desired VLANs for CIGESM1. Perform the following from the enable mode: config t vlan 10 name Web vlan 15 name User vlan 30,35,40,45,50 Perform the following from the CMS interface: 1. In the top toolbar, click VLAN → VLAN. 2. Click the Configure VLANs tab. 3. Click Create. 4. Enter 10 in the VLAN ID field. 5. Enter Web in the VLAN Name field. 6. Click OK. 7.
Description and comments Actions via IOS CLI for CIGEMS1 Actions via CMS for CIGEMS1 Step 3.1.3: Configure 802.1Q trunking toward 6500s and add allowed VLANs for both EtherChannel and single trunked link. int port-channel 1 description EtherChannel-To-6500-1 switchport trunk native vlan 2 switchport trunk allowed vlan 2,10,15,30,35,40,45,50,500 switchport mode trunk 1. In the top toolbar, click VLAN → VLAN. 2. Click po1. 3. Click Modify. 4.
Description and comments Actions via IOS CLI for CIGEMS1 Actions via CMS for CIGEMS1 Step 3.1.4: Configure RSPAN on port g0/19 on CIGESM1. Create the RSPAN VLAN and set it to support RSPAN: vlan 500 remote-span Configure the port to be monitored (g0/1 in this case), as well as the port performing the function of the reflector-port (g0/19): monitor session 1 source interface g0/1 CMS does not support configuring RSPAN at this time. Use the CLI to configure RSPAN.
Description and comments Actions via IOS CLI for CIGEMS1 Actions via CMS for CIGEMS1 Step 3.1.6: Configure access links to BladeServer2 and set access VLAN. int g0/2 switchport mode access switchport access vlan 10 1. In the top toolbar, click VLAN → VLAN. 2. Click port Gi0/2. 3. Click Modify. 4. In the Administrative Mode field, select Static Access. 5. In the Static-Access VLAN field, enter 10. 6. Click OK. 7. Click Apply or OK. This places BladeServer2’s first NIC into VLAN 10. Step 3.1.
Step 3.2: Configuring the second Cisco Systems IGESM (CIGESM2) Table 7-17 shows the step-by-step instructions used to configure CIGESM2, showing both CLI and CMS commands. Important: The current version of CMS supported on the Cisco Systems IGESM has a limitation in its ability to completely control VLANs being placed on a given trunk: It always includes VLAN 1 and 1001-1005, even if you do not set them as allowed.
Description and comments Actions via IOS CLI for CIGEMS2 Actions via CMS for CIGEMS2 Step 3.2.3: Configure 802.1Q trunking toward 6500s and add allowed VLANs for both EtherChannel and single trunked link. int port-channel 1 description EtherChannel-To-6500-3 switchport trunk native vlan 2 switchport trunk allowed vlan 2,20,25,30,35,40,45,50 switchport mode trunk 1. In the top toolbar, click VLAN → VLAN. 2. Click po1. 3. Click Modify. 4. In the Trunk-Allowed VLAN field, enter 2,20,25,30,35,40,45,50. 5.
Description and comments Actions via IOS CLI for CIGEMS2 Actions via CMS for CIGEMS2 Step 3.2.5: Configure 802.1Q trunking to BladeServer1 and add allowed VLANs. int g0/1 switchport trunk allowed vlan 2,20,25 1. In the top menu bar, click VLAN → VLAN. 2. Click port Gi0/1. 3. Click Modify. 4. In the Trunk-Allowed VLAN field, enter 2,20,25. 5. Click OK. 6. Click Apply or OK. Important: As noted in step 3.1.
Description and comments Actions via IOS CLI for CIGEMS2 Actions via CMS for CIGEMS2 Step 3.2.8: Configure 802.1Q trunking to BladeServer4 and add allowed VLANs. int g0/4 switchport trunk allowed vlan 2,35,40,45,50 end 1. In the top menu bar, click VLAN → VLAN. 2. Click port Gi0/4. 3. Click Modify. 4. In the Trunk-Allowed VLAN field, enter 2,35,40,45,50. 5. Click OK. 6. Click Apply or OK. Important: As noted in step 3.1.
A quick rundown for verifying your RSPAN session On CIGESM1, run the command show monitor, and review the output for the desired configuration: Session 1 --------Type Source Ports Both Reflector Port Dest RSPAN VLAN: : Remote Source Session : : Gi0/1 : Gi0/19 500 On 6500-1, run the command show monitor, and review the output for the desired configuration: Session 5 --------Type : Remote Destination Session Source RSPAN VLAN : 500 Destination Ports : Gi2/2 Attach a sniffer or other network monitor to port
To Core Routers Po1 Data Center 6500-1 Aggregation Layer Mod 6 - 1 Mod 6 - 2 Data Center 6500-3 Mod 6 - 1 Mod 6 - 2 Mod 2 - 25 26 RSPAN Port 25 26 - Mod 2 Po2 Po2 Po1 Po1 17 18 20 19 CIGESM1 Trunked VLAN 10 and 15 20 17 18 CIGESM 19 Access Access VLAN 10 VLAN 30 Trunked VLAN 35, 40, 45 and 50 RSPAN Port Trunked VLAN 20 Trunked and 25 VLAN 35, Access 40, 45 VLAN 20 and 50 Access VLAN 30 M M 1 Management Netw ork M M 2 Management Workstation Trunked VLAN 10 and 15 1 2 Blade Server 1
Step 1: Taking down the link or links It is always advisable to disable the link or links prior to making any configuration changes. See Table 7-1 on page 122 for the needed procedures.
Description and comments On the 6500-1 On the 6500-3 Step 2.2: Configure VLAN and trunking options.
Step 3: Configuring Cisco Systems IGESMs This section steps through the sequence of actions required to configure the Cisco Systems IGESMs for this example. It is broken into two major sections, one for configuring the Cisco Systems IGESM in bay 1 and one for configuring the Cisco Systems IGESM in bay 2.
Table 7-19 Configuring CIGESM1 Description and comments Actions via IOS CLI for CIGEMS1 Actions via CMS for CIGEMS1 Step 3.1.1: Configure desired VLANs for CIGESM1. Perform the following from the enable mode: config t vlan 10 name Web vlan 15 name User vlan 20 name Application vlan 25 name Backup vlan 30,35,40,45,50 Perform the following from the CMS interface: 1. In the top menu bar, click VLAN → VLAN. 2. Click the Configure VLANs tab. 3. Click Create. 4. Enter 10 in the VLAN ID field. 5.
Description and comments Actions via IOS CLI for CIGEMS1 Actions via CMS for CIGEMS1 Step 3.1.3: Configure 802.1Q trunking toward 6500s and add allowed VLANs for both EtherChannel and single trunked link. int port-channel 1 description EtherChannel-To-6500-1 switchport trunk native vlan 2 switchport trunk allowed vlan 2,10,15,20,25,30,35,40,45,50,500 switchport mode trunk 1. In the top menu bar, click VLAN → VLAN. 2. Click po1. 3. Click Modify. 4.
Description and comments Actions via IOS CLI for CIGEMS1 Actions via CMS for CIGEMS1 Step 3.1.4: Configure RSPAN on port g0/19 on CIGESM1. Create the RSPAN VLAN and set it to support RSPAN: vlan 500 remote-span CMS does not support configuring RSPAN at this time. Use the CLI to configure RSPAN. RSPAN can be configured several ways to capture traffic from a single port, multiple ports, or even VLANs.
Description and comments Actions via IOS CLI for CIGEMS1 Actions via CMS for CIGEMS1 Step 3.1.6: Configure access links to BladeServer2 and set access VLAN. int g0/2 switchport mode access switchport access vlan 10 1. In the top toolbar, click VLAN → VLAN. 2. Click port Gi0/2. 3. Click Modify. 4. In the Administrative Mode field, select Static Access. 5. In the Static-Access VLAN field, enter 10. 6. Click OK. 7. Click Apply or OK. This places BladeServer2’s first NIC into VLAN 10. Step 3.1.
Step 3.2: Configuring the second Cisco Systems IGESM (CIGESM2) Table 7-20 shows the step-by-step instructions used to configure CIGESM2, showing both CLI and CMS commands. Important: The current version of CMS supported on the Cisco Systems IGESM has a limitation in its ability to completely control VLANs being placed on a given trunk: It always includes VLAN 1 and 1001-1005, even if you do not set them as allowed.
Description and comments Actions via IOS CLI for CIGEMS2 Actions via CMS for CIGEMS2 Step 3.2.3: Configure 802.1Q trunking toward 6500s and add allowed VLANs for both EtherChannel and single trunked link. int port-channel 1 description EtherChannel-To-6500-3 switchport trunk native vlan 2 switchport trunk allowed vlan 2,10,15,20,25,30,35,40,45,50 switchport mode trunk 1. In the top menu bar, click VLAN → VLAN. 2. Click po1. 3. Click Modify. 4.
Description and comments Actions via IOS CLI for CIGEMS2 Actions via CMS for CIGEMS2 Step 3.2.5: Configure 802.1Q trunking to BladeServer1 and add allowed VLANs. int g0/1 switchport trunk allowed vlan 2,20,25 1. In the top menu bar, click VLAN → VLAN. 2. Click port Gi0/1. 3. Click Modify. 4. In the Trunk-Allowed VLAN field, enter 2,20,25. 5. Click OK. 6. Click Apply or OK. Important: As noted in step 3.1.
Description and comments Actions via IOS CLI for CIGEMS2 Actions via CMS for CIGEMS2 Step 3.2.8: Configure 802.1Q trunking to BladeServer4 and add allowed VLANs. int g0/4 switchport trunk allowed vlan 2,35,40,45,50 end 1. In the top menu bar, click VLAN → VLAN. 2. Click port Gi0/4. 3. Click Modify. 4. In the Trunk-Allowed VLAN field, enter 2,35,40,45,50. 5. Click OK. 6. Click Apply or OK. Important: As noted in step 3.1.
7.6 Miscellaneous blade server configurations This section includes several blade server configurations not covered elsewhere in this chapter. Note that any configurations using SLB (Active/Active or Active/Standby) are not recommend for use with topology 1 (as noted in the discussions in 7.5.1, “Topology 1: Dual IGESMs, four-port aggregation to two 6500s” on page 124) unless Trunk Failover is also configured per section 7.7, “Trunk Failover feature description and configuration” on page 193.
Figure 7-21 Verifying an Active/Standby SLB team SLB Active/Active teaming on BladeCenter HS20 with Red Hat Linux In this section, we demonstrate how to create an SLB team to connect to the Cisco Systems IGESMs as an access port on a BladeCenter HS20 running Red Hat AS 2.1.
#TEAM_PA2_NAME=eth2 #TEAM_PA2_ROLE=0 # 1st virtual interface in the team TEAM_VA0_NAME=sw0 TEAM_VA0_VLAN=0 TEAM_VA0_IP=10.1.30.3 TEAM_VA0_NETMASK=255.255.255.0 # Optional default gateway # One default gateway is usually specified for the system and it should be # reachable from one network interface TEAM_VA0_GW=10.1.30.254 802.1Q trunk links on BladeCenter HS20 with Red Hat Linux In this section, we demonstrate how to configure BACS to set a NIC to receive multiple VLANs through an 802.
# 1st virtual interface in the team TEAM_VA0_NAME=sw0 TEAM_VA0_VLAN=10 TEAM_VA0_IP=10.1.10.11 TEAM_VA0_NETMASK=255.255.255.0 # 2nd virtual interface in the team TEAM_VA1_NAME=sw1 TEAM_VA1_VLAN=15 TEAM_VA1_IP=10.1.15.11 TEAM_VA1_NETMASK=255.255.255.0 # Optional default gateway # One default gateway is usually specified for the system and it should be # reachable from one network interface TEAM_VA0_GW=10.1.10.
7.7 Trunk Failover feature description and configuration This section provides an explanation of the Trunk Failover feature (available in 12.1(14)AY4 and above IOS for the IGESM) as well as several configuration examples. For more about Trunk Failover operation and configuration, reference the IGESM Software Configuration Guide (link provided in the online resources section later in this document). 7.7.
6500 -1 XX X If failure anywhere on the link toward the upstream switch, NIC on BladeServer does not know about the failure and may continue to send traffic toward the top IGESM, which will discard the traffic - Trunk Failover feature addresses this issue 1 2 3 4 6500 -3 Po1 1 2 3 4 Po2 Po2 Po1 Po1 17 18 19 20 17 18 19 20 CIGESM Top CIGESM Bot X If IGESM fails in such a way that the link toward the BladeServer goes down, or NIC fails, BladeServer can sense this and redirect traffic out the
Important: The configuration of the Trunk Failover feature is only available through the IGESM CLI and not is not configure-able or monitor-able through CMS. 7.7.2 Example of Topology 1 using Trunk Failover Figure 7-23 logically depicts using Trunk Failover and NIC Teaming with Topology 1.
down internal downstream defined port(s). This alerts NIC Teaming to an upstream failure, at which point NIC Teaming switches to the other IGESM. This example shows a single VLAN to the Teamed NIC. It is possible to also carry multiple VLANs to the Teamed NIC. If multiple VLANs are necessary, you must carry all VLANs to both NICs and on all of the external uplinks as well as on Po1 between 6500-1 and 6500-3. Steps to configure for topology 1 Trunk Failover example 1. Configure global command. 2.
VLAN X Must be carried between IGESMs via an external path 6500-1 1 Po2 2 3 4 6500-3 Po1 1 2 Po3 3 4 Po2 Po3 VLAN X Must be carried on uplinks from IGESM VLAN X Must be carried on uplinks from IGESM Link state group 1 Upstream Po1 Po2 Link state group 1 Upstream Po2 Po1 17 18 19 20 17 18 19 20 CIGESM Top CIGESM Bot Link state group 1 Downstream VLAN X Link state group 1 Downstream Trunk Failover Config NIC Teaming Config Teamed Active/Standby NIC 1 NIC 2 Logical NIC Interface Blade
Steps to configure for topology 2 Trunk Failover example 1. Configure global command. 2. Configure upstream port (or ports) or Etherchannel (poX). 3. Configure downstream port (or ports). Configuring downstream before upstream will result in downstream ports going down until upstream is configured.
As noted, for a more detailed description of Serial over LAN, reference the SoL Configuration Guide (link provided in the online resources section later in this document). Some general rules for Serial over LAN Not all operating systems support SoL: – W2K Server not supported – Most Linux and W2K3 supported on HS series – AIX® and Linux on JS20 is supported Requires OS and BIOS configuration steps on some blade server models (e.g. HS20 running Linux or W2K3).
7.8.2 Configuring Serial over LAN This section provides an example of configuring SoL on an IGESM. Introduction to configuring Serial over LAN Successfully configuring Serial over LAN in the IBM BladeCenter involves several items. At a minimum you should configure both the Management Module and the IGESM. As already noted, depending on the blade servers installed, you may also need to configure both the CMOS/BIOS settings and the operating system on the server to also support Serial over LAN.
After SoL is configured for the IGESM, the Management Module and, if necessary, the blade servers must be configured. When all elements of SoL are configured, it will be possible to connect to the blade servers via the SoL connection. (See the Serial over LAN configuration guide for details). Important: When configuring the SoL VLAN on the Management Module, it never asks you to reboot after saving.
202 Cisco Systems Intelligent Gigabit Ethernet Switch Module
8 Chapter 8. Cisco Systems IGESM troubleshooting In this chapter, we discuss troubleshooting techniques and commands that can be used in support of the IGESM. © Copyright IBM Corp. 2004, 2005. All rights reserved.
8.1 Basic rules and unique symptoms Before going into detail about troubleshooting, it is important to first discuss certain common rules and symptoms for this environment. Certain interactions within the BladeCenter between the IGESM and the Management Module require that certain important rules be followed. Failure to follow these rules can produce unexpected results when deploying the BladeCenter containing an IGESM.
Table 8-1 Specific issues and recommendations Symptoms specific to BladeCenter environment and IGESM Possible cause/solution Duplicate IP address reported on IGESM Cause: IP address changed directly on IGESM rather than via Management Module. Solution: Change IP address to desired setting for IGESM on Management Module and click Save. See Appendix A, “Hints and tips” on page 227 for more about this issue.
Symptoms specific to BladeCenter environment and IGESM Possible cause/solution Upstream network goes down when hooking up IGESM to upstream Cause: Attempting to connect any two switches in a production network Server running Red Hat and tg3 driver not connecting to network, taking the upstream port down, or both Unable to enable external ports (g0/17-20) from IGESM. Reports Shutdown not allowed on this interface.
Otherwise, troubleshooting the IGESM is similar to other products in that there are typical basic types of failures that can be encountered. Some of these are: Hardware failures of IGESM – Not very common. – The only solution is RMA of defective IGESM. Software failures (bug in IGESM) – Not very common although, as with all products, software bugs do exist. – Reference the latest code readme file for a list of resolved bugs with each release of code.
Gathering this information usually requires the involvement of several technical support teams; for example, a network diagram usually comes from a network administration team, and blade server configuration usually comes from a systems administration team. From a support person’s perspective, answers to the following questions can also aid in this process.
If a critical condition is found during POST, the fault LED will be lit on back of IGESM. A POST code of FF indicates that the IGESM booted successfully. Additional messages may be found through a console port connection to the IGESM.
To execute different levels of diagnostics, log into the MM GUI and go to I/O Module tasks. Under Power/Restart, select one of the following options: 1. Run Standard Diagnostics Usually takes less than two minutes to test and complete boot; it runs: – Flash memory test – CPU cache memory test – DRAM test – Data path test – ASIC test 2. Run Extended Diagnostics Usually takes less than five minutes and runs the regular POST tests plus: – Extended DRAM test 3.
could be part of more than one group. These commands are discussed in more detail in 8.6, “Useful IOS CLI troubleshooting commands” on page 212.
The following list provides some tips for using the CLI: Left/right arrow keys Up/down arrow keys Tab key Backspace key Spacebar Enter key ? Ctrl+B Ctrl+F Ctrl+A Ctrl+E Esc+B Esc+F Ctrl+P Ctrl+N Ctrl+D Ctrl+W Ctrl+l Ctrl+R Move left or right one character on command line Scroll through command history Complete a command Delete previous character Scroll a page at a time Scroll a line at a time Help Move one character back Move one character forward Move to the beginnin
This shows the configuration running in memory, and the command show startup shows the configuration as stored in NVRAM. To synchronize the information in show running with the information in NVRAM (in other words, to save the running config into NVRAM) use write mem or copy running startup. show vlan Verifies that desired VLANs exist. If a VLAN does not exist in here, the switch will not carry data for that VLAN even if a port is configured to use it.
show show show show show show show show show show show show show flash: all process memory process cpu vlan clock etherchannel summary int trunk cdp neighbors spanning-tree summary mac-address-table count log region buffers Some helpful items that are not in show tech-support: show platform summary show int status show span blocked show platform summary show platform summary is important for showing data that no other IGESM command can show, such as that certain options a
8.6.2 Administrative term monitor (term no monitor) Redirects console output to current terminal emulation session. If Telneted in, will not see important console messages unless this command is run first (must run every time you Telnet in if you want to see con messages). clear counters Clears interface counters, more easily monitor counters from a given point in time. clear log Clears out old log messages so you can start fresh.
Sending 10, 1000-byte ICMP Echos to 172.26.146.1, timeout is 1 seconds: !!!!!!!!!! Success rate is 100 percent (10/10), round-trip min/avg/max = 4/4/4 ms show cdp neighbor show cdp neighbor is a powerful troubleshooting command. At a minimum, it tells what ports are connected on each side of a link. It can also tell the type of device on the other side of a link, the IOS version on the other side of the link, and the IP address of the devices on the other side of the link.
show int status This command offers a snapshot of connection status on the IGESM (Figure 8-6 ).
shut - no shut (interface config mode) Used to administratively shut down or bring up an interface. Run from interface config mode. If Telneted in, use term mon to see port up/down messages. Use show int status to see whether port is administratively shut down. shut - no shut is very handy for clearing ports in err-disable state. (Exception: If using the trunk failover feature, shut - no shut is not the tool to clear err-disabled.
show int trunk Figure 8-9 shows some of the attributes of the show int trunk command. • Lets you know what VLANs can be carried and are being carried on trunk ports cigesm_t#sh int trunk Port Gi0/15 Po1 Po2 Native VLAN in use on trunk link Mode Encapsulation Status Native vlan on 802.1q trunking 30 on 802.1q trunking 1 on 802.
show spanning-tree blockedports This command is important for showing what ports are blocked and which are forwarding before a problem starts. Should be predictable under all link up/down conditions.
Upstream Interfaces : Downstream Interfaces : (Up):Interface up (Dwn):Interface Down (Dis):Interface disabled Make sure upstream interfaces include the expected interfaces. Upstream interfaces should show all up if everything is working correctly. Make sure downstream interfaces include the expected interfaces (may be all interfaces, depending on desired config). Any configured downstream interfaces should show Up. Downstream ports will go to Dis when all configured upstream ports for the group go dow.
debug A vast number of debug commands are available with IOS that are used to monitor various activities within the IGESM. Warning: Use debug commands with extreme care. Debug is recommended for use only by experienced administrators. Incorrect use can lead to unexpected and undesired operation of the IGESM, and can disrupt the flow of traffic to and through the switch, which can result in a network down condition. Do not use unless you understand its consequences. Do not use debug all.
9 Chapter 9. Service and support Support for the Cisco Systems Intelligent Gigabit Ethernet Switch Module is provided to our customers using the following methods. © Copyright IBM Corp. 2004, 2005. All rights reserved.
9.1 Placing the call to IBM For U.S., AP, CAN, and EMEA: Use one of the following numbers when calling IBM for technical support: Within the United States, call the IBM Support Center at 1-800-IBM-SERV (426-7378). Within Canada: – For support, call HelpPC at 800-426-7378. – For more information or to place an order, call 800-465-7999. Outside the United States and Canada, contact your IBM HelpWare® number, your place of purchase, or your local IBM office.
9.4 Other support sites Listed here are other helpful Web sites (these may require a Cisco user name and password): TAC Main Support page http://www.cisco.com/en/US/partner/support/index.html TAC Service Request Tool http://www.cisco.com/cgi-bin/front.x/case_tools/caseOpen.pl SVO Submit http://www.cisco.com/cgi-bin/front.x/agents/svo_tools/SVOToolDispatcher Cisco CCO – Online documentation http://www.cisco.com/univercd/home/home.htm Cisco TAC - Catalyst Switch Best Practices http://www.cisco.
226 Cisco Systems Intelligent Gigabit Ethernet Switch Module
A Appendix A. Hints and tips In this section, we provide hints and tips that may prove useful during the setup, configuration, and operation of your Cisco Systems Intelligent Gigabit Ethernet Switch Module for the IBM Eserver BladeCenter. As noted elsewhere in this document, the information herein applies to the 4-port copper-based IGESM running a 12.1(14) version of IOS. If working with the 4-port SFP-based IGESM or a 4-port copper-based IGESM running 12.
drivers necessary for supporting the NICs on a blade server are not part of a standard Windows 2000 install, and the NICs will be generically listed in Windows 2000 Device Manager as two or more Ethernet Controllers (with a question mark next to them) until the necessary drivers are loaded. For these NICs to become active, a third-party driver, supplied by IBM, must be installed.
solution for your environment, which will have its own unique requirements. It is only used in these examples for simplicity. Microsoft has published Knowledge Base article 157025 that discusses the different approaches for default gateways on multihomed systems. This article can be found at: http://support.microsoft.com/default.
The most common cause is trying to change the management IP address for the Cisco Systems IGESM directly on the Cisco Systems IGESM (either through CLI or through CMS).
Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 192.168.70.126, timeout is 2 seconds: ..... Success rate is 0 percent (0/5) The cause of this issue is related to the duplicate IP address message reported in Duplicate IP address: part 1 (the Management Module responding to ARP requests for addresses on its own internal subnet). The way to prevent this situation from occurring is to change only the IP address of the management VLAN through the Management Module’s Web interface.
Figure 9-1 BASC example of a window without an option to cancel Cisco Systems IGESM stuck at switch: prompt Any keystrokes received through the serial console connection during the early phases of the boot-up process of the Cisco Systems IGESM may be interpreted as a break signal and may put the Cisco Systems IGESM into an incomplete boot-up state, with a prompt that simply says switch:.
Key sequence to switch between blade servers The BladeCenter has a KVM (keyboard/video/monitor) switch built into the Management Modules, allowing traditional access to the installed blade servers. To switch the keyboard, mouse, and monitor between blade servers, perform this keystroke combination from the keyboard attached to the active Management Module: NumLock NumLock Enter Where is the number of the blade server bay where the blade server is installed.
condition (in the config term mode, run the command no monitor session x, where x is the monitor session number configured for RSPAN use). Important: We recommend extreme caution when using the RSPAN feature on the Cisco Systems IGESM if you are not using 12.1(14)AY1 or a later revision of the code.
The option for Fast POST in the Management Module Advanced Setup for I/O Modules only affects how thoroughly the diagnostics are run during POST, which in turn affects how fast the switch boots but does not ultimately affect any management or data paths on the IGESM.
Switch#sh platform summary Platform Summary: •Switch is in Bay 1 Switch Slot: 1 Current IP Addr: 172.26.147.208, 255.255.254.0, gw: 172.26.146.1 Default IP Addr: 10.10.10.91, 255.255.255.0, gw: 0.0.0.0 IP Fields read from VPD: 172.26.147.208, 255.255.254.0, gw: 172.26.146.1 Static IP Fields in VPD: 172.26.147.208 255.255.254.0 172.26.146.
• One version that has been known to not exhibit this issue (although it has the port down issue previously discussed) is tg3.c:v3.6RH (June 12, 2004). Note that after installing a working tg3 driver you will still have to perform a shut and no shut on the interface on the IGESM that was placed in err-disable by the faulty tg3 driver, to bring the interface back up.
2. Log on to IGESM, change IP information to desired information. 3. Save IGESM config to NVRAM (write mem). 4. Reload IGESM. After the IGESM is reloaded it will be in control of its own IP information in the event of a IGESM reload, but only if the IGESM reloads. If the Management Module reloads, it will push its IGESM IP addressing information back onto the IGESM.
Related publications The publications listed in this section are considered particularly suitable for a more detailed discussion of the topics covered in this Redpaper. IBM Redbooks For information about ordering these publications, see “How to get IBM Redbooks” on page 242. Some of the documents referenced here may be available only in softcopy.
Online resources These Web sites are also relevant as further information sources. (Some Cisco pages require your user name and password.) Whatis.com (definitions for thousands of the most current IT-related words) http://whatis.techtarget.com/ IBM Eserver BladeCenter http://www.ibm.com/servers/eserver/bladecenter/index.html IBM Eserver BladeCenter support http://www.ibm.com/servers/eserver/support/bladecenter/index.html IBM Eserver Storage http://www.pc.ibm.com/us/eserver/xseries/storage.
Broadcom Advanced Server Program (BASP) driver V6.2.1 for Linux http://www.ibm.com/pc/support/site.wss/document.do?lndocid=MIGR-54186 Switch Management Interface and Native VLAN in the Best Practices document http://www.cisco.com/en/US/partner/products/hw/switches/ps700/products_white_paper09186a 00801b49a4.shtml Cisco Business Ready Data Center http://www.cisco.com/go/datacenter 6500 IOS Best Practices guide http://www.cisco.
VLAN security best practices http://www.cisco.com/en/US/products/hw/switches/ps708/products_white_paper09186a00801315 9f.shtml IBM/Cisco Systems IGESM IOS Code Download http://www.ibm.com/pc/support/site.wss/document.do?lndocid=MIGR-58132 CiscoWorks IDUs to support the Cisco Systems IGESM Version 10 and above IDUs support Cisco Systems IGESM Minimum code on Cisco Systems IGESM to support CiscoWorks is 12.1(14)AY1 http://www.cisco.com/kobayashi/sw-center/cw2000/lan-planner.
Abbreviations and acronyms 802.3 10BASE-T Ethernet 802.3ad Link Aggregation 802.1D IOS Cisco Internetworking Operating System Spanning Tree Protocol IP Internet Protocol 802.1p Class of Service (CoS) IP DSCP IP Differentiated Services Code Point 802.1Q Trunking Protocol ISL Cisco Inter-Switch Link 802.1s Multiple Spanning Tree Protocol ISO 802.
UTP Unshielded Twisted Pair VLAN Virtual Local Area Network VMPS VLAN Membership Policy Server VTP VLAN Trunking Protocol 244 Cisco Systems Intelligent Gigabit Ethernet Switch Module
Index Numerics 1000BASE-T 14, 17 100BASE-TX 17 100-ohm STP 17 10BASE-T 17 1800 watt power supplies 106 64-bit computing 4 6500 106–109, 126 6509 107 802.1D 1, 14 802.1Q 1, 15, 118, 143 802.1Q trunk 118, 124, 138 802.1s 1, 14 802.1w 1, 14 802.1X 15 802.
CLI command 25, 39 CLI command modes 26 CLI-based sessions 13 cluster 31 Cluster Management Suite See CMS Cluster Management Suite GUI 24 cluster menu 31 CMS 13, 15, 24, 29–30, 45, 99, 143 CMS Front Panel View 30, 34 CMS menu 30 collaboration 3 color-coding 53 command-line interface See CLI commands 28 console baud rate 31 cross-over cable 81, 117 cryptographic software image 13 D data center 99 database applications 4 daughter card 7–8 DDR-SDRAM memory channel 8 Device Configuration Manager 53 device menu
Hot Standby Router Protocol (HSRP) 109, 243 hot-pluggable module 100 HP OpenView 54 HS20 4, 83, 88–89, 106 HS20 architecture 8 HS40 4 HTTP port 31 HTTP Web interface 81 hybrid mode 117 I I/O buses 8 I/O module 84 I2C 9 I2C bus 9, 40 IBM Director 5, 25, 54, 81 IBM Integrated System Management Processor 9 IBM on demand operating environment 2 IBM TotalStorage 5 IBM UpdateXpress 89 IDE channel 9 IEEE 802.1d Spanning Tree Protocol 17 IEEE 802.1D Spanning Tree Protocol (STP) 14 IEEE 802.
Network Time Protocol (NTP) 13, 243 NIC 83 NIC teaming 95, 231 NVRAM 121 O out-of-band management 40, 66, 83 P PCI Bus 9 Per-VLAN Spanning Tree (PVST) 1, 14 ping and trace 34 ping dialog 39 Policy Feature Card 3 107 Port Aggregation Protocol (PAgP) 1, 14, 118, 243 port menu 32 port pop-up menu 30 port search 32 port security 15, 32 port security aging 15 port security option 15 port settings 32 port statistics 33 port switch 16 POST 84 POST/BIOS code 9 Preboot Execution Environment (PXE) 54 preserve new I
T W TACACS+ 1 Tape Drive Management Assistant 5 Telnet 12–13, 24–25, 39, 86–87 Telnet client 9 Telnet session 39, 87 Terminal Access Controller Access Control System Plus (TACACS+) 15 terminal emulation 24 TFTP 86 TFTP server 86 thin IMB bus 8 ThinkPad 5 time-out events 16 Tivoli 2, 54 toolbar buttons 30 tools menu 34 traceroute 16 traffic analysis 16 Trivial File Transfer Protocol (TFTP) 13 troubleshooting 28 trunk 102, 118 trunking 118 Web browser 25 Web server 4 weighted round-robin (WRR) 16 window me
250 Cisco Systems Intelligent Gigabit Ethernet Switch Module
Back cover Cisco Systems Intelligent Gigabit Ethernet Switch Module for IBM Eserver BladeCenter Copper Ethernet switching technology integrated into the BladeCenter chassis This IBM Redpaper positions the Cisco Systems Intelligent Gigabit Ethernet Switch Module for the IBM Eserver BladeCenter and describes how it enhances the BladeCenter value proposition by seamlessly interfacing into a customer’s existing data network.
