Cisco SFS 3012 Multifabric Server Switch Hardware Guide (Topspin 360 Hardware Guide) Web Release v2.1.
Copyright © 2004 - 2005 Topspin Communications, Inc. All rights reserved. The Topspin Server Switch, Topspin Host Channel Adapter, Topspin Element Manager, and collateral software programs and documentation are subject to and made available only pursuant to the license agreement signed by you and Topspin, Communications, Inc. (or if no signed license agreement exists, the license agreement included with your original media) and may only be used in accordance with the terms of that agreement.
iii Contents Regulatory Notices........................................ vii FCC Statement .......................................................................................................................................... vii Safety Information .................................................................................................................................... vii Electrical Cautions .....................................................................................................
iv Introduction..................................................................................................................................................5 Controller Cards...........................................................................................................................................6 Install a Controller Card ................................................................................................................. 6 Ethernet and Fibre Channel Gateways...............
v Installing a Power Supply ............................................................................................................. 44 Removing a Power Supply ........................................................................................................... 45 Replacing Blower Trays ............................................................................................................................46 Installing a Blower Tray .............................................................
vi
vii Regulatory Notices FCC Statement This equipment has been tested and found to comply with the limits for a Class A digital device, pursuant to Part 15 of the FCC Rules. These limits are designed to provide reasonable protection against harmful interference when the equipment is operated in a commercial environment.
viii CAUTION: Grounding is supplied by the ground-prong on the 3-prong power cable. Do not attach a separate ground cable. Do not use adapter plugs. Do not remove the ground prong from the cable. Ensure the ground connection on the power supply is correct and functioning before applying power to the Topspin 360. CAUTION: Always ground yourself before touching any internal Topspin 360 component to avoid damage from electrostatic discharge (ESD). CAUTION: The Topspin 360 has two power cables.
ix CAUTION: Use only Topspin-approved gateway expansion modules, controller cards, fans, and power supply modules. CAUTION: Never place your hand inside an empty card or module bay. You should never have cause to place a hand anywhere inside the Topspin chassis. Unused card and module bays should always have a Topspin cover over the bay to ensure proper safety, ventilation, and cooling.
x CAUTION: Only open, modify, or service laser devices as specified in the manufacturer documentation. CAUTION: Use of controls or adjustments or performance of procedures other than those specified herein may result in hazardous laser radiation exposure. Laser Specs Table iii-1: Topspin P/N Application Optical Parameters 48-00001-01 Fibre Channel *CDRH/IEC Laser Class: 1/1 Maximum Data Rate: 2.125 GB/s Wavelength: 850nm I/O connector: * Standards used: CDRH - 21CFR Chapter I §1040.
xi Product Registration THANK YOU FOR PURCHASING YOUR TOPSPIN® PRODUCT! Topspin Communications is pleased that you have selected Topspin for your computing infrastructure. Topspin provides comprehensive customer support resources via telephone and Web access. Topspin’s Customer Support Web site has the latest TopspinOS (IB Switch and I/O Chassis Operating System), Linux and Windows Host Side Drivers, Boot over InfiniBand Firmware and technical documentation.
xii • Phone: (800) 499-1473 • Online Help Desk: http://support.topspin.
xiii Standard Product Agreement PLEASE READ THESE TERMS AND CONDITIONS CAREFULLY BEFORE USING THE PRODUCTS THAT ARE PROVIDED WITH THIS AGREEMENT. BY USING ANY PRODUCTS, YOU ACKNOWLEDGE THAT YOU HAVE READ AND UNDERSTOOD ALL THESE TERMS AND CONDITIONS AND YOU WILL BE CONSENTING TO BE BOUND BY THEM. IF YOU DO NOT ACCEPT THESE TERMS AND CONDITIONS, DO NOT USE THE PRODUCTS AND RETURN PRODUCTS WITHIN 30 DAYS UNUSED IN THE ORIGINAL SHIPPING CONTAINER TO THE PLACE OF PURCHASE FOR A FULL REFUND. 1. General.
xiv 3. Payment/Financing. Topspin will determine Customer's credit terms on a per-order basis and such terms are subject to a credit review and approval by Topspin in its sole discretion. The amount of credit may be changed or credit withdrawn by Topspin at anytime in its sole discretion. All payments shall be made in U.S. Dollars net thirty (30) days from the date of invoice unless otherwise stated by Topspin in writing.
xv 6. Intellectual Property Rights. Title to and ownership of the Software and any associated documentation, and any improved, updated, modified or additional parts thereof, and all copyright, patent, trade secret, trademark and other intellectual property rights embodied in the Products, shall at all times remain the property of Topspin or Topspin's licensors.
xvi HAVE OTHER RIGHTS, WHICH VARY FROM STATE/JURISDICTION TO STATE/JURISDICTION. 10. Confidentiality. “Confidential Information” means any oral, written, graphic or machine-readable information disclosed by Topspin that is designated in writing to be confidential or proprietary or should be reasonably understood to be confidential.
xvii acquisition or sale of all or substantially all of assets relating to this Agreement. Topspin or any successor may assign all of part of the right to payments under this Agreement. Any assignment or transfer of this Agreement made in contravention of the terms hereof shall be null and void. Subject to the foregoing, the terms and conditions of this Agreement shall be binding upon and inure to the benefit of the parties to it and their respective heirs, successors, assigns and legal representatives.
xviii
1 1 Topspin 360 Overview The following sections appear in this chapter: • “Introduction” on page 1 • “Features” on page 2 Introduction The Topspin 360 Server Switch provides data center managers with a high-performance, low-latency interconnect that runs Fibre Channel and Ethernet traffic transparently over an InfiniBand (IB) backplane. Figure 1-1 displays the front of the Topspin 360 Server Switch (with the bezel on).
2 Figure 1-2 displays the back of the Topspin 360 Server Switch and numbers the slots on the chassis. 3 4 5 6 7 8 9 2 10 11 12 13 14 1 15 16 Figure 1-2: Topspin 360 Chassis - Back View (Service End) with Slot Numbers The vertical slots at the top of the service end of the Topspin 360 are numbered 1 to 14 from left to right. The horizontal slots at the bottom of the service end of the chassis are numbered 15 to 16 from top to bottom.
3 Redundancy and High-Availability The Topspin 360 is architected for full redundancy within a single box. Every component in the data path can be dual-connected from end to end. For example, a server with a 2-port HCA can be dual-attached to the two switch blades on the Topspin 360. Each gateway is also dual attached to the dual switch blades. In turn, the gateway modules are dual attached to Ethernet or Fibre Channel networks in pairs.
4
5 2 Topspin 360 Components The following sections appear in this chapter: • “Introduction” on page 5 • “Controller Cards” on page 6 • “Ethernet and Fibre Channel Gateways” on page 9 • “InfiniBand Switch Cards” on page 10 • “Power Supply Modules” on page 13 • “Blower Trays” on page 15 Introduction The components of the Topspin 360 support the scalability and high availability of the Server Switch.
6 CAUTION: Never place your hand inside an empty card or module bay. You should never have cause to place a hand anywhere inside the Topspin 360 chassis. Unused card and module bays should always have a Topspin 360 blanking panel over the bay to ensure proper safety, ventilation, and cooling. Controller Cards Controller cards manage the Topspin 360 and provide Ethernet and Serial Console port access to the Server Switch. You can install controller cards in slots 1 and 14 of the Topspin 360.
7 3. Remove the controller card from any packaging. 4. Extend the ejector lever on the card completely. 5. Insert the controller card module into slot 1 or slot 14 such that the ejector lever sits on the bottom-left-hand side of the module. 6. Push the controller card firmly into the slot. The ejector lever begins to close.
8 7. Push the ejector lever up until it clicks into place.
9 Ethernet and Fibre Channel Gateways Topspin Ethernet and Fibre Channel gateways connect the Topspin 360, and IB-connected hosts, to IP and FC networks. All gateways are dual-connected to the redundant InfiniBand backplane of the Topspin 360. Figure 2-4 displays an Ethernet gateway. Figure 2-5 displays a Fibre Channel gateway.
10 Install an Ethernet or Fibre Channel Gateway To install a gateway, perform the following steps: 1. Ground yourself appropriately. 2. Remove the blanking panel if one resides in the slot. 3. Remove the gateway module from any packaging. 4. Extend the ejector lever on the gateway completely. 5. Insert the gateway into an expansion module slot such that the ejector lever sits on the bottom-left-hand side of the module. NOTE: You can install gateways in slots 2 - 13.
11 Install InfiniBand Switch Cards To install a switch card, perform the following steps: 1. Ground yourself appropriately. 2. Remove the blanking panel if one resides in the slot with a #1 Phillips-head screwdriver. 3. Remove the switch card from any packaging. 4. Extend the ejector levers on the left- and right-hand sides of the switch card completely. 5. Insert the switch card into slot 15 or slot 16 such that the ejector levers sit on the bottom of the switch card. 6.
12 7. Close both fastener-levers simultaneously and completely.
13 8. Tighten the screws on either side of the switch card to secure it to the chassis. Figure 2-9: Securing Switch Card Remove InfiniBand Switch Cards To remove an InfiniBand switch card from the Topspin 360 chassis, perform the following steps: 1. Make sure the Topspin 360 chassis is secure. 2. If you are removing the InfiniBand card in slot 16 (the bottom card), you must power down the chassis before removing the card. The InfiniBand card in slot 16 is not hot-swappable.
14 power supply, you can add a second power supply while the chassis runs. If you have two power supplies installed, you can remove one of them without removing power from the chassis. Figure 2-10 displays the location of the power supplies in the Topspin 360. Power Supplies Figure 2-10: Power Supply Modules Install Power Supply Modules To insert a power supply, perform the following steps: 1. Ground yourself appropriately. 2. Remove the blanking panel if one resides in the slot. 3.
15 NOTE: The power socket on the back (service end) of the Topspin 360 corresponds to the power supply module on the same side of the Server Switch. Removing a Power Supply To remove the power supply, perform the following steps: 1. Remove the power cord from the power supply. 2. Disengage the power supply by pulling it toward you. 3. Slide the power supply out of the chassis. 4. Fasten a blanking panel to the empty chassis bay to ensure appropriate cooling.
16 3. Insert the blower tray into an open slot until it is fully seated, such that the captive screw sits at the top of the tray. 4. Tighten the captive screw on the front panel to secure the tray to the chassis. Removing Blower Trays To remove a blower tray, perform the following steps: 1. Ground yourself appropriately. 2. Unfasten the captive screw. 3. Pull the fan toward you until it completely disengages from the chassis.
17 4. Insert a replacement blower tray into the empty chassis bay to ensure appropriate cooling. NOTE: Each blower module contains two fans for a total of four fans. The system can tolerate one out of four blowers failing. When you remove a blower module, you have three minutes to replace it with a fully-functioning blower module.
18
19 3 Install and Boot the Topspin 360 The following sections appear in this chapter: • “Introduction” on page 19 • “Prepare the Site for the Topspin 360” on page 19 • “Install Topspin 360 Components” on page 20 • “Rack the Topspin 360” on page 20 • “Attach a Serial Console Cable to a PC or Terminal” on page 26 • “Boot your Topspin 360 and Configure Basic Connectivity” on page 27 • “Connect InfiniBand Hosts” on page 29 • “Manage the Topspin 360” on page 32 • “Viewing and Updating the Softwa
20 Read the Cautionary Statements Refer to the“FCC Statement” on page vii and the “Safety Information” on page vii before you install your Topspin 360. Prepare the Physical Environment for the Topspin 360 • Ground yourself using an approved ground wrist-strap. • Make sure you have the right cables and sufficient ventilation. • Unpack the Topspin 360 package. CAUTION: Do not lift the Topspin 360 by the front and back (bezel end and service end). Lift the unit by the sides to prevent damage.
21 7. Attach the side flanges to the sides of the chassis with the screws provided, as shown Figure 3-1 and Figure 3-2. Figure 3-1: Positioning the Side Flanges Figure 3-2: Fastening the Side Flanges 8. If necessary for your rack, add fasteners with appropriate threads to your rack to prepare your rack for the rails.
22 9. Position the chassis brackets to the rack rails, as shown in Figure 3-3.
23 10. Screw the top and bottom of the front of the rails to the rack but do not fully tighten the screws. Figure 3-4 displays the rail loosely connected to the rack. NOTE: When you connect the rails, do not fully tighten the screws. When you slide the chassis into the rails, you may need to reposition the rails to align the flange screws with the rails.
24 11. Screw the top and bottom of the back of the rails to the rack, as shown in Figure 3-5.
25 12. Slide the Topspin 360 onto the rails as shown in Figure 3-6 and Figure 3-7.
26 13. Adjust the front of the rails (that you left loose in step 10) so that the threaded hole in the center of the front of the rail aligns with the screw in the flange. 14. Screw the flange screws tightly into the rails, as shown in Figure 3-8. Figure 3-8: Securing the Chassis to the Rails 15. Firmly tighten the front rail screws. For the VFrame Evaluation Test Plan, continue to step b of “Hardware Installation Steps” on page 13 in the “VFrame Evaluation Test Plan”.
27 • Stop Bits: 1 • Flow control: None For the VFrame Evaluation Test Plan, continue to Step b of “Hardware Configuration Steps” on page 14 in the “VFrame Evaluation Test Plan”. Boot your Topspin 360 and Configure Basic Connectivity To configure basic connectivity, you must power-on the Server Switch and configure it with the Serial Console port and a terminal or workstation.
28 3. Enter the interface mgmt-ethernet command. Example Topspin-360(config)# interface mgmt-ethernet 4. Enter the IP address of the management port followed by the net mask. Example Topspin-360(config-if-mgmt-ethernet)# ip address 10.10.0.22 255.255.255.0 5. Set the default gateway of the management port. Example Topspin-360(config-if-mgmt-ethernet)# gateway 10.10.0.1 6. Enable the management port with the no shutdown command.
29 5. Enable the management port with the no shutdown command. You can now telnet to your Topspin 360, or enter the IP address of your Topspin 360 in a Web browser to launch Chassis Manager. Example Topspin-360(config-if-mgmt-ethernet)# no shutdown 6. Save the configuration to preserve it between reboots.
30 Use InfiniBand cables to connect the HCA in your host to the InfiniBand switch card of your Topspin 360. To plug in an InfiniBand cable, push the connector into the interface until you hear/feel a click. Figure 3-9: Fully Installed IB Cable with Pinch Connector Figure 3-10: Fully Installed IB Cable with Pull Connector NOTE: If your host does not provide an ample amount of free space around a given IB port, double-check that your IB cable connector engages fully.
31 To remove a cable with a pinch connector, pinch both sides of the back of the connector and pull the connector away from the port.
32 To remove a cable with a pull connector, grasp the connector with one hand and push it toward the port, then pull the latch away from the port with your other hand and gently wiggle the connector away from the port. 1. push connector toward port 2. pull latch away from port 3.
33 Viewing and Updating the Software Image on the Chassis View the Software Image Version with the CLI 1. To view the primary software image version for all cards in the chassis, use the show card-inventory command. Look for the primary-image-source field for each slot. (The slots are described in Figure 1-2.) 2. Note that it is possible to have different versions of software installed on different slots.
34 An installed image file will have a slash (/) in the name. Example Topspin-360> enable Topspin-360# dir image ============================================================================= Existing Boot-Images on System ============================================================================= slot date-created size file-name -------------------------------------------------------------------------------1 Tue Nov 25 18:46:32 2003 30695264 Topspin360-TopspinOS-2.0.0-build212.
35 3. View the software version on individual cards in the chassis: a. Double-click on any card in the main GUI. b. Click the Inventory tab. The properties window appears for that card. Figure 3-14: View Current Image of an Individual Card c. View the Current Image Source field. If you find a card that has a different image than the others, you can update a single card by re-installing the image on the chassis, but rebooting only the card that needs to be updated.
36 NOTE: If you do not have enough available memory to install an additional image, delete an old image. “Delete an Image with the CLI” on page 38. 3. Specify the new boot image to be used. a. Find the name of your new system image by entering the dir image command. Copy the name of the new image. b. Enter boot-config primary-image-source, then paste the copied image file name. Example Topspin-360# configure Topspin-360(config)# boot-config primary-image-source TopspinOS-2.1.
37 3. Click the Import button. The Import File window appears. Figure 3-15: Copy an Image File to the Switch a. Select Image from the File Type drop-down menu. b. Click the Remote FTP radio button from the Copy From section. c. Enter the name or IP address of the FTP server from which to copy the file in the Server Name or IP Address field. d. Enter the network name of a recognized user in the User Name field. e. Enter the password for the specified user in the Password field. f.
38 • All cards must have an oper-status of “Up” a. Click on the newly copied.img file in the File Management window. b. Click the Install button. c. Click Yes to the warning asking if you wish to proceed. NOTE: If you do not have enough available memory to install an additional image, delete an old image. “Delete an Image with the GUI” on page 39. 6. Specify the new boot image to be used. a. Select Maintenance-> Boot Config….The Boot Configuration window appears.
39 The following example deletes an image file from slot 1 (slots 1 and 14 can contain controller cards). Example Topspin-360# delete 1:image:Topspin360-TopspinOS-2.0.0-build488.img Delete file 14:Topspin360-TopspinOS-2.0.0-build488.img? [yes(default) | no] yes ****** Delete an Image with the GUI You can delete an .img file through the GUI or the CLI. A .img file is an image file that is not installed. Do not delete an installed image file. An installed image will have a slash (/) in the name. 1.
40
41 4 Topspin 360 Field Replaceable Units The following sections appear in this chapter: • “Controller Cards” on page 41 • “Expansion Modules” on page 41 • “InfiniBand Switch Cards” on page 42 • “Power Modules” on page 44 • “Replacing Blower Trays” on page 46 Controller Cards The Topspin 360 has two removable controllers, one on each side of the Server Switch. Expansion Modules When possible, expansion modules should be added before you rack the Topspin 360.
42 InfiniBand Switch Cards Each InfiniBand switch card is held in place by two screws, one on the left side and one on the right side of the card. Installing an InfiniBand Switch Card NOTE: Slot 16 must always have an Infiniband switch card inserted in order to maintain connectivity. To install the InfiniBand switch card in the Topspin 360 chassis: 1. Remove the blanking panel from the switch card bay, if it is still in place. 2. Insert the switch card into an open slot.
43 • The keyword all (The CLI defaults to “all” if you enter just “show card”).
44 3. Reboot the switch card. Example Topspin-360# reload System configuration has been modified. Save? [yes(default)/no/*.cfg] yes Proceed with reload? [confirm] Topspin-360# Removing an InfiniBand Switch Card To remove an InfiniBand switch card from the Topspin 360 chassis: 1. Make sure the Topspin 360 chassis is secure. 2. Properly ground yourself to avoid potential electrostatic discharge and damage to the card. 3.
45 NOTE: The power supplies should be inserted in opposite directions of each other. See Figure 4-2. Power Supply Figure 4-2: Power Supply Module Positioning 3. Press the power supply firmly into the chassis. Removing a Power Supply To remove the power supply, perform the following steps: 1. Remove the power cord from the power supply. 2. Disengage the power supply by pulling it toward you. 3. Slide the power supply out of the chassis. 4.
46 Replacing Blower Trays The Topspin 360 fan tray is hot-swappable. You do not need to turn off power to the chassis to replace the fan tray. Access the fan tray from the front of the Topspin 360 chassis. Installing a Blower Tray To insert a blower tray, perform the following steps: 1. If a blanking panel is already attached to the module bay, remove it. 2. Insert the blower tray into an open slot until it is fully seated.
47 3. Tighten the two captive panel screws on the front panel to secure the card to the chassis. Removing a Blower Tray To remove a blower tray, perform the following steps: 1. Unfasten the fan tray screws. 2. Pull the fan toward you until it completely disengages from the chassis. 3. Fasten a blanking panel to the empty chassis bay to ensure appropriate cooling. The blanking panel is a flat sheet of metal with the same dimensions as the fan tray faceplate.
48
49 5 Topspin 360 LEDs The following sections appear in this chapter: • “Introduction” on page 49 • “Topspin 360 System LED” on page 50 • “Power Supply LEDs” on page 51 • “Blower Tray LEDs” on page 52 • “Controller Card LEDs” on page 53 • “IB Switch Card LEDs” on page 54 • “IB Port LEDs” on page 54 • “Fibre Channel Gateway LEDs” on page 55 • “FC Port LEDs” on page 55 • “Ethernet Gateway LEDs” on page 56 • “Ethernet Port LEDs” on page 56 Introduction LEDs on your Topspin 360 and install
50 • Ethernet cards • Topspin 360 bezel side The LEDs have special significance depending on the chassis component. This section describes the LED indicators of each component Topspin 360 System LED The Topspin 360 system LED appears on the top-left-hand corner of the front (bezel end) of the Server Switch. You can view the system LED with the bezel on or off. Table 5-1 describes system LED indications.
51 Power Supply LEDs Power supply status LEDs are located on either side of the power supply module pull handle. Controller cards mirror Power Supply LEDs on the service end of the chassis. Figure 5-1 displays power supply LEDs. DC AC Figure 5-1: Power Supply LEDs Table 5-2 displays AC LED indications. Table 5-2: AC Power Supply LED Activity Indication solid green AC input present. off AC input absent. Table 5-3 displays DC LED indications.
52 Blower Tray LEDs Blower tray LEDs appear immediately to the left of the System LED on the bezel end of the Topspin 360. Each blower LED is a dual-color green/yellow LED. Controller Cards mirror the blower LEDs. Figure 5-2 displays the blower LEDs. Figure 5-2: Blower Tray LEDs Table 5-4 explains blower tray LED indications. Table 5-4: Blower LED Indications Activity Indication solid green Blower runs with no errors. solid yellow A blower has failed. off No system power.
53 Controller Card LEDs Controller cards include card status LEDs as well as Management Ethernet port LEDs. The top-left-most LED represents the controller card in slot 1, and the top-right-most LED represents the controller card in port 14. Figure 5-3 displays the LEDs on the controller card.
54 IB Switch Card LEDs Each InfiniBand switch card includes two LEDs (one green and one yellow) at one side of the interface-end of the card. Figure 5-4 displays the IB switch card LEDs. Figure 5-4: InfiniBand Switch Card LEDs Table 5-7 explains IB switch card LED indications. Table 5-7: InfiniBand Switch Card LED Indications LED State Indication green off, yellow off No power to the switch card. green solid, yellow off Switch card runs with no errors.
55 Fibre Channel Gateway LEDs The FC gateway provides one yellow (amber) LED and one green LED. Table 5-9 lists and describes the LED indications. Table 5-9: Gateway LED Indications Display Indication yellow on, green off Auxiliary power has reached the gateway. green on, yellow off Gateway runs successfully. green on and yellow on An error has occurred FC Port LEDs Fibre Channel Port LEDs are adjacent to each FC port. They indicate port status and activity.
56 Table 5-11 explains the FC port activity LED. Table 5-11: Interpreting the Fibre Channel Port Activity LEDs Color Description off No system power or LED failure. solid yellow Fan error detected. Operator intervention required. solid green Fans running with no detected errors. Ethernet Gateway LEDs The Ethernet gateway provides one yellow (amber) LED and one green LED. Table 5-12 lists and describes the LED indications.
57 6 Specifications and Compliance This chapter details the Topspin 360 specifications and compliance certifications. General Specifications Table 6-1: General Specifications Specification Detail Operating Temperature Range 0 to 35C Non-Operating Temperature Range -40 to 65 degrees Celsius Operating Temperature Gradient 20C maximum per 60 min Operating Altitude 0 - 10,000 ft Non-Operating Alt.
58 Table 6-1: General Specifications Specification Detail Non-Operating Vibration, Sinusoidal 3 axis with the following parameters • “X and Y axis: 0.0004 G 2 / Hz flat from 5-500Hz (0.472 GRMS) • Z-axis: 0.0008 G 2 / Hz flat from 5-500Hz (0.
59 Table 6-4: Laser Specifications Topspin P/N Application Optical Parameters *Standards used: CDRH - 21CFR Chapter I§ 1040.
60
Index A T architecture ..............................................................2 transparent topology ...............................................2 B basic connectivity .................................................27 C configure basic connectivity .................................27 connectivity, basic ................................................27 controller, installing & removing ..........................41 H high availability ......................................................
