InfiniScale® IV 36-Port QSFP 40Gb/s InfiniBand Switch User Manual P/N:MIS5024Q-1BFR, MIS5024Q-1BRR Rev 1.2 www.mellanox.
Rev 1.2 NOTE: THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT (“PRODUCT(S)”) AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES “AS-IS” WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUSTOMER'S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO(S) AND/OR THE SYSTEM USING IT.
QDR InfiniBand Switch Platform User Manual Rev 1.2 Contents Contents 3 List of Figures 5 List of Tables 6 Revision History 7 About this Manual 8 Intended Audience Related Documentation Conventions Mellanox Part Numbering Legend Chapter 1 8 8 8 9 Overview 10 1.1 Serial Number and Product Version Information 10 Chapter 2 Installation Safety Warnings 11 Chapter 3 Externally Managed Switches 14 Chapter 4 Hardware Basic Operation and Installation 15 4.
Rev 1.
QDR InfiniBand Switch Platform User Manual Rev 1.
Rev 1.
QDR InfiniBand Switch Platform User Manual Rev 1.2 Revision History Table 1 - Revision History Table Date Revision Description May 2011 Rev 1.2 Changes to the firmware update procedure Jan. 2011 Rev 1.1 Minor fixes to graphics reordered Interface Port Led Table Table 5 changes Orange to Flashing Orange No UID LED support for end users. Jan. 2011 Rev 1.
Rev 1.2 About this Manual This manual describes the installation and basic use of the Mellanox IS5024 switch, which is based on the InfiniScale® IV InfiniBand switch device. Intended Audience This manual is intended for users and system administrators responsible for installing and setting up the switch platforms listed above. The manual assumes familiarity with the InfiniBand® Architecture Specification.
QDR InfiniBand Switch Platform User Manual Rev 1.2 This symbol indicates information that is helpful to the user. BEWARE! This symbol indicates a situation that can potentially cause personal injury or damage to hardware or software.
Rev 1.2 1 Overview Overview This User Manual provides an overview of the InfiniBand switch product, a 36-port switch capable of up to 2.88Tb/s of non-blocking switching capacity. Based on the fourth generation InfiniScale IV InfiniBand switch device. The platform delivers industry leading bandwidth, latency, and scalability in a 1U form factor. The platform can be tiered to build larger multi stage networks in a variety of topologies such as Fat-Tree, Mesh, 3D Torus etc.
QDR InfiniBand Switch Platform User Manual 2 Rev 1.2 Installation Safety Warnings Warnings in French can be found on page 40. Warnings in German can be found on page 42. Warnings in Spanish can be found on page 44. 1. Installation Instructions Read all installation instructions before connecting the equipment to the power source. 2. Bodily Injury Due to Weight Use enough people to safely lift this product. <40 lbs <18 kgs 40 - 70 lbs 18 - 32 kgs 70 - 121 lbs 32 - 55 kgs >121 lbs >55 kgs 3.
Rev 1.2 Installation Safety Warnings 6. Copper InfiniBand Cable Connecting/Disconnecting Copper InfiniBand cables are heavy and not flexible, as such they should be carefully attached to or detached from the connectors. Refer to the cable manufacturer for special warnings/instructions. 7. Rack Mounting and Servicing When this product is mounted or serviced in a rack, special precautions must be taken to ensure that the system remains stable.
QDR InfiniBand Switch Platform User Manual Rev 1.2 12. Add GND connection information Before connecting this device to the power line, the protective earth terminal screws of this device must be connected to the protective earth in the building installation. (GND Connection Information): The building installation shall provide a means for connection to protective earth; and the equipment shall to be connected to that means permanently by a service person.
Rev 1.2 3 Externally Managed Switches Externally Managed Switches Externally managed (unmanaged) switches are plug and play out of the box. All switches come with the latest firmware (FW) burned on the Flash. Updating the FW on externally managed switches can be performed in-band only. When new FW is available, an email notification is sent to registered users with the link to the Mellanox FW download site. The download site has the Mellanox FW tool package and full instructions for updating FW.
QDR InfiniBand Switch Platform User Manual Rev 1.2 4 Hardware Basic Operation and Installation 4.1 Switch Platform Hardware Overview Figure 2 shows the front view of the switch. The figure shows port configurations for the switch systems, I2C connector, and various status LEDs. Externally Managed switches come with an I2C RJ45 connector.
Rev 1.2 Hardware Basic Operation and Installation Figure 3: Status LEDs UID Mellanox ® If the Status LED shows red after 30 seconds unplug the switch and call your Mellanox representative for assistance. If the Fan LED shows red unplug the switch and call your Mellanox representative for assistance. If the switch shuts down due to over temperature, unplug the switch, wait 5 minutes and replug in the switch. For more information See “Troubleshooting” on page 29. 4.1.1.
QDR InfiniBand Switch Platform User Manual 4.1.1.3 Rev 1.2 Fan Indicators The Fan indicator is located on the left side of the front panel (connector side) of the unit. The following fan status conditions are possible: Table 4 - Fan LED Configurations LED Configuration FAN LED Off There is no power to the fans. Green OK – The fans are running. Red Error – fans are not operating properly. Replace the switch. Fans must be operating while the power supply is plugged in.
Rev 1.2 4.1.1.5 Hardware Basic Operation and Installation UID LED Switch Identifier The UID LED is a debug feature that will become available to customers in the near future. For details please contact Mellanox Technologies support. 4.1.1.6 Port Connector LED Above the ports are two LEDs one for the upper port and one for the lower port lowing table shows the port status according to the LED indication. .
QDR InfiniBand Switch Platform User Manual 4.4 Rev 1.2 Interfaces 4.4.1 Port Connector Interfaces The Connector side of the switch has 36 QSFP ports. These are placed in two rows, 18 ports to a row. The ports are labelled as shown in Figure 4. The bottom row ports are flipped from the top row. See Figure 5. Figure 4: Port Numbering 1 3 5 7 9 11 13 2 4 6 8 10 12 14 15 17 19 21 23 25 16 18 20 22 24 27 29 31 33 35 26 28 30 32 34 36 Figure 5: Top and Bottom Ports 4.
Rev 1.2 Hardware Basic Operation and Installation Figure 6: RJ45 I2C Connector I2C IS5024 5 36 P/N: MIS5024Q-1BFR Rev: X1 S/N: MT0924X00266 GUID: 0002C9020040DDB0 Made in IL All firmware updates should be done in-band using Mellanox Firmware Management Tools. The I2C connector is available only on externally managed switches and is used to update firmware should in-band firmware update not work. You will need to order an MTUSB-1 USB to I2C adapter to make use of the I2C interface.
QDR InfiniBand Switch Platform User Manual 5 Installation 5.1 Package Contents Rev 1.2 The package includes: • 1 switch • 1 installation kit • 1 box containing 1 power cord Type B 6ft US 125V 10A chord • 1 installation guide • 1 parts list Before you install your new switch, unpack the system and check to make sure that all the parts have been sent, check this against the parts list. Check the parts for visible damage that may have occurred during shipping.
Rev 1.2 Installation Parts Included in the Installation Kit: • 2 rails • 2 rail slides • 6 recessed flat head screws • 8 caged nuts with pan head screws Figure 7: Rail and Rail Slide Rail Rail slide Flat head screws Caged nuts and bolts 15. 1. 2. 22 Place the ESD mat on the floor where you will be working and put on the ESD strap. Make sure the ESD strap is touching your skin and that the other end is connected to a verified ground. Screw the rail slides to the switch.
QDR InfiniBand Switch Platform User Manual Rev 1.2 Figure 8: Screw on the Switch Slide This has only 3 columns 3. 4. 5. 6. 7. 8. 9. Put 8 caged nuts into the rack at the level you want to install the switch. Screw the rails to the rack. Slide the switch onto the rail slides, and screw the switch to the rack. Tighten all of the screws. Ground the switch. Plug in the power cord. You can start connecting all of the cables to the switch. 5.2.
Rev 1.2 Installation 5.2.2 Power Connections and Initial Power On The power cord is included in the installation kit. Caution: The switch platform will automatically power on when AC power is applied. There is no power switch. Caution: After inserting a power cable and confirming the green system status LED light is on; make sure that the Fan Status indicator shows green. If the fan status indicator is not green then unplug the power connection and call your supplier. 5.2.
QDR InfiniBand Switch Platform User Manual 5.3 Rev 1.2 Disassembly of the Switch from the Rack 5.3.1 Disassembly Procedure — Single Switch Centered Installation To disassemble the switch from the rack: 1. 2. 3. 4. 5. Unplug and remove all connectors. Unplug the power cord. Remove the ground wire. Remove the four M6 screws holding the switch to the rack. Support the switch while removing the screws, or the switch may fall out of the rack. Slide the switch out of the rack.
Rev 1.2 Installation You will also need to download and unzip the firmware binary image. This is provided in the Mellanox Web site at: http://www.mellanox.com => Support > Download Center and go to the InfiniScale IV Switch systems. Click in the Table for the firmware image that you need. 5.5.1 Current Firmware Revision The user can query for the currently loaded firmware revision by using the command: ibstat (part of mft package). Other tools like flint can also be useful. 5.5.1.
QDR InfiniBand Switch Platform User Manual Rev 1.2 Device ID: 48438 Chip Revision: A0 Description: Node Sys image GUIDs: 0002c902004177d8 0002c902004177db Board ID: n/a (MT_0D00110012) VSD: n/a PSID: MT_0D00110012 Burn the InfiniScale IV switch: flint -d /dev/mst/SW_MT48438_lid-0x0003 -i ./fw-IS4-rel-7_4_0-Q_A1.bin -qq b 5.5.1.2 Reprogramming the Switch Through The I2C Port This interface is for Debug and Troubleshooting only. This interface is for FAEs and advanced users only.
Rev 1.2 Installation Figure 11: I2C Cable Connected to IS5024 IS5024 36 With the switch connected to a computer through the MTUSB, it is now possible to reprogram the SPI Flash memory. 5.5.1.3 Instructions for Reprogramming Through the I2C Port 1. Make sure that: •the MTUSB I2C cable is connected to the switch and USB cable is connected to a computer •MFT is running on the computer •"mtusb-1" mst device appears in "mst status" 2. Set the switch I2C switch to allow access by running: 3.
QDR InfiniBand Switch Platform User Manual 6 Rev 1.2 Troubleshooting As soon as a switch is plugged in, continuously monitor the status light. Make sure that it turns green within 30 seconds of booting up. Status LED and Fan LED If the Fan LED is red make sure that the ambient temperature is within the range specified in the Specification section of the Appendix. If any of the LEDs are red after 30 seconds, unplug the switch and call your Mellanox representative. Power supply unit: 1.
Rev 1.2 Troubleshooting 4. 5. 6. 7. 8. If the switch does not come on, check the power supply. If the switch comes on, Use the management software to determine the cause of the shutdown. Check the temperature of the switch. Check the ambient air temperature to make sure it is not too high. Check the Fan status. The switch is not working and unresponsive: 1. 2. 3. 4. 5. 30 Unplug the switch. Wait 5 minutes. Plug in the switch. If the switch does not come on, check the power supplies.
QDR InfiniBand Switch Platform User Manual Rev 1.2 Appendix A: Specification Table 7 - IS5024 Specification Data Physical Size: Weight: Mounting: 1.73” (1U) H x 16.9” W x 15.8” D 44mm X 428mm X 403.4mm 14 lb. / 6.
Rev 1.2 Table 7 - IS5024 Specification Data Physical Power and Environmental Scalability and Performance Reliability, Availability and Serviceability Features Switching Performance: Addressing: Switching Capacity A.1 Simultaneous wire-speed any port to any port 48K Unicast Addresses Max. per Subnet 16K Multicast Addresses per Subnet 2.88Tb/s EMC Certification Statements Table 8 lists the approved certification status per switch in different regions of the world.
QDR InfiniBand Switch Platform User Manual Rev 1.2 This device complies with Part 15 of the FCC Rules. Operation is subject to the following two conditions: 1. 2. This device may not cause harmful interference, and This device must accept any interference received, including interference that may cause undesired operation. § 15.
Rev 1.2 • Connect the equipment into an outlet on a circuit different from that to which the receiver is connected. Consult the dealer or an experienced radio/TV technician for help. A.1.2 EN Statements (Europe) EN55022 Class A Statement: Warning This is a class A product. In a domestic environment this product may cause radio interference in which case the user may be A.1.3 ICES Statements (Canada) Class A Statement “This Class A digital apparatus complies with Canadian ICES-003.
QDR InfiniBand Switch Platform User Manual Rev 1.2 Translation: Class A Device This device is registered for EMC requirements for industrial use. The seller or buyer should be aware of this. If this type was sold or purchased by mistake, it should be replaced with a residential-use type.
Rev 1.
ResetL SCL SDA GND RX3p RX3n GND ModSelL RX1n 15 RX1p 14 GND 13 VccRx 12 19 11 1 0 9 18 8 GND 20 GND 21 RX2n 22 RX2p 23 GND 24 RX4n 25 RX4p 26 GND 27 ModPrsL 28 IntL 29 VccTx 30 Vcc1 31 LPMode 14 13 12 11 1 0 9 8 7 6 5 4 3 2 1 GND 32 GND TX2n 7 17 6 16 5 TX4p 15 TX2p 33 TX3p 16 GND 34 TX3n TX4n 4 TX4n TX4p 18 17 GND 35 GND ModSelL 3 GND ResetL 36 TX1p SCL 2 TX2p SDA 1 TX2n GND 19 GND 3 8 TX1n 3 7 TX1p 3 6 GND 3 5 TX3n 3 4 32 TX3p 3 3 GND R
Rev 1.2 Appendix C: RJ45 I2C Interface The RJ45 I2C interface uses the EIA 568A standard wiring color coding.
QDR InfiniBand Switch Platform User Manual Rev 1.2 Appendix D: Replacement Parts Ordering Numbers Table 10 - Replacement Parts Ordering Numbers Part Description OPN RS232 Cable DB9 to DB9 Harness HAR000040 RS232 Cable RJ45 to DB9 Harness HAR000028 I2C DB9 or RJ45 to USB Adapter MTUSB-1 Power cord Type C13-C14 250V 10A 1.
Rev 1.2 Appendix E: Avertissements de sécurité d’installation (French) 1. Instructions d’installation Lisez toutes les instructions d’installation avant de brancher le matériel à la source d’alimentation électrique. 2. Température excessive Ce matériel ne doit pas fonctionner dans une zone avec une température ambiante dépassant le maximum recommandé de 45°C (113°F). Un flux d’air de 200LFM à cette température ambiante maximale est nécessaire.
QDR InfiniBand Switch Platform User Manual Rev 1.2 8. Elimination du matériel L’élimination de ce matériel doit s’effectuer dans le respect de toutes les législations et réglementations nationales en vigueur. 9. Codes électriques locaux et nationaux Ce matériel doit être installé dans le respect des codes électriques locaux et nationaux. 10.
Rev 1.2 Appendix F: Installation - Sicherheitshinweise (German) 1. Installationsanleitungen Lesen Sie alle Installationsanleitungen, bevor Sie das Gerät an die Stromversorgung anschließen. 2. Übertemperatur Dieses Gerät sollte nicht in einem Bereich mit einer Umgebungstemperatur über der maximal empfohlenen Temperatur von 45°C (113°F) betrieben werden. Es ist ein Luftstrom von 200 LFM bei maximaler Umgebungstemperatur erforderlich. Außerdem sollten mindestens 8 cm (3 in.
QDR InfiniBand Switch Platform User Manual Rev 1.2 8. Geräteentsorgung Die Entsorgung dieses Geräts sollte unter Beachtung aller nationalen Gesetze Bestimmungen erfolgen. 9. Regionale und nationale elektrische Bestimmungen Dieses Gerät sollte unter Beachtung der regionalen und nationalen elektrischen Bestimmungen installiert werden.
Rev 1.2 Appendix G: Advertencias de seguridad para la instalación (Spanish) 1. Instrucciones de instalación Antes de conectar el equipo a la fuente de alimentación, leer todas las instrucciones de instalación. 2. Sobrecalentamiento No se debe utilizar el equipo en un área con una temperatura ambiente superior a la máxima recomendada: 45°C. Además, para garantizar una circulación de aire adecuada, se debe dejar como mínimo un espacio de 8 cm (3 pulgadas) alrededor de las aberturas de ventilación. 3.
QDR InfiniBand Switch Platform User Manual Rev 1.2 8. Asegurar confinamientos adecuados El fabricante del producto final o el usuario final deberán suministrar un confinamiento adecuado para componentes eléctricos y mecánicos y contra incendio. 9. Eliminación de equipos La eliminación definitiva de este equipo se debe efectuar conforme a todas las leyes y reglamentaciones nacionales. 10.
Rev 1.2 13. Códigos de instalación Este dispositivo se debe instalar conforme a la versión más reciente de los códigos eléctricos nacionales del país en cuestión. En América del Norte, el equipo se debe instalar de acuerdo con las disposiciones vigentes del Código Eléctrico Nacional de los EE.UU. y del Código Eléctrico de Canadá. 14.
QDR InfiniBand Switch Platform User Manual Rev 1.2 Appendix H: Special Regulations Regarding Finland, Sweden, Denmark, and Norway Denmark- This unit is class I and must be connected with an AC cord compliant with all national electrical codes in Denmark. The AC cord shall have an integral ground wire, and can only be plugged into a fully grounded outlet.
