ConnectX®-2 Dual Port QSFP and SFP+ Adapter Card User Manual P/N: MHZH29-XTR, MHZH29-XSR, MHZH29B-XTR, MHZH29B-XSR Rev 1.4 www.mellanox.
Rev 1.4 NOTE: THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT (“PRODUCT(S)”) AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES “AS-IS” WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUSTOMER'S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO(S) AND/OR THE SYSTEM USING IT.
ConnectX-2 VPI Card User Manual Rev 1.4 Table of Contents Table of Contents 3 List of Figures 5 List of Tables 6 Revision History 7 About this Manual 8 Intended Audience Related Documentation Online Resources Document Conventions Technical Support Firmware and Software Updates Chapter 1 Overview 1.1 1.2 1.3 1.
Rev 1.4 C.1 Remove the Existing Bracket from the Adapter Card C.
ConnectX-2 VPI Card User Manual Rev 1.
Rev 1.
ConnectX-2 VPI Card User Manual Rev 1.4 Revision History This document was printed on 2/14/11. Table 1 - Revision History Table Date Rev Comments/Changes Feb. 14th, 2011 1.4 July 22nd, 2010 1.3.4 Replaced Figure 1 photo of card with graphic July 21st, 2010 1.3.3 Fixed TOC Fixed Table 5 June 23rd, 2010 1.3.2 Minor formatting May 13th, 2010 1.3.1 Minor formatting April 2010 1.3 Replaced power numbers with the latest power numbers Feb. 2010 1.
Rev 1.4 About this Manual This User Manual describes Mellanox Technologies ConnectX®-2 Dual Port VPI InfiniBand and Ethernet PCI Express x8 adapter cards. It provides details as to the interfaces of the board, specifications, required software and firmware for operating the board, and relevant documentation. Intended Audience This manual is intended for the installer and user of these cards. The manual assumes basic familiarity with InfiniBand® and Ethernet networks and architecture specifications.
ConnectX-2 VPI Card User Manual Rev 1.4 • URL: http://www.mellanox.com => Support • E-mail: support@mellanox.com • Tel: +1.408.916.0055 Customers who purchased Mellanox M-1 Global Support Services, please see your contract for details regarding Technical Support. Customers who purchased Mellanox products through a Mellanox approved reseller should first seek assistance through their reseller.
Rev 1.4 1 Overview Overview This document is a User Manual for Mellanox Technologies network VPI adapter cards based on MT25408 the ConnectX®-2 VPI integrated circuit device. The cards described in this manual have the following main features: • IEEE Std 802.3 compliant • Virtual Protocol Interconnect IBTA v1.2.
ConnectX-2 VPI Card User Manual 1.1 Rev 1.4 Adapter Cards Covered in this Manual The MHZH29 card is a VPI adapter card with a 40Gb/s InfiniBand QSFP connector and a 10GigE SFP+ connector. The QSFP connector is compatible with InfiniBand Architecture Specifications The SFP+ connector is compatible with 10GigE. Table 3 lists the Single port VPI adapter cards described in this manual.
Rev 1.
ConnectX-2 VPI Card User Manual 1.2 Rev 1.4 Mellanox Part Numbering Legend Table 4 describes the Mellanox Technologies adapter cards part numbering legend.
Rev 1.4 1.3 Overview Finding the GUID/MAC and Serial Number on the Adapter Cards All Mellanox adapter cards have a label on the printed side of the adapter card that has the card serial number, the card MAC for Ethernet protocol, and the card GUID for InfiniBand protocol VPI Cards have both a MAC and a GUID. For VPI cards the MAC is derived from the GUID.
ConnectX-2 VPI Card User Manual Rev 1.4 3. During Lightning - Electrical Hazard During periods of lightning activity, do not work on the equipment or connect or disconnect cables. 4. Copper Cable Connecting/Disconnecting Some copper cables are heavy and not flexible, as such they should be carefully attached to or detached from the connectors. Refer to the cable manufacturer for special warnings and instructions. 5.
Rev 1.4 Adapter Card Interfaces 2 Adapter Card Interfaces 2.1 I/O Interfaces Each adapter card includes the following interfaces: • QSFP port • SFP+ port • PCI Express x8 edge connector • I/O panel LEDs • I2C compatible connector (for debug) Port 1 connects to the InfiniBand port of the device, while port 2 connects to the Ethernet port of the device. Figure 4: Port Numbering Port 1 QSFP Port 2 SFP+ 2.1.
ConnectX-2 VPI Card User Manual Rev 1.4 2.1.3 VPI Port Configuration VPI ports are auto-sensing but can be manually configured using a script. Port type management: By default ConnectX® -2 ports are initialized as Infiniband ports. If you wish to change the port type use the connectx_port_config script after the driver is loaded. The script is installed as part of the Mellanox OFED for Linux package (under /sbin). See the Mellanox OFED for Linux User’s Manual available at http://www.mellanox.
Rev 1.4 Adapter Card Interfaces The following configurations are supported by dual port VPI: Table 5 - Supported Port Configurations Port 1 Port 2 Ethernet Ethernet IB IB auto-sensing auto-sensing IB Ethernet IB auto-sensing auto-sensing Ethernet The following options are not supported by dual port VPI: Table 6 - Unsupported Port Configurations Port 1 Port 2 Ethernet IB Ethernet auto-sensing auto-sensing IB 2.1.
ConnectX-2 VPI Card User Manual Rev 1.
Rev 1.4 Adapter Card Interfaces 2.1.6 I2C Compatible Interface A three-pin header on the adapter card is provided as the I2C compatible interface. See Appendix A,“Specifications,” on page 38 for the location on the board. Figure 7: I2C Connector 2.2 Power All adapter cards receive 12V and 3.3V power from the PCI Express Edge connector. All other required power voltages are generated by on-board switch mode regulators. See “Specifications” on page 38. 2.
ConnectX-2 VPI Card User Manual Rev 1.4 Table 8 - Jumper Configuration Description Flash present/ not present Option connection open – Flash present connection shorted – Flash not present Card Default Configuration connection open – Flash present Comments Header 1x2 Figure 8: Flash Jumper 2.4.3 EEPROM Each board incorporates an EEPROM that is accessible through the I2C. The EEPROM is used for storing the Vital Product Data (VPD). The EEPROM capacity is 4Kb. 2.
Rev 1.4 2.5.
ConnectX-2 VPI Card User Manual Rev 1.
Rev 1.4 Driver Software and Firmware 3 Driver Software and Firmware 3.1 Driver Software 3.1.1 Linux For Linux, download and install the latest OpenFabrics Enterprise Distribution (OFED) software package available via the Mellanox Web site at: http://www.mellanox.com => Support > Download Center. Follow the installation instructions included in the download package. 3.1.2 Windows Support > Download Center.
ConnectX-2 VPI Card User Manual Rev 1.4 • RHEL5.4 2.6.18-164.el5 x86_64 or later • Mellanox OFED with GPUDirect support • NVIDIA Development Driver for Linux version 195.36.15 or later Hardware: • Mellanox ConnectX-2 HCA card • NVIDIA Tesla series 3.3.2 Installation For installation instructions visit: www.mellanox.com => Support => VPI SW/Driver. 3.3.2.
Rev 1.4 Driver Software and Firmware - Check the driver version, for example: # cat /proc/driver/nvidia/version => version 195.36.15 (or later) 3.4 CORE-Direct (Collectives Offload Resource Engine) CORE-Direct (Collectives Offload Resource Engine) provides the most advanced solution for handling collectives operations, ensures maximum scalability, minimizes the CPU overhead and provides the capability for overlapping communications with computations.
ConnectX-2 VPI Card User Manual Rev 1.4 • OFED 1.5.1 or later • firmware version 2.7.700 or later 3.5.2 Installation For installation instructions visit: www.mellanox.com => Support => Ethernet SW/Driver 3.6 Updating Adapter Card Firmware Each card is shipped with the latest version of qualified firmware at the time of manufacturing. Firmware is updated occasionally, and the most recent firmware can be obtained from: http://www.mellanox.com =>Support > Download Center.
Rev 1.
ConnectX-2 VPI Card User Manual Rev 1.4 4 VPI Adapter Card Installation 4.1 Hardware and Software Requirements Before installing the adapter card, please make sure that the system meets the hardware and software requirements listed in Table 11. Refer to Chapter 3,“Driver Software and Firmware” on page 24 for download and installation instructions.
Rev 1.4 4.3 VPI Adapter Card Installation Set Up This section is valid for InfiniBand, Ethernet and VPI cards. Disregard sections that are not relevant to your card. The basic steps to embed Mellanox cards in your computer are: 1. Identify the card in your system. 2. Verify the computer is recognizing the new adapter. 3. Install the adapter drivers. 4. Update the adapter firmware if needed. The configuration of the card is dependant upon your choice to use InfiniBand or Ethernet. 4.3.
ConnectX-2 VPI Card User Manual Rev 1.4 Figure 10: Hardware Devices 3. Select a PCI Device / InfiniBand Controller entry. 4. Right-click. 5. Select “Properties to display the PCI Device Properties” window. 6. Click the Details tab and select Device Instance Id (Windows 2003) or Hardware Ids (Windows 2008/R2) from the Property pull-down menu.
Rev 1.4 VPI Adapter Card Installation Figure 11: PCI Device 7. In the Value display box, check the fields VEN and DEV (fields are separated by ‘&’). In the display example above, notice the sub-string “PCI\VEN_15B3&DEV_6368”: VEN is equal to 0x15B3 – this is the Vendor ID of Mellanox Technologies; and DEV is equal to 0x6368 – this is a valid Mellanox Technologies PCI Device ID. The list of Mellanox Technologies PCI Device IDs can be found in the PCI ID repository at http://pci-ids.ucw.cz/read/PC/15b3.
ConnectX-2 VPI Card User Manual Rev 1.4 4.3.1.2 Linux Get the device location on the PCI bus by running lspci and locating lines with the string “Mellanox Technologies”: > lspci | grep Mellanox 2:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE] (rev b0) Make sure that either the MLNX_OFED driver or the MLNX_EN driver is loaded and configured. Check the link status First check the network interface name by running the “ifconfig –a” command.
Rev 1.4 VPI Adapter Card Installation Rate: 10 Base lid: 6 LMC: 0 SM lid: 3 Capability mask: 0x0251086a Port GUID: 0x0002c903000c8711 Link layer: IB Check the OFED version To get the version of the running Mellanox OFED/BXOFED, run the following command: Host# ofed_info | head -1 BXOFED-1.5.1-1.3.7-rc19: Troubleshooting MLNX_OFED Installation For troubleshooting driver installation, please check Mellanox OFED driver user manual: http://www.mellanox.com => Support > Adapter IB/VPI SW.
ConnectX-2 VPI Card User Manual Rev 1.4 The “Usage: openibd {start|stop|restart|status}” command to modify this file and thereby control the drivers. Ethernet Driver Usage and Configuration To assign an IP address to the interface run: #> ifconfig eth where 'n' is the OS assigned interface number. • To check driver and device information run: #> ethtool -i eth Example: #> ethtool -i eth2 driver: mlx4_en (MT_04A0140005) version: 1.5.1 (March 2010) firmware-version: 2.7.
Rev 1.4 VPI Adapter Card Installation 4.4.1 Optical modules for SFP+ SFP+ transceiver modules can be used to connect fiber optic cables to the cards greatly increasing the cable reach. The adapter cards are shipped without optical modules. Mellanox 10GBASE-SR (MFM1T02ASR) and 10GBASE-LR (MFM1T02A-LR) optical modules are recommended. The figure below shows the Mellanox SFP+ module. Inserting the Optical Transceiver Module To insert the module into the cage: 1.
ConnectX-2 VPI Card User Manual Rev 1.4 4.4.4 Cable Installation All cables can be inserted or removed with the unit powered on. To insert a cable, press the connector into the port receptacle until the connector is firmly seated. The GREEN LED indicator will light when the physical connection is established (that is, when the unit is powered on and a cable is plugged into the port with the other end of the connector plugged into a functioning port).
Rev 1.4 Appendix A: Specifications A.1 MHZH29B-X[ST]R Specifications Table 12 - Specifications for MHZH29-X[ST]R Physical Power and Environmental Size: Air Flow: QSFP40Gb/s Connector: SFP+ Connector: 2.71in. x6.60in. (68.90mm x 167.65mm) InfiniBand (Copper and optical) Max power per port 2.0 W. Cable power budget class 2 Maximum Power: 8.66W for passive cables only 11.36W for active optic modules Temperature: 0°C to 55°C Regulatory IBTA v1.2.
ConnectX-2 VPI Card User Manual A.2 Rev 1.4 MHZH29-X[ST]R Specifications Table 13 - Specifications for MHZH29-X[ST]R Physical Power and Environmental Size: Air Flow: QSFP40Gb/s Connector: SFP+ Connector 2.71in. x6.60in. (68.90mm x 167.65mm) InfiniBand (Copper and optical) Max power per port 2.0 W. Cable power budget class 2 Maximum Power: 8.66W for passive cables only 11.36W for active optic modules Temperature: 0°C to 55°C Regulatory IBTA v1.2.
Rev 1.4 All dimensions are in millimeters. All the mechanical tolerances are +/-0.1mm Figure 14: Schematic of the ConnectX-2 MHZH29B Adapter Card 142.25 L8 Q1 Q2 U1 J8 64.40 J7 U14 U3 56.15 J4 J1 J3 U4 12.75 15.00 45.96 3.65 45.01 40 Mellanox Technologies 1.
ConnectX-2 VPI Card User Manual Rev 1.4 Figure 15: Schematic of the ConnectX-2 MHZH29 Adapter Card J3 – Flash Jumper J1 – I2C Connector J1 U1 15.00 3.65 33.35 43.13 57.13 A.4 U11 U22 L15 Q3 68.90 Q4 J5 J6 U4 L7 J7 64.40 56.97 U2 J4 J1 U3 167.65 U16 1.90 96.30 EMC Certification Statements the approved certification status per adapter card in different regions of the world.
Rev 1.4 A.4.1 FCC Statements (USA) Class A Statements: § 15.19(a)(4) This device complies with Part 15 of the FCC Rules. Operation is subject to the following two conditions: 1. This device may not cause harmful interference, and 2. This device must accept any interference received, including interference that may cause undesired operation. § 15.
ConnectX-2 VPI Card User Manual A.4.4 Rev 1.4 VCCI Statements (Japan) Class A Statement: (Translation - "This is a Class A product based on the standard of the Voluntary Control Council for Interference by Information Technology Equipment (VCCI). If this equipment is used in a domestic environment, radio interference may occur, in which case the user may be required to take corrective actions.") A.4.
Rev 1.4 Appendix B: Interface Connectors Pinout B.1 I2C-Compatible Connector Pinout Figure 16: Compatible Connector Plug and Pinout Table 15 - I2C-Compatible Connector B.2 4 3 2 1 5 5 1 2 3 4 Connector Pin Number Signal Name 1 SPSDA 2 SPSCL 3 GND 4 NC 5 NC PCI Express x8 Connector Pinout The adapter cards use a standard PCI Express x8 edge connector and the PCI Express x8 standard pinout according to the PCI Express 2.0 specification.
ConnectX-2 VPI Card User Manual Rev 1.4 B.3 PCI Express Connector Pinout B.4 QSFP Connector Pinout Figure 17: Connector and Cage Views Top 18.
Rev 1.
ConnectX-2 VPI Card User Manual SFP+ Connector Pinout Figure 18: Rear View of Module With Pin Placement Top 13.70 8.
Rev 1.4 Table 17 - SFP+ Connector Pinout Pin Symbol Name Description 1 VeeT Transmitter Ground (Common with Receiver Ground) a 2 TX_Fault Transmitter Fault.b 3 TX_Disable Transmitter Disable. Laser output disabled on high or open. c 4 SDA 2-wire Serial Interface Data Line d 5 SCL 2-wire Serial Interface Clock Line d 6 MOD_ABS Module Absent. Grounded within the module d 7 RS0 No connection required 8 RX_LOS Loss of Signal indication. Logic 0 indicates normal operation.
ConnectX-2 VPI Card User Manual Rev 1.4 Appendix C: Replacing a Tall Bracket With a Short Bracket This section provides instructions on how to remove the tall bracket of a standard Mellanox Technologies adapter card and replace it with a short one. To replace the bracket you will need the following parts: • the new bracket of the proper height • one new QSFP EMI gasket • the 2 screws saved from the removal of the bracket • the 2 fiber washers saved from the removal of the bracket C.
Rev 1.4 2. Place the gasket onto the new bracket. Make sure to correctly align the gasket with the hole in the bracket. 3. If the old gasket is still on the card, remove it before installing the new bracket. Make sure that only one gasket is used. Figure 20: Gasket Installation 4. Place the bracket onto the card until the screw holes line up. Do not force the bracket onto the card. You may have to gently push the LEDs using a small screwdriver to align the LEDs with the holes in the bracket. 5.
ConnectX-2 VPI Card User Manual Rev 1.4 Appendix D: Avertissements de sécurité d’installation (Warnings in French) 1. Instructions d’installation Lisez toutes les instructions d’installation avant de brancher le matériel à la source d’alimentation électrique. 2. Température excessive Ce matériel ne doit pas fonctionner dans une zone avec une température ambiante dépassant le maximum recommandé de 55°C (131°F). Un flux d’air de 200LFM à cette température ambiante maximale est nécessaire.
Rev 1.4 Appendix E: Sicherheitshinweise (Warnings in German) 1. Installationsanleitungen Lesen Sie alle Installationsanleitungen, bevor Sie das Gerät an die Stromversorgung anschließen. 2. Übertemperatur Dieses Gerät sollte nicht in einem Bereich mit einer Umgebungstemperatur über der maximal empfohlenen Temperatur von 55°C (131°F) betrieben werden. Es ist ein Luftstrom von 200 LFM bei maximaler Umgebungstemperatur erforderlich. Außerdem sollten mindestens 8 cm (3 in.
ConnectX-2 VPI Card User Manual Rev 1.4 Appendix F: Advertencias de seguridad para la instalación (Warnings in Spanish) 1. Instrucciones de instalación Antes de conectar el equipo a la fuente de alimentación, leer todas las instrucciones de instalación. 2. Sobrecalentamiento No se debe utilizar el equipo en un área con una temperatura ambiente superior a la máxima recomendada: 55°C(131°F).
Rev 1.4 Códigos eléctricos locales y nacionales Este equipo se debe instalar conforme a los códigos eléctricos locales y nacionales.
