ConnectX®-2 VPI Single and Dual Port QSFP InfiniBand and Ethernet Adapter Card User Manual P/N: MHRH29B-XTR, MHRH29B-XSR, MHQH29B-XTR, MHQH29B-XSR, MHQH29C-XTR, MHQH29CXSR, MHRH19C-XTR, MHRH19C-XSR, MHQH19C-XTR, MHQH19C-XSR, MHQH19B-XSR, MHQH19B-XTR, MHRH19B-XSR, MHRH19B-XTR Rev 1.6 www.mellanox.
Rev 1.6 ConnectX®-2 VPI Single and Dual Port QSFP InfiniBand and Ethernet Adapter Card User Manual NOTE: THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT (“PRODUCT(S)”) AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES “AS-IS” WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS.
ConnectX®-2 VPI Single and Dual Port QSFP InfiniBand and Ethernet Adapter Card User Manual Rev 1.6 Table of Contents Table of Contents 3 List of Figures 5 List of Tables 6 Revision History 7 About this Manual 8 Intended Audience Related Documentation Online Resources Document Conventions Technical Support Firmware and Software Updates Chapter 1 Chapter 2 Overview 10 1.1 1.2 1.3 1.4 11 13 14 14 16 2.1 I/O Interfaces 2.1.1 InfiniBand Interface 2.1.2 Ethernet Interface 2.1.
Rev 0.10 4.3 4.4 4.5 4.6 4.7 FlexBoot NVIDIA GPUDirect Support 4.4.1 Hardware and Software Requirements 4.4.2 Installation 4.4.2.1 Kernel Installation: 4.4.2.2 MLNX OFED Drivers Installation: 4.4.2.3 NVIDIA Driver Installation: CORE-Direct (Collectives Offload Resource Engine) 4.5.1 Hardware and Software Requirements 4.5.2 Installation RDMA over Converged Ethernet (RoCE) 4.6.1 Hardware and Software Requirements 4.6.2 Installation Updating Adapter Card Firmware Appendix A Specifications A.1 A.2 A.3 A.
ConnectX®-2 VPI Single and Dual Port QSFP InfiniBand and Ethernet Adapter Card User Manual Rev 1.
Rev 1.
ConnectX®-2 VPI Single and Dual Port QSFP InfiniBand and Ethernet Adapter Card User Manual Rev 1.6 Revision History This document was printed on October 30, 2011. Table 1 - Revision History Table Comments/Changes Date Rev Feb. 14th, 2010 1.6 Added MHRH19B-X[ST]R and MHQH19B-X[ST]R Changed Air Flow to min. 200LFM Made changes to the Set Up section Minor fixes Nov. 24th, 2010 1.5 Added MHRH19C-X[ST]R and MHQH19C-X[ST]R Nov. 21st, 2010 1.4 Added MHQH29C-X[ST]R Oct. 27th, 2010 1.
Rev 1.6 About this Manual This User Manual describes Mellanox Technologies ConnectX®-2 Single and Dual Port VPI InfiniBand and Ethernet PCI Express x8 adapter cards. It provides details as to the interfaces of the board, specifications, required software and firmware for operating the board, and relevant documentation. Intended Audience This manual is intended for the installer and user of these cards.
ConnectX®-2 VPI Single and Dual Port QSFP InfiniBand and Ethernet Adapter Card User Manual Rev 1.6 • URL: http://www.mellanox.com => Support • E-mail: support@mellanox.com • Tel: +1.408.916.0055 Customers who purchased Mellanox M-1 Global Support Services, please see your contract for details regarding Technical Support. Customers who purchased Mellanox products through a Mellanox approved reseller should first seek assistance through their reseller.
Rev 1.6 1 Overview Overview This document is a User Manual for Mellanox Technologies VPI adapter cards based on the ConnectX®-2 VPI integrated circuit device. The cards described in this manual have the following main features: • Virtual Protocol Interconnect • InfiniBand Architecture Specification v1.2.1 compliant • Compliant with QSFP MSA spec Rev. 1.0 • PCI Express 2.0 (1.
ConnectX®-2 VPI Single and Dual Port QSFP InfiniBand and Ethernet Adapter Card User Manual 1.1 Rev 1.6 Adapter Cards Covered in this Manual Table 3 lists the single VPI adapter cards described in this manual. Table 4 lists the dual port VPI adapter cards described in this manual. Table 3 - Single Port Cards List PCI Express SERDES Speed Data Transmission Rate/ # of ports MHRH19C-XSR PCIe Gen2 5.
Rev 1.6 Overview Table 4 - Dual Port Cards List Short / Tall Bracket RoHS HCA IC Part Number 20Gb/s (DDR) Dual port Short R-6 MT25408B0-FCCR-GI PCIe Gen2 5.0 GT/s 20Gb/s (DDR) Dual port Tall R-6 MT25408B0-FCCR-GI MHQH29B-XSR PCIe Gen2 5.0 GT/s 40Gb/s (QDR) Dual port Short R-6 MT25408B0-FCCR-QI MHQH29B-XTR PCIe Gen2 5.0 GT/s 40Gb/s (QDR) Dual port Tall R-6 MT25408B0-FCCR-QI MHQH29C-XSR PCIe Gen2 5.
ConnectX®-2 VPI Single and Dual Port QSFP InfiniBand and Ethernet Adapter Card User Manual 1.2 Rev 1.6 Mellanox Part Numbering Legend Table 5 describes the Mellanox Technologies adapter cards part numbering legend.
Rev 1.6 1.3 Overview Finding the GUID/ MAC and Serial Number on the Adapter Cards Each Mellanox adapter card has a label on the print side that shows the card serial number, the card MAC for Ethernet protocol, and the card GUID for InfiniBand protocol. VPI cards have both a GUID and a MAC (derived from the GUID). Figure 3: Card Product Label Port 1 uses the GUID or MAC ID described on the label. For Port 2 GUID or MAC add 1 to Port 1's respective values. 1.
ConnectX®-2 VPI Single and Dual Port QSFP InfiniBand and Ethernet Adapter Card User Manual Rev 1.6 2. Over-temperature This equipment should not be operated in an area with an ambient temperature exceeding the maximum recommended: 55°C (131°F). To guarantee proper air flow, allow at least 8cm (3 inches) of clearance around the ventilation openings. 3. During Lightning - Electrical Hazard During periods of lightning activity, do not work on the equipment or connect or disconnect cables. 4.
Rev 1.6 Adapter Card Interfaces 2 Adapter Card Interfaces 2.1 I/O Interfaces Each adapter card includes the following interfaces: • QSFP ports • PCI Express x8 edge connector • I/O panel LEDs • I2C-compatible connector (for debug) For dual port cards, port 1 connects to connector 1 of the device, while port 2 connects to connector 2 of the device. Figure 4: Dual Port Numbering Port 1 Port 2 2.1.
ConnectX®-2 VPI Single and Dual Port QSFP InfiniBand and Ethernet Adapter Card User Manual Rev 1.6 2.1.2 Ethernet Interface The ConnectX-2 adapter card is compliant with the IEEE Std 802.3. The VPI adapter cards listed above, based on this device, provide access to the Ethernet ports by means of QSFP connectors.The Mellanox QSA (QSFP to SFP+) adapter modules can be ordered through Mellanox Technologies authorized dealers. 2.1.
Rev 1.6 Adapter Card Interfaces 2.1.5 LED Assignment There are two I/O LEDs per port, green and yellow, located on the I/O panel. Table 8 outlines the different LED functions.
ConnectX®-2 VPI Single and Dual Port QSFP InfiniBand and Ethernet Adapter Card User Manual 2.2 Rev 1.6 Power All adapter cards receive 12V and 3.3V power from the PCI Express Edge connector. All other required power voltages are generated by on-board switch mode regulators. See “Specifications” on page 44. 2.3 QSFP Power Level The card supports power level 3, according to SFF Committee SFF-8436 Specification for QSFP (Quad Small Form-factor Pluggable) Transceiver. 2.
Rev 1.6 Adapter Card Interfaces The PCI VPD (Vital Product Data) layout, for each of the adapter cards complies with the format defined in the PCI 2.3 Specification, Appendix I.
ConnectX®-2 VPI Single and Dual Port QSFP InfiniBand and Ethernet Adapter Card User Manual Rev 1.
Rev 1.
ConnectX®-2 VPI Single and Dual Port QSFP InfiniBand and Ethernet Adapter Card User Manual Rev 1.
Rev 1.
ConnectX®-2 VPI Single and Dual Port QSFP InfiniBand and Ethernet Adapter Card User Manual Rev 1.
Rev 1.
ConnectX®-2 VPI Single and Dual Port QSFP InfiniBand and Ethernet Adapter Card User Manual 3 VPI Adapter Card Installation 3.1 Hardware Requirements Rev 1.6 Before installing the adapter card, please make sure that the system meets the hardware requirements listed in Table 17. Refer to Chapter 4,“Driver Software and Firmware” on page 32 for download and installation instructions. Table 17 - Hardware Requirements Requirement Hardware 3.
Rev 1.6 VPI Adapter Card Installation 4. Click the Details tab and select Device Instance Id (Windows 2003) or Hardware Ids (Windows 2008/R2) from the Property pull-down menu. If you cannot find any PCI device, click Action --> Scan for hardware changes. If no PCI devices are detected, check that the network adapter card(s) is correctly installed in the PCI slot or try installing the adapter card into a different PCI slot. Figure 8: PCI Device 5.
ConnectX®-2 VPI Single and Dual Port QSFP InfiniBand and Ethernet Adapter Card User Manual Rev 1.6 . The list of Mellanox Technologies PCI Device IDs can be found in the PCI ID repository at http://pci-ids.ucw.cz/read/PC/15b3. 3.3.1.1 Linux Get the device location on the PCI bus by running lspci and locating lines with the string “Mellanox Technologies”: > lspci |grep -i Mellanox 08:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE] (rev b0) 3.
Rev 1.6 VPI Adapter Card Installation 3. After plugging in a cable, lock the connector using the latching mechanism particular to the cable vendor. When a logical connection is made the YELLOW LED will come on. When data is being transferred the yellow led will blink. When installing cables make sure that the latches engage. Always install and remove cables by pushing or pulling the cable and connector in a straight line with the card. 4. 5.
ConnectX®-2 VPI Single and Dual Port QSFP InfiniBand and Ethernet Adapter Card User Manual Rev 1.6 Figure 9: Connector Orientation (example) 3.4.3.2 Removing a Cable from the Adapter Card 1. 2. 3. Pull on the latch release mechanism thereby unlatching the connector and pull the connector out of the cage. Do not apply torque to the connector when removing it from the adapter card. Remove any cable supports that were used to support the cable’s weight.
Rev 1.6 Driver Software and Firmware 4 Driver Software and Firmware 4.1 Driver Software 4.1.1 Linux For Linux, download and install the latest OpenFabrics Enterprise Distribution (OFED) software package available via the Mellanox web site at: http://www.mellanox.com => Products => Software/Drivers => InfiniBand & VPI Software/Drivers => Mellanox OFED => Download. Follow the installation instructions included in the download package (also available from the download page).
ConnectX®-2 VPI Single and Dual Port QSFP InfiniBand and Ethernet Adapter Card User Manual Rev 1.6 To check the IB link status, for IB and VPI cards, run “ibstat” and focus on the Physical state attributes. Example: Host# ibstat CA 'mlx4_0' CA type: MT26428 Number of ports: 2 Firmware version: 2.9.
Rev 1.6 Driver Software and Firmware To get the version of the running Mellanox OFED/BXOFED, run the following command: # ofed_info |head -1 MLNX_OFED_LINUX-1.5.3-0.1.6 (OFED-1.5.3-0.1.6): Troubleshooting MLNX_OFED Installation For troubleshooting driver installation, please check Mellanox OFED driver user manual ar http://www.mellanox.com => Support => InfiniBand Products => Mellanox OFED. Loading the Ethernet Driver By default, the Mellanox OFED stack loads mlx4_en.
ConnectX®-2 VPI Single and Dual Port QSFP InfiniBand and Ethernet Adapter Card User Manual Rev 1.6 #> ethtool -i eth Example: ethtool -i eth0 driver: igb version: 2.1.0-k2 firmware-version: 1.4-3 bus-info: 0000:01:00.0 #> ethtool -i eth0 driver: bnx2 version: 2.0.2 firmware-version: 5.0.11 NCSI 2.0.4 bus-info: 0000:01:00.
Rev 1.6 Driver Software and Firmware Displaying the Device Manager will show the Mellanox adapter devices and an IPoIB (network) device for each port. Note that MLNX_VPI configures the ports of VPI cards – at installation time – to run the InfiniBand protocol. See Chapter 8: “Port Protocol Configuration” for instructions to change this configuration. Figure 10: Device Manager Note: If the cards are connected to a managed switch, there is no need to run openSM. Only one OpenSM should run per subnet.
ConnectX®-2 VPI Single and Dual Port QSFP InfiniBand and Ethernet Adapter Card User Manual Rev 1.6 • "eth" - Always Ethernet • "ib" - Always Infiniband • "auto" - Link sensing mode - detect the port type based on the attached network type. If no link is detected, the driver retries link sensing every few seconds. The port link type can be configured for each device in the system at run time using the "/sbin/ connectx_port_config" script.
Rev 1.6 Driver Software and Firmware Step 2. Note: The "Port Protocol” tab is displayed only if the NIC is a VPI (IB and ETH). If the NIC is either only IB or ETH, the tab will not be shown. Note: The figure below is an example of the displayed Port Protocol sheet for a dual-port VPI adapter card. Step 3. 4.3 Right-click on the Mellanox ConnectX VPI network adapter and left-click Properties. Select the Port Protocol tab from the Properties sheet.
ConnectX®-2 VPI Single and Dual Port QSFP InfiniBand and Ethernet Adapter Card User Manual Rev 1.6 http://www.mellanox.com => Products => Software/Drivers => InfiniBand & VPI Software/Drivers => FlexBoot => Download 4.4 NVIDIA GPUDirect Support Utilizing the high computational power of the Graphics Processing Unit (GPU), the GPU-to-GPU method has proven valuable in various areas of science and technology.
Rev 1.6 Driver Software and Firmware # cat /sys/module/ib_core/parameters/gpu_direct_enable => 1 The number of shared pages by GPUDirect is reported under: # cat /sys/module/ib_core/parameters/gpu_direct_shares 4.4.2.3 NVIDIA Driver Installation: Install NVIDIA Development Driver for Linux x86_64, available under: http://developer.nvidia.com/object/cuda_3_0_downloads.html For example: - Run: devdriver_3.0_linux_64_195.36.15.
ConnectX®-2 VPI Single and Dual Port QSFP InfiniBand and Ethernet Adapter Card User Manual Rev 1.6 low latency features. This includes reliable connected service, datagram service, RDMA and send/ receive semantics, atomic operations, user level multicast, user level I/O access, kernel bypass, and zero copy. ConnectX-2 with RoCE based network management is the same as that for any Ethernet and DCBbased network management, eliminating the need for IT managers to learn new technologies. 4.6.
Rev 1.6 Driver Software and Firmware a. Install the MFT package. The package is available at http://www.mellanox.com => Products => Adapter IB/VPI SW => Firmware Tools. Make sure to download the package corresponding to your computer’s operating system. b. Enter: mst start c. Get the Mellanox mst device name using the command "mst status". The mst device name will be of the form: /dev/mst/mt_pci{_cr0|conf0 d.
ConnectX®-2 VPI Single and Dual Port QSFP InfiniBand and Ethernet Adapter Card User Manual 3. Rev 1.6 If a newer firmware version exists for your adapter card, update the firmware a. Download the firmware image zip file from the Download Center (see Step 2a above) b. Unzip the firmware image c. Burn the firmware image. Enter: > flint -d -i burn d. Reboot the computer e. Enter: mst start f.
Rev 1.6 Appendix A: Specifications A.1 MHQH19C-X[TSR] Specifications Table 18 - Specifications for MHQH19C-X[ST]R Physical Power and Environmental Size: Air Flow: QSFP 40Gb/s Connector: 2.11in. x5.6 in. (53.58mm x 142.24 mm) InfiniBand (Copper and optical) Max power per port 2.0 W. Cable power budget class 2. 12V, 3.3V Typ Power: Passive Cables 7.04W Active Cables 9.04W Maximum Power: Passive Cables 7.69W Active Cables 9.
ConnectX®-2 VPI Single and Dual Port QSFP InfiniBand and Ethernet Adapter Card User Manual A.2 Rev 1.2 MHRH19B-X[ST]R Specifications Table 19 - Specifications for MHRH19B-X[ST]R Physical Size: Air Flow: QSFP 10Gb/s Connector: 2.1in x 5.6in (53.6mm x 142.24mm) Power and Environmental Voltage: Typ. Power: Passive Cables 6.69W Active Cables 8.69W Maximum Power: Passive Cables 7.34W Active Cables 9.34W a 200LFM InfiniBand (Copper and optical) Max power per port 2.0 W Cable power budget class 2.
Rev 1.6 A.3 MHQH19B-X[ST]R Specifications Table 20 - Specifications for MHQH19B-X[ST]R Physical Size: Air Flow: QSFP 20Gb/s Connector: 2.1in x5.6in (53.60mm x 142.24mm) Power and Environmental Voltage: Typ. Power: Passive Cables 7.04W Active Cables 9.04W Maximum Power: Passive Cables 7.69W Active Cables 9.69W a 200LFM InfiniBand (Copper and optical) Max power per port 2.0W. Cable power budget class 2 Temperature: Protocol Support InfiniBand: IBTA v1.2.
ConnectX®-2 VPI Single and Dual Port QSFP InfiniBand and Ethernet Adapter Card User Manual A.4 Rev 1.2 MHQH29C-X[ST]R Specifications Table 21 - Specifications for MHQH29C-X[ST]R Physical Power and Environmental Size: Air Flow: QSFP 40Gb/s Connector: 2.66in. x5.6 in. (67.5mm x 142.24 mm) InfiniBand (Copper and optical) Max power per port 2.0 W. Cable power budget class 2. QoS: Passive Cables 8.78W Active Cables 12.78W Maximum Power: Passive Cables 9.36W Active Cables 13.
Rev 1.6 A.5 MHRH29B-X[ST]R Specifications Table 22 - Specifications for MHRH29B-X[ST]R Physical Power and Environmental Size: Air Flow: QSFP 20Gb/s Connector: 2.69in. x6.60in. (68.40mm x 167.65mm) InfiniBand (Copper and optical) Max power per port 2.0 W. Cable power budget class 2. QoS: Passive Cables 8.15W Active Cables 12.15W Maximum Power: Passive Cables 8.80W Active Cables 12.80W Temperature: 0°C to 55°C Regulatory IBTA v1.2.
ConnectX®-2 VPI Single and Dual Port QSFP InfiniBand and Ethernet Adapter Card User Manual A.6 Rev 1.2 MHQH29B-X[ST]R Specifications Table 23 - Specifications for MHQH29B-X[ST]R Physical Power and Environmental Size: Air Flow: QSFP 40Gb/s Connector: 2.69in. x6.60in. (68.40mm x 167.65mm) InfiniBand (Copper and optical) Max power per port 2.0 W. Cable power budget class 2. QoS: Passive Cables 8.78W Active Cables 12.78W Maximum Power: Passive Cables 9.36W Active Cables 13.
Rev 1.6 A.7 Board Mechanical Drawing and Dimensions All dimensions are in millimeters. All the mechanical tolerances are +/-0.
ConnectX®-2 VPI Single and Dual Port QSFP InfiniBand and Ethernet Adapter Card User Manual Rev 1.2 Figure 13: Schematic of the Dual Port Adapter Card MH[RQ]H29B-X[ST]R 167.65 J1- I2C Connector J3 - Flash jumper 56.15 64.40 56.97 68.40 15.00 33.35 57.15 96.
Rev 1.6 A.8 Regulatory Statements For regulatory statements for all ConnectX®-3 cards please refer to: http://www.mellanox.com/related-docs/user_manuals/Regulatory_and_Compliance_Guide.
ConnectX®-2 VPI Single and Dual Port QSFP InfiniBand and Ethernet Adapter Card User Manual Rev 1.6 Appendix B: Interface Connectors Pinout B.1 I2C-Compatible Connector Pinout Figure 15: Compatible Connector Plug and Pinout Table 24 - I2C-Compatible Connector Connector Pin Number Signal Name 1 SPSDA 2 SPSCL 3 GND 4 NC 5 NC B.
Rev 1.6 B.
ConnectX®-2 VPI Single and Dual Port QSFP InfiniBand and Ethernet Adapter Card User Manual Rev 1.
Rev 1.6 Appendix C: Replacing a Tall Bracket With a Short Bracket This section provides instructions on how to remove the tall bracket of a standard Mellanox Technologies adapter card and replace it with a short one.
ConnectX®-2 VPI Single and Dual Port QSFP InfiniBand and Ethernet Adapter Card User Manual C.1 Rev 1.6 Remove the Existing Bracket from the Adapter Card Figure 17: Remove the Bracket Screws Gaskets LEDs Screws Gasket LEDs 1. 2. Remove the two screws holding the bracket in place. The bracket comes loose from the card. Be careful not to put stress on the LEDs. 3. C.2 Save the two screws and the two fiber washers. Installing the New Bracket 1. 2.
Rev 1.6 3. If the old gasket is still on the card, remove it before installing the new bracket. Make sure that only one gasket per port is used. Figure 18: Gasket Installation 4. Place the bracket onto the card until the screw holes line up. Do not force the bracket onto the card. You may have to gently push the LEDs using a small screwdriver to align the LEDs with the holes in the bracket. 5. 58 Screw on the bracket using the screws and washers saved from the procedure above.
ConnectX®-2 VPI Single and Dual Port QSFP InfiniBand and Ethernet Adapter Card User Manual Rev 1.6 Figure 19: Placing the Bracket on the Card Gaskets in place on the bracket. 6. 7. Make sure that the LEDs are aligned onto the bracket holes.
Rev 1.6 Appendix D: Avertissements de sécurité d’installation (Warnings in French) 1. Instructions d’installation Lisez toutes les instructions d’installation avant de brancher le matériel à la source d’alimentation électrique. 2. Température excessive Ce matériel ne doit pas fonctionner dans une zone avec une température ambiante dépassant le maximum recommandé de 55°C (131°F). Un flux d’air de 200LFM à cette température ambiante maximale est nécessaire.
ConnectX®-2 VPI Single and Dual Port QSFP InfiniBand and Ethernet Adapter Card User Manual Rev 1.6 7. Codes électriques locaux et nationaux Ce matériel doit être installé dans le respect des codes électriques locaux et nationaux. 8. Exposition au rayonnement grave Mise en garde – l'utilisation de commandes ou de réglages ou l'exécution de procédures autres que ce qui est spécifié dans les présentes peut engendrer une exposition au rayonnement grave.
Rev 1.6 Appendix E: Installation - Sicherheitshinweise (German) 1. Installationsanleitungen Lesen Sie alle Installationsanleitungen, bevor Sie das Gerät an die Stromversorgung anschließen. 2. Übertemperatur Dieses Gerät sollte nicht in einem Bereich mit einer Umgebungstemperatur über der maximal empfohlenen Temperatur von 55°C (131°F) betrieben werden. Außerdem sollten mindestens 8 cm (3 in.) Freiraum um die Belüftungsöffnungen sein, um einen einwandfreien Luftstrom zu gewährleisten. 3.
ConnectX®-2 VPI Single and Dual Port QSFP InfiniBand and Ethernet Adapter Card User Manual Rev 1.6 Appendix F: Advertencias de seguridad para la instalación (Warnings in Spanish) 1. Instrucciones de instalación Antes de conectar el equipo a la fuente de alimentación, leer todas las instrucciones de instalación. 2. Sobrecalentamiento No se debe utilizar el equipo en un área con una temperatura ambiente superior a la máxima recomendada: 55°C(131°F).
