Guide to Myrinet/PCI Host Interfaces Hardware & Software Installation Principles of Operation Myricom, Inc. Date: 18 February 2002 The most recent version of this document can be downloaded from http://www.myri.com/scs/doc/guide_to_interfaces.pdf © 2002 Myricom, Inc.
Regulatory Information Electromagnetic Compatibility (EMC) Myrinet-Fiber (M3F), Myrinet-Serial (M3S), and Myrinet-LAN (M2L) host interfaces and their associated cables are fully compliant with the following standards and specifications for the emission of and susceptibility to electromagnetic interference (EMI): European Union ➢ BS EN55024 (1998) ➢ BS EN55022 (1998) Class A Australia/New Zealand AS/NZS 3548 (w/A1 & A2 1997) Class A using: ➢ BS EN55022 (1998) Class A United States FCC Part 15 Subpart B Cla
Introduction Myricom currently produces two series of Myrinet/PCI host interfaces: the 64B series with 133MHz RISC and memory, and the 64C series with 200MHz RISC and memory.
PCI Mezzanine Cards PCI mezzanine cards (PMC) (as pictured below) are available only with Myrinet-SAN connections. PCI mezzanine card (PMC64B) All of these 64/32-bit, 66/33MHz, Myrinet-2000/PCI interfaces (64A, 64B, and 64C) are “universal”. They function correctly in hosts with 32-bit or 64-bit PCI slots that are compliant with PCI specifications (version 2.2), with a 33–66MHz PCI clock, and with either 3.3V or 5V PCI-bus signal levels. (3.
A label similar to that above appears on any Myrinet/PCI interface. This label contains the product code, the serial number, and the Ethernet MAC address. Please provide this information in any correspondence with Myricom support (help@myri.com). Installation As a first step in using Myrinet technology, you must install the Myrinet/PCI interfaces into your hosts.
Case 2: Rack-mounted host If you have a rack-mounted machine, the PCI slots will look something like the following: PCI slots (64-bit and 32-bit) In this photo the PCI slots are the white connectors. There are two 64-bit PCI slots and four 32-bit PCI slots. Case 3: PCI mezzanine motherboard If you have a PCI mezzanine motherboard, then the PCI slot should look something like the following: Photo of PCI mezzanine slot © 2002 Myricom, Inc.
where you can see a previously installed Myrinet/PMC interface to the left of the circled PMC connectors. 64-bit PMC slots or cards have three connectors, as shown. 32-bit PMC slots or cards have just two connectors. It is alright to plug a 64-bit card (PMC64A, PMC64B, or PMC64C) into a 32-bit slot, or a 32-bit card (PMC32A, PMC32B, PMC32C) into a 64-bit slot, but in either case the card will function as a 32-bit interface. Step 2: Insert the Myrinet/PCI interface into a PCI slot.
Installed interface in a 64-bit PCI slot Case 2: Rack-mounted host Since space is very restricted in a rack-mounted unit, the Myrinet/PCI interface must be installed with a riser card. The riser card is inserted on the PCI connector side of the card, and the riser card is then inserted into a PCI slot (as shown in the following photograph).
Caution: Although PCI riser cards are commonly used, they will generally violate PCI specifications for the length of signal traces. A riser card may also introduce impedance discontinuities and signal degradation between the mother board, riser card, and interface card. If you observe PCI-communication errors when using a riser card, see if the problem persists when you plug the Myrinet/PCI interface directly into the PCI slot.
Case 3: PCI mezzanine motherboard Insert the Myrinet/PMC interface horizontally into the two or three PCI slots on the motherboard. Inserting a PMC interface Note: Inserting a 64-bit interface into a 32-bit PCI slot As previously mentioned, it is possible to install a 64-bit Myrinet/PCI interface into a 32bit PCI slot, as shown below. A 64-bit Myrinet/PCI card in a 32-bit PCI slot © 2002 Myricom, Inc.
However, if this is done, the data path is 32 bits, and the 64-bit interface will be able to transfer data only at half the data rate of a 64-bit slot. Step 3: Secure the card in place with a locking screw (if applicable), replace the exterior cover of the host, and attach the cable (Fiber, Serial, or SAN) between the card and the switch (as detailed on pages 3-4 of the document “Guide to Myrinet-2000 Switches and Switch Networks,” available at http://www.myri.com/myrinet/m3switch/guide/index.html).
Linux If your host is running Linux, you can issue the command /sbin/lspci, which will return all devices attached to the PCI bus. Screen image of /sbin/lspci command. If the output you receive is similar to that above, namely you see the “MYRICOM Inc” entry, the Myrinet/PCI interface has been correctly detected, and you can now proceed to installing software. Refer to the “Guide to Software” for details of the required software and for installation instructions. © 2002 Myricom, Inc.
Solaris If your host is running Solaris, you can issue the command /usr/sbin/prtconf, which will print the system configuration. Screen dump of /usr/sbin/prtconf command If the output you receive is similar to that above, namely you see the “MYRICOM” entry, the Myrinet/PCI interface has been correctly detected, and you can now proceed to installing software. Refer to the “Guide to Software” for details of the required software and for installation instructions. © 2002 Myricom, Inc.
Tru64 If your host is running Tru64, prior to booting the machine, or after halting the operating system, at the SRM console prompt, type show conf Screen dump of show conf command In Slot 13, under “Option” you will see 804314C1. This field contains the card ID, 8043, for the Myrinet/PCI interface, and the card vendor ID for Myricom, 14C1 (as listed in the PCI Vendor registry).
FreeBSD If your host is running FreeBSD, you can issue the command /usr/sbin/pciconf -l. Screen dump of /sbin/pciconf –l command The third column gives the contents of the subvendor id register. The field consists of the card ID in the upper half of the value and the card vendor ID in the lower half of the value. The card ID for the Myrinet/PCI interface is 8043, and the vendor ID for Myricom (as listed in the PCI Vendor registry) is 14c1.
IRIX If your host is running IRIX, you can issue the command /sbin/hinv –v –c iobd. Screen dump of /sbin/hinv command The PCI adapters with vendor ID 5313 are Myrinet/PCI interfaces. If the output you receive is similar to that above, the Myrinet/PCI interface has been correctly detected, and you can now proceed to installing software. Refer to the “Guide to Software” for details of the required software and for installation instructions. © 2002 Myricom, Inc.
MacOS X If your host is running MacOS X, you can issue the command ioreg, which will return all devices attached to the PCI bus. An excerpt of the sample output is as follows: <…> || || || || | +-o USBKeyLargo +-o pci14c1,8043@12 | +-o gm +-o usb@18 <…> If the output you receive contains the “8043” device, the Myrinet/PCI interface has been correctly detected, and you can now proceed to installing software.
Windows 2000 and Windows NT For Windows 2000 and Windows NT, a “Found New Hardware” dialog box (as shown below) will appear when the host detects a new PCI card. Once a Myrinet/PCI interface has been installed and the host powered on, the appearance of this dialog box will signal that Windows has detected the interface card. Photo of Dialog Box © 2002 Myricom, Inc.
Principles of Operation As illustrated in the following block diagram, all Myrinet/PCI interfaces include a RISC processor to execute the Myrinet Control Program (MCP), local memory, a packet interface to and from the Myrinet port, and a versatile DMA controller to support zero-copy APIs. Each of these parts support high-availability and data-integrity features, such as “heartbeat” linkcontinuity monitoring, packet checksums, and memory parity.
The front panel of the PCI-Short-Card interface pictured above is the vertical metal plate on the left-hand side of the interface. Two LEDs and the cable connector penetrate this PCI front panel (also known as a face plate). For the PCI mezzanine cards (pictured below), the two LEDs are on either side of the SAN cable connector.
interface, the topology of the network changes, and the routes would need to be updated by the GM mapper (see “Guide to Software”). A summary of the specifications of the PCI64B and PCI64C Myrinet/PCI interfaces is listed below: PCI-bus Interface: 64/32-bit, 66/33MHz, supports all burst modes and write-invalidate, master or slave.
Myrinet-2000-Serial port: 2.0+2.0 Gb/s at an HSSDC connector to a Myrinet Serial-Link cable up to 10m in length. Myrinet-2000-SAN port: The default data rate can be switched between SAN-2000 (2.0+2.0 Gb/s) and SAN-1280 (1.28+1.28 Gb/s) with a mechanical switch on the circuit board. The host can over-ride the default data rate. The SAN port appears on the A link of the SAN connector (refer to http://www.myri.com/myrinet/cables/m2m-cl.html for details); the B link is unused.
