Guide to Myrinet/PCI-X Network Interface Cards Hardware & Software Installation Principles of Operation Myricom, Inc. Date: 11 March 2005 The most recent version of this document can be downloaded from http://www.myri.com/scs/doc/guide_to_pcix_nics.pdf © 2005 Myricom, Inc.
Regulatory Information Electromagnetic Compatibility (EMC) Myrinet-Fiber (M3F) network interface cards (NICs) and their fiber cables are fully compliant with the following standards and specifications for the emission of and susceptibility to electromagnetic interference (EMI): European Union ® BS EN55024 (1998) ® BS EN61000-3-2 (2001) ® BS EN61000-3-3 (1995) W/A1:98 ® BS EN55022 (1998) Class A Australia/New Zealand AS/NZS 3548 (1005 W/A1 & A2: 97) Class A using: ® BS EN55022 (1998) Class A Canada ICES-003
Introduction Myricom currently produces three series of Myrinet/PCI-X Network Interface Cards (NICs): the PCIXD series with 225MHz RISC and memory, and the PCIXE series and PCIXF series with 333MHz RISC and memory. These series of NICs have the “low profile PCI Short Card” form factor.
M3F2-PCIXE-2 two-port Myrinet-Fiber / PCI-X NIC with a standard PCI faceplate M3F-PCIXF-2 Myrinet-Fiber / PCI-X NIC with standard PCI faceplate These PCIX-series NICs operate in hosts with 75-133MHz PCI-X slots or 66MHz (3.3V) PCI slots. The PCIXD NICs are also available as host-card adaptors (HCA) for the IBM BladeCenter. Contact sales@myri.com for additional information on the IBM BladeCenter HCAs. © 2005 Myricom, Inc.
Identification on each Myrinet/PCI-X NIC Product code Serial number Ethernet MAC address The label on a Myrinet/PCI-X NIC A label similar to that above appears on the underside of every Myrinet/PCI-X NIC. This label contains the product code, the serial number, and the Ethernet MAC address. Please provide this information in any correspondence with Myricom support (help@myri.com). Installation As a first step in using Myrinet technology, you must install the Myrinet/PCI-X NICs into your hosts.
which PCI-X slots share a PCI bus. To minimize the signal degradation between the mother board and the NIC (and for best performance), we recommend that the Myrinet/PCI-X NIC is inserted in the PCI-X slot closest to the PCI chipset. We also recommend that the Myrinet/PCI-X NIC not share the PCI bus with another PCI device.
PCI-X slots on a rack-mountable mother board In this photo, there are 6 PCI-X slots. One of the PCI-X slots is green, and the remaining five PCI-X slots are white connectors. Step 2: Insert the Myrinet/PCI-X NIC into a PCI-X slot. Case 1: Tower unit host With a tower unit host, a low-profile PCI-short-card Myrinet/PCI-X NIC with a standard PCI faceplate is installed in a PCI-X slot.
Installed NIC in a PCI-X slot Case 2: Rack-mounted host If the rack-mounted host is a 2U or larger, the low-profile PCI-short-card Myrinet/PCI-X NIC with low-profile PCI faceplate may not require a riser card for installation. If at all possible, avoid the use of a riser card. If you must use a riser card and the riser card has multiple slots, insert the Myrinet/PCI-X NIC into the riser card slot closest to the PCI-X slot on the mother board.
Insertion of NIC (without riser card) into a PCI-X slot Installed NIC (without riser card) However, since space is very restricted in a 1U rack-mounted host, if the rackmounted host is a 1U host, the Myrinet/PCI-X NIC must be installed with a riser card. The riser card is inserted on the PCI connector side of the card, and the riser card is then inserted into a PCI slot (as shown in the following photograph). If you © 2005 Myricom, Inc.
must use a riser card and the riser card has multiple slots, insert the Myrinet/PCIX NIC into the riser card slot closest to the PCI-X slot on the mother board. Inasmuch as some PCI connectors are more close-fitting than others, you may need to push quite hard to get the card to seat securely. Installed NIC (with attached riser card) Installed NIC (with riser card) © 2005 Myricom, Inc.
Caution: If at all possible, avoid the user of a riser card. Although PCI riser cards are commonly used, they will generally violate PCI specifications for the length of signal traces. A riser card may also introduce impedance discontinuities and signal degradation between the mother board, riser card, and network interface card. If you observe PCI-communication errors when using a riser card, see if the problem persists when you plug the Myrinet/PCI-X NIC directly into the PCI slot.
Linux If your host is running Linux, you can issue the command /sbin/lspci, which will return all devices attached to the PCI bus. Screen image of /sbin/lspci command. If the output you receive is similar to that above, namely you see the “Network controller: MYRICOM Inc” entry, the Myrinet/PCI-X NIC has been correctly detected. The card ID for the Myrinet/PCI-X NIC is 8043, the (rev 04) denotes a PCIXD NIC, and the (rev 05) denotes a PCIXE NIC.
Windows 2000 and Windows XP For Windows 2000 and Windows XP, a “Found New Hardware” dialog box (as shown below) will appear when the host detects a new PCI card. Once a Myrinet/PCI-X NIC has been installed and the host powered on, the appearance of this dialog box will signal that Windows has detected the network interface card. Photo of Dialog Box © 2005 Myricom, Inc.
Solaris If your host is running Solaris, you can issue the command /usr/sbin/prtconf, which will print the system configuration. You will see the following text in the output of prtconf. <…> pci14c1,8043, instance #0 <…> This output contains the card ID, 8043, for the Myrinet/PCI-X NIC, and the card vendor ID for Myricom, 14C1 (as listed in the PCI Vendor registry).
AIX If your host is running AIX, issue the command lsdev -Ct gmpci, and you should see output similar to the following. <…> ent2 Available 1V-08 Myrinet GM/Ethernet PCI Adapter ent3 Defined 1D-08 Myrinet GM/Ethernet PCI Adapter ent4 Available 1H-08 Myrinet GM/Ethernet PCI Adapter <…> Before the GM module is loaded on the host, the Myrinet NICs will be listed as Defined. As soon as the GM software is loaded on the host, the entry will change to Available.
Mac OS X If your host is running Mac OS X, you can issue the command ioreg, which will return all devices attached to the PCI bus. An excerpt of the sample output is as follows: <…> || || || || | +-o USBKeyLargo +-o pci14c1,8043@12 | +-o gm +-o usb@18 <…> If the output you receive contains the “8043” device, the Myrinet/PCI-X NIC has been correctly detected, and you can now proceed to the software installation. © 2005 Myricom, Inc.
FreeBSD If your host is running FreeBSD, you can issue the command /usr/sbin/pciconf -l. Screen dump of /sbin/pciconf –l command The third column gives the contents of the subvendor id register. The field consists of the card ID in the upper half of the value and the card vendor ID in the lower half of the value. The card ID for the Myrinet/PCI-X NIC is 8043, and the vendor ID for Myricom (as listed in the PCI Vendor registry) is 14c1.
Principles of Operation All Myrinet/PCI-X NICs include a RISC processor to execute the Myrinet Control Program (MCP), local memory, a packet interface to and from the Myrinet port, and a versatile DMA controller to support zero-copy APIs. Each of these parts supports highavailability and data-integrity features, such as “heartbeat” link-continuity monitoring, packet checksums, and memory parity.
The green “link” LED will illuminate when the board is powered, the firmware (such as the GM-2 or MX-2G Myrinet Control Program) is loaded, and the link is active. The link will be active if the port is plugged through a cable to a powered switch, to another initialized NIC or to a loopback plug. The green LED blinks at up to ~5Hz to show when packet traffic is flowing through the port. The green LEDs on Myrinet-2000 switches operate similarly.
EEPROM: 512KB, which includes the PCI-configuration-space data; Vital Product Data (VPD), including the serial number, MAC address, product code, and other information; initialization programs; and optional self-test and bootstrap programs. The EEPROM can be re-programmed by the Lanai-XP/Lanai-2XP. Myrinet-2000-Fiber port: 2.0+2.0 Gb/s data rate, 2.5+2.5 Gbaud signaling, 8b/10b encoded Myrinet serial link. An “LC” optical connector attaches to a fiber pair up to 200m of 50/125 multi-mode fiber.
Refer to http://www.myri.com/myrinet/performance/ for full details on the applicationlevel performance difference between the PCIXD, PCIXE, and PCIXF NICs. © 2005 Myricom, Inc.
