Datasheet

All contents are Copyright © 1992–2006 Cisco Systems, Inc. All rights reserved. This document is Cisco Public Information. Page 3 of 6
in the InfiniBand fabric and still have full access to Ethernet or Fibre Channel I/O connectivity provided by the Cisco SFS 3000
Series.
Q. How do I use the Cisco SFS 7000D Series to build InfiniBand fabrics larger than the size of a single switch?
A. InfiniBand fabrics, for both high-performance computing and unified I/O grids, which are larger than a single switch, are built with
2-tier designs by connecting multiple switches together. Switches are connected together using the standard ports on the switch; there
are no special ports for connecting Cisco SFS 7000D Series switches together. When building these large fabrics, Cisco SFS 7000D
Series switches fulfill two different roles in the big fabric: edge switch and core switch. Any one of the Cisco SFS 7000D Series
switches can serve as either an edge switch or a core switch, but not both at the same time. Edge switches use some of their switch
ports to connect to servers, and the remaining ports to connect to core switches. Each core switch brings the edge switches together in
a web-like fashion by connecting to every edge switch. The number of core switches in the fabric and number of connections between
an individual core switch and an edge switch will vary based on fabric design. Core switches do not connect to servers directly. Using
this 2-tier model, it is possible to build very large InfiniBand fabrics without sacrificing server-to-server performance.
Q. What is the maximum distance between two ports, either switch ports or HCA ports, in a fabric?
A. Currently, the Cisco SFS 7000D Series supports copper cable distances of up to 15 meters (m) port-to-port in SDR mode and up
to 10 meters in DDR mode. It is possible for two servers to be much farther apart than that by stringing switches together, but the
distance between two ports cannot exceed 15 m over copper links.
Q. Does the Cisco SFS 7000D Series support optical connections?
A. Yes, the Cisco SFS 7000D Series switches can use optical connections though external pluggable optical modules (POMs). A POM
enables an individual port on a Cisco SFS 7000D Series switch to use 50-micron or 62.5-micron fiber cables. With 50-micron cables,
port-to-port fiber distances can exceed 150 m, and with 62.5-micron cables, port-to-port fiber distances go up to 75 m. One POM is
needed for each port involved in the connection; so, one switch-to-switch connection would require two POMs. The POMs can be
ordered directly from vendors such as Emcore or Fujitsu.
Q. What is a blocking factor in an SFS InfiniBand fabric?
A. InfiniBand fabrics can be designed to be fully nonblocking, which provides full 20-Gbps bandwidth from every server to every
other server in the fabric, or partially blocking. Partially blocking designs require fewer switches to build, but provide less average
bandwidth to each server. Blocking factors are typically introduced on the edge switch by using more ports to connect servers than
to connect the edge switch into the core switches. On the edge switches, if an equal number of ports is used to connect servers and to
connect to the core, the fabric is said to be nonblocking. If twice as many ports are used to connect servers as used to connect the edge
switch to the core, the blocking factor is said to be two to one or a 50 percent blocking factor. If three times as many ports are used to
connect servers as used to connect the edge switch to the core, the blocking factor is said to be three to one or a 67 percent blocking
factor. All single switch designs are nonblocking because Cisco SFS 7000D Series switches are internally fully nonblocking.
Q. Can you combine SDR and DDR in a fabric?
A. Yes. The Cisco SFS 7000D Series DDR InfiniBand switches have autosensing ports that will negotiate port speed to be double data
rate (DDR–20-Gbps) or single data rate (SDR–10-Gbps).
Q. What is the impact of mixing SDR and DDR on a fabric’s blocking factor?
A. The Cisco SFS 7000D Series DDR InfiniBand switches support autosensing of SDR or DDR attached devices and also supports
the capability to forward packets between SDR and DDR connections that provides better utilization of resources and investment
protection for existing InfiniBand 4X devices. To exemplify how this impacts blocking factor, the 24-port SFS 7000D can support
16 4X SDR server connections with 8 4X DDR uplinks to the InfiniBand fabric core and offer non-blocking capability while offering
a higher server-to-fabric density than a pure SDR switch.