Specifications
46
High Performance Trading/Algo Speed with Wombat Design and Implementation Guide
OL-15617-01
Appendix B—Building and Configuring Switches
Note Dashed borders in Figure 18 delineate pods.
Remember, do not rack anything yet! Just keep in mind
where it is all going. This section is about planning, not
executing.
How many HCAs do I need? You need at least one HCA per host. You can install one-port
HCAs or two-port HCAs.
Where do I put my hosts? Ideally, 32 to 36 hosts reside in each non-core rack, along
with 2 or 3 leaf switches. At this time, note the number of
hosts that you plan to bring up and the number of racks that
they will require. When it becomes time to assign IP
addresses, the best course of action is to match certain
addresses to certain racks. Ideally, each node then receives
an address with the following format:
10.0.rackNumber.nodeNumber (e.g. 10.0.2.1 applies to
the first host in the second rack).
What cable lengths do I need, and how many of each cable
do I need?
After you plan where your switches and hosts will reside in
your data center, measure your cabling requirements. All
hosts connect to leaf switches. All leaf switches connect to
all core switches.
Note All cables from leaf switches to core switches must
run overhead or under the floor, so allow for room
for cables to reach up to and down from overhead
tracks.
Failure rates for InfiniBand cables average at 1%. Be sure
to include an additional 1% of cables for each cable size.
How many Velcro straps do I need to organize cables, and
what size straps do I need?
Cisco recommends that you buy bulk double-sided Velcro
straps so you can 1) cut as many straps as you need and 2)
cut all straps to size.
How do I configure my Ethernet management network? You must develop, independently, a plan for Ethernet
installation, management, and networking. This document
deals with InfiniBand installation and management only, but
requires an Ethernet management network as part of the
installation.
Note You must successfully bring up your Ethernet
network after you rack all hardware but before you
begin to bring up pods in the InfiniBand network.
How many IP addresses will I need? You will need IP addresses for all of the following:
• Each host (to run the IPoIB protocol)
• Each SFS chassis (for connectivity via an Ethernet
management network)
Table 11 Planning Requirements (continued)