10GBASE-T for Broad 10 Gigabit Adoption in the Data Center
SFP+
SFP+’s support for both ber optic cables
and DAC make it a better (more exible)
solution than CX4. SFP+ is ramping today,
but has limitations that will prevent this
media from moving to every server.
10GBASE-SR (SFP+ Fiber)
Fiber is great for latency and distance (up
to 300 meters), but it is expensive. Fiber
offers low power consumption, but the
cost of laying ber networking every-
where in the data center is prohibitive
due largely to the cost of the electronics.
The ber electronics can be four to ve
times more expensive than their copper
counterparts, meaning that ongoing active
maintenance, typically based on original
equipment purchase price, is also more
expensive. Where a copper connection
is readily available in a server, moving to
ber creates the need to purchase not
only the ber switch port, but also a ber
NIC for the server.
10GBASE-SFP+ DAC
DAC is a lower cost alternative to ber,
but it can only reach 7 meters and it is
not backward-compatible with existing
GbE switches. DAC requires the purchase
of an adapter card and requires a new
top of rack (ToR) switch topology. The
cables are much more expensive than
structured copper channels, and cannot
be eld terminated. This makes DAC more
expensive than 10GBASE-T. The adoption
rate of DAC for LOM will be low since it
does not have the exibility and reach of
10GBASE-T.
10GBASE-T
10GBASE-T offers the most exibility, the
lowest cost media, and is backward-com-
patible with existing 1 GbE networks.
Reach
Like all BASE-T implementations,
10GBASE-T works for lengths up to 100
meters, giving IT managers a far greater
level of exibility in connecting devices in
the data center. With exibility in reach,
10GBASE-T can accommodate either top
of the rack, middle of row, or end of the
row network topologies. This gives IT
managers the most exibility in server
placement since it will work with existing
structured cabling systems.
For higher grade cabling plants (category
6A and above) 10GBASE-T operates in
low power mode (also known as data
center mode) on channels under 30 m.
This means a further power savings per
port over the longer 100 m mode. Data
centers can create any-to-all patching
zones to assure less than 30 m channels
to realize this savings.
Backward Compatibility
Because 10GBASE-T is backward-compat-
ible with 1000BASE-T, it can be deployed
in existing 1 GbE switch infrastructures
in data centers that are cabled with CAT6
and CAT6A (or above) cabling, enabling IT
to keep costs down while offering an easy
migration path to 10 GbE.
Power
The challenge with 10GBASE-T is that
the early physical layer interface chips
(PHYs) consumed too much power for
widespread adoption. The same was true
when gigabit Ethernet products were
released. The original gigabit chips were
roughly 6.5 Watts per port. With process
improvements, the chips improved from
one generation to the next. The resulting
GbE ports are now under 1 W per port.
The same has proven true for 10GBASE-
T. The good news with 10GBASE-T is
that the PHYs benet greatly from the
latest manufacturing processes. PHYs are
Moore’s Law-friendly, and the newer pro-
cess technologies will continue to reduce
both the power and cost of the latest
10GBASE-T PHYs.
When 10GBASE-T adapters were rst
introduced in 2008, they required 25
W of power for a single port. Power has
been reduced in successive generations of
using newer and smaller process tech-
nologies. The latest 10GBASE-T adapt-
ers require only 10 W per port. Further
improvements will reduce power even
3
10GBASE-T for Broad 10 Gigabit Adoption in the Data Center
more. In 2011, power will drop below 6 W
per port, making 10GBASE-T suitable for
motherboard integration and high-density
switches.
Latency
Depending on packet size, latency for
1000BASE-T ranges from below 1 µs to
over 12 µs. 10GBASE-T ranges from just
over 2 µs to less than 4 µs--a much tighter
latency range. For Ethernet packet sizes
of 512 bytes or larger, 10GBASE-T’s over-
all throughput offers an advantage over
1000BASE-T. Latency for 10GBASE-T
is more than three times lower than
1000BASE-T with larger packet sizes. Only
the most latent-sensitive applications such
as High Performance Computing )HPC) or
high frequency trading systems would be
affected by normal 10 GbE latency.
The incremental 2 µs latency of 10GBASE-
T is of no consequence to most users. For
the large majority of enterprise applica-
tions that have been operating for years
with 1000BASE-T latency, 10GBASE-T la-
tency only makes things better. Many LAN
products purposely add small amounts of
latency to reduce power consumption or
CPU overhead. A common LAN feature is
interrupt moderation. Enabled by default,
this feature typically adds ~100 µs of
latency in order to allow interrupts to be
coalesced and greatly reduce the CPU
burden. For many users this trade-off
provides an overall positive benet.
Cost
As power metrics have dropped dramati-
cally over the last three generations, cost
has followed a similar downward curve.
First-generation 10GBASE-T adapters
cost $US 1000 per port. Today’s third-
generation dual-port 10GBASE-T adapters
are less than $US 400 per port. In 2011,
10GBASE-T will be designed in as LAN on
Motherboard (LOM) and will be included in
the price of the server. By using the new
resident 10GBASE-T LOM modules, users
will see a signicant savings over the pur-
chase price of more expensive SFP+ DAC
and ber optic adapters and will be able to
free up and I/O slot in the server.