Datasheet

CORPORATE STYLE GUIDE
19 | Mellanox Technologies | July 2011
Main Title
Univers LT 59 UltraCondensed
26 pt / 29
Documents and Marketing Collateral
TECHNOLOGY BRIEFS
TECHNOLOGY BRIEF
©2011 Mellanox Technologies. All rights reserved.
The two commonly known RDMA (remote DMA) technologies are InniBand and iWARP (Internet
Wide Area RDMA Protocol). InniBand has enjoyed signicant success to date in HPC applications.
iWARP solutions over Ethernet have seen limited success because of implementation and deployment
challenges. Recent enhancements to the Ethernet data link layer under the umbrella of IEEE data center
Bridging (DCB) open signicant opportunities to proliferate the use of RDMA technology into mainstream
data center applications by taking a fresh and yet evolutionary look at how such services can be more
easily and efciently delivered over Ethernet. The proposed DCB standards include: IEEE 802.1bb –
Priority-based ow control, 802.1Qau – Congestion Notication, and 802.1az – Enhanced Transmission
Selection (ETS) and DCB Capability Exchange. The lossless delivery features in DCB, enabled by Priority-
based Flow Control (PFC), are analogous to those in the InniBand data link layer. As such, the natural
choice for building RDMA services over PFC-based DCB Ethernet is to apply use InniBand-based native
RDMA transport services. The IBTA (InniBand Trade Association) has recently released a specication
called RDMA over Converged Ethernet (RoCE, pronounced as “Rocky”) that applies the InniBand-based
native RDMA transport services over Ethernet. ConnectX-2 with RoCE (RDMA over Ethernet) implements
the RoCE standard to deliver InniBand-like ultra low latency and high scalability over Ethernet fabrics.
ConnextX-2 with RoCE is born out of combining InniBand native RDMA transport with Ethernet per the
IBTA RoCE specication. The data link InniBand-based layer 2 is replaced by Ethernet layer 2, as shown
in the gure below. The InniBand transport is applied over a PFC-based loss less Ethernet data link.
LRH
(L2 Hdr)
GRH
IB Transport
Headers
IB Payload
ICRC
VCRC
InfiniBand
RoCE
Eth L2
Header
GRH
IB Transport
Headers
IB Payload
ICRC
FCS
Application
IB Software transport
interface
IB-based transport
Network
Ethernet w/PFC (Data Link)
OFA verbs
Figure: Low Latency Ethernet packet
format and protocol stack
Software Interface: ConnextX-2 with RoCE is compliant with the Open Fabrics Alliance OFED
verbs denition and is interoperable with the OFA software stacks (similar to InniBand and iWARP).
ConnextX-2 with RoCE uses the proven and feature rich InniBand verbs interface available in the OFA
stacks. OFED v1.5.1 supports RoCE and ConnextX-2 with RoCE.
Transport Layer: ConnextX-2 with RoCE uses the InniBand transport layer, as dened in the IBTA RoCE
specication. The adaptation from InniBand data link to Ethernet data link is straight forward because
the InniBand transport layer was designed ground up to be data link layer agnostic. The InniBand
transport layer expects certain services from the data link layer related to lossless delivery of packets, and
these are delivered by a PFC enabled Ethernet data link layer. ConnextX-2 with RoCE inherits a rich set
of transport services beyond those required to support OFA verbs including connected and unconnected
modes, reliable and unreliable services. Built on top of these services is a full set of verbs-dened
operations including kernel bypass, send/receive, RDMA read/write, and atomic operations.
1.0 Opportunities with
Evolution of Ethernet
1.1 How ConnectX-2
RoCE Works
ConnectX
®
-2 with RoCE
(ConnectX-2 VPI and ConnectX-2 EN)
April 2010
. &$;
TECHNOLOGYBRIEF: ConnectX
®
2withRoCE
350OakmeadParkway,Suite100,Sunnyvale,CA94085


:;-.70(&'2;;$** ,-6;$"',-*-&($1;**;0(&'21;0$1$04$#
$** ,-6;0(#&$;-,,$"2;(0$"2;,9,(* 12;,9,(0(#&$;,9,(-12;,9,(;,9,(" *$;,9,(;'7;(023 *;0-2-"-*;,2$0"-,,$"2; ,#;-*2 (0$; 0$;0$&(12$0$#;
20 #$+ 0)1;-%;$** ,-6;$"',-*-&($1;2#; !0(";(1; ;20 #$+ 0);-%;$** ,-6;$"',-*-&($1;2#;**;-2'$0;20 #$+ 0)1; 0$;.0-.$027;-%;2'$(0;0$1.$"2(4$;-5,$01
Management Feature Required by
IB Transport Layer & Apps using
IB Transport Layer
How InfiniBand delivers them in
the InfiniBand subnet
How Ethernet (and CEE) delivers
them using standard Ethernet
management practices
L2 address assignment Subnet Manager L2 address
assignment
Fixed assigned L2 address or other
Ethernet mechanisms
L2 topology discovery and switch FDB
configuration
Subnet Manager topology discovery
using direct routed subnet manage-
ment packets (SMP). Subnet Manager
path computation and path distribution
Spanning Tree and Learning mecha-
nisms. Also, IEEE Transparnet Intercon-
nection of Lots of Links (TRILL) when
available and other eth practices
Address Resolution SA based path resolution Address Resolution Protocol (ARP) or
direct mapping
QoS QoS Manager extension to Subnet
Manager
Standard Ethernet QoS management
practices. Local API to access fabric
policy settings
Congestion management Congestion Manager for IB 802.1Qau congestion management
features
Performance management IB Performance Manager SNMP/RMON MIBS
Device/baseboard management IB Baseboard Manager SNMP/RMON MIBS
-,,$62;5(2';-; # .2$01;! 1$#;-,;2'$;;-;1.$"(9" 2(-,; 0$; 4 (* !*$;2-# 7;%0-+;$** ,-6;
$"',-*-&($1; ,#;' 4$;!$$,;#$+-,120 2$#;2-;#$*(4$0;$,#;2-;$,#; ..*(" 2(-,;*$4$*;* 2$,"($1;-%; 1;*-5; 1;
;+("0-1$"-,#1;$** ,-6; ,#;-2'$0;(,#31207;*$ #$01; 0$;"-** !-0 2(,&;-,;&0-5(,&;2'$;$"-1712$+;-%;
-! 1$#; # .2$01; ,#;(,#$.$,#$,2;1-%25 0$;4$,#-0; ..*(" 2(-,1;2' 2;" .(2 *(8$;-,;2'$;!$,$921;-%;
-,,$62;5(2';-;-+$;$6 +.*$1;-%;2 0&$2; ..*(" 2(-,1; 0$;9, ,"( *;1$04("$1;!31(,$11;(,2$**(&$,"$;
# 2 ;5 0$'-31(,&;"*-3#;"-+.32(,&; ,#;$!;
 1$#;-,;2'$;#(1"311(-,; !-4$;(2;(1;-!4(-31;2' 2;-,,$62;5(2';-;"-+$1;5(2';+ ,7; #4 ,2 &$1;
,#;'-*#1;2'$;.0-+(1$;2-;$, !*$;5(#$1.0$ #;#$.*-7+$,2;-%;;2$"',-*-&($1;(,;+ (,120$ +;# 2 ;
"$,2$0; ..*(" 2(-,1;;
1. -,,$62;5(2';-;32(*(8$1; #4 ,"$1;(,;2'$0,$2;;2-;$, !*$;$%9"($,2; ,#;*-5;"-12;(+.*$
+$,2 2(-,1;-%;;-4$0;2'$0,$2;;
2. -,,$62;;20 %9";" ,;!$;"* 11(9$#; 2;2'$;# 2 ;*(,);* 7$0;5'("';(1;% 12$0; ,#;0$/3(0$1;*$11;
;-4$0'$ #
3. -,,$62;5(2';-;#$*(4$01;31$"; ..*(" 2(-,;2-; ..*(" 2(-,;* 2$,"7;5'("';(1;2';-%;-2'$0;
(,#31207;12 ,# 0#;(+.*$+$,2 2(-,1;-4$0;2'$0,$2;$,"'+ 0)(,&;5(2';.-.3* 0;9, ,"( *;1$04("$1; .
.*(" 2(-,1;1'-5;+-0$;2' ,;;*-5$0;* 2$,"7; ..*(" !*$;2-;" .(2 *;+ 0)$2;# 2 ;.0-"$11(,&; ,#;20 #$;
$6$"32(-,1
4. -,,$62;5(2';-;13..-021;2'$;$,2(0$;!0$ 2';-%;; ,#;*-5;* 2$,"7;%$ 230$1;'(1;(,"*3#$1;
0$*( !*$;"-,,$"2$#;1$04("$;# 2 &0 +;1$04("$;; ,#;1$,#0$"$(4$;1$+ ,2("1; 2-+(";-.$0 2(-,1;
31$0;*$4$*;+3*2(" 12;31$0;*$4$*;; ""$11;)$0,$*;!7. 11; ,#;8$0-;"-.7;;
5. '$;;4$0!1;31$#;!7;-,,$62;5(2';-; 0$;! 1$#;-,;,9,( ,#; ,#;' 4$;!$$,;.0-4$,;(,;
* 0&$;1" *$;#$.*-7+$,21; ,#;5(2';+3*2(.*$;; ..*(" 2(-,1;!-2';(,;2'$;; ,#;;1$"2-01;3"';
..*(" 2(-,1;" ,;,-5;!$;1$ +*$11*7;-%%$0$#;-4$0;-,,$62;5(2';-;5(2'-32; ,7;.-02(,&;$%%-02;
0$/3(0$#
6. -,,$62;5(2';-;! 1$#;,$25-0);+ , &$+$,2;(1;2'$;1 +$; 1;2' 2;%-0; ,7;2'$0,$2; ,#;
! 1$#;,$25-0);+ , &$+$,2;$*(+(, 2(,&;2'$;,$$#;%-0;;+ , &$01;2-;*$ 0,;,$5;2$"',-*-&($1
1.2ConnectX2with
RoCEAdvantages
Body Heading
Univers LT 47 CondensedLt, Bold
12 pt/13 pt
PMS 274 75%
Set as Inline Heading anchored to
beginning of paragraph
Main Title 2
Univers LT 47 CondensedLt
15 pt /15
.75”
.5”
2.625”