Understanding iWarp

To learn more about iWARP, please visit:
www.intel.com/technology/comms/iWARP
INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL® PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED
BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL’S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER, AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED
WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE,
MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. UNLESS OTHERWISE AGREED IN WRITING
BY INTEL, THE INTEL PRODUCTS ARE NOT DESIGNED NOR INTENDED FOR ANY APPLICATION IN WHICH THE FAILURE OF THE INTEL PRODUCT COULD CREATE A
SITUATION WHERE PERSONAL INJURY OR DEATH MAY OCCUR.
 Intelmaymakechangestospecicationsandproductdescriptionsatanytime,withoutnotice.Designersmustnotrelyontheabsenceorcharacteristicsofanyfeaturesorinstructions
marked“reserved”or“undened.”Intelreservestheseforfuturedenitionandshallhavenoresponsibilitywhatsoeverforconictsorincompatibilitiesarisingfromfuturechangesto
them.Theinformationhereissubjecttochangewithoutnotice.Donotnalizeadesignwiththisinformation.Theproductsdescribedinthisdocumentmaycontaindesigndefectsor
errorsknownaserratawhichmaycausetheproducttodeviatefrompublishedspecications.Currentcharacterizederrataareavailableonrequest.ContactyourlocalIntelsalesofce
oryourdistributortoobtainthelatestspecicationsandbeforeplacingyourproductorder.Copiesofdocumentswhichhaveanordernumberandarereferencedinthisdocument,or
otherIntelliterature,maybeobtainedbycalling1-800-548-4725,orbyvisitingIntel’sWebSitehttp://www.intel.com/.
 *Othernamesandbrandsmaybeclaimedasthepropertyofothers.
 Copyright©2010IntelCorporation.Allrightsreserved.IntelandtheIntellogoaretrademarksorregisteredtrademarksofIntelCorporationoritssubsidiariesintheUnitedStatesand
othercountries.0710/TS/MESH/PDF324032-001US
With traditional Ethernet, the processor dedicates substantial resources
to maintaining the network stack. It must maintain connection context,
segment and reassemble payloads, and process interrupts. This
overhead increases linearly with wire speed, limiting scalability.
The iWARP extensions enable the transport processing to be done in the
network controller. This enables the processor to perform more
application processing, providing a deterministic, low-latency solution that
is optimized for applications that demand low latency.
Under conventional Ethernet, data is copied (and re-cached each time)
by the processor several times as it passes from the server adapter’s
receive buffer to the application buffer. Those operations consume
time and memory bandwidth that the application could otherwise use.
Using RDMA, iWARP enables direct copies from the server adapter’s
receive buffer to the application buffer. This provides a direct data
placement implementation that eliminates the intermediate operations,
which signicantly reduces latency.
Traditionally, when an application issues commands, such as reads and writes,
to a server adapter, those commands are transmitted through user-space
layers of the application to kernel-space layers in the OS stack. This requires
a compute-intensive context switch between user space and the OS.
The iWARP extensions use RDMA to enable the application to post
commands directly to the server adapter. This capability eliminates
expensive calls into the OS, and that lower overhead reduces latency.
How iWARP Reduces Ethernet Overhead and Latency
KernelBypass:Removes the need for context switching
from kernel-space to user-space
DirectDataPlacement:Eliminates intermediate buffer
copies by reading and writing directly to application memory
TransportAcceleration:Performs transport processing on
the network controller instead of the processor
Without iWARP
With iWARP
application
I/O library
I/O adapter iWARP-enabled
Ethernet
I/O adapter
device driver device driver
OSOS
user
kernel
context
switch
server
software
application
I/O library
I/O cmnd
I/O cmnd
I/O cmnd
I/O cmnd
I/O cmnd
I/O cmnd
I/O cmnd
I/O cmnd
TCP/IP
Without iWARP
With iWARP
application
I/O library
I/O adapter
Standard Ethernet TCP/IP packet Ethernet TCP/IP packet
with iWARP extension
device driver device driver
OSOS
user
kernel
context
switch
server
software
application
I/O library
app buffer
OS buffer
driver buffer
adapter buffer
app buffer
adapter buffer
TCP/IP +RDMA
TCP/IP
application
I/O library
I/O adapter
device driver
OS
user
kernel
context
switch
server
software
TCP/IP
OperatingSystemSupport
Support for iWARP is required in operating systems to bring the
benets of RDMA to HPC applications.
• Windows*HPCServer2008: The Network Direct Interface for low-
latency networking is validated for iWARP on NetEffect Ethernet
Server Cluster Adapters from Intel, with support for the Microsoft
Message Passing Interface (MS-MPI) for cluster computing.
• Linux*: The OpenFabrics Enterprise Distribution (OFED*) from
Open Fabrics Alliance provides open source RDMA software for
HPC applications. Linux distributors including Red Hat incorporate
this software in their releases. OFED enables MPI APIs such as
Open MPI, MVAPICH, MVAPICH2, Platform Computing MPI, and
Intel® MPI, as well as popular network storage protocols, including
block storage (iSER) and file storage (NFS-RDMA).
The robust deployment capabilities provided by Windows and
Linux are complemented by third-party offerings, such as
Clustercorp Rocks+*, which help to streamline installation and
management of the software stack.