User Manual

Rev 4.70

Mellanox Technologies

The proposed RoCEv2 packets use a well-known UDP destination port value that unequivocally

distinguishes the datagram. Similar to other protocols that use UDP encapsulation, the UDP

source port field is used to carry an opaque flow-identifier that allows network devices to imple-

ment packet forwarding optimizations (e.g. ECMP) while staying agnostic to the specifics of the

protocol header format.

Furthermore, since this change exclusively affects the packet format on the wire, and due to the

fact that with RDMA semantics packets are generated and consumed below the AP applications

can seamlessly operate over any form of RDMA service (including the routable version of RoCE

as shown in Figure 2), in a completely transparent way

Figure 3: RoCEv2 Protocol Stack

8.7.2 RoCE Configuration

In order to function reliably, RoCE requires a form of flow control. While it is possible to use

global flow control, this is normally undesirable, for performance reasons.

The normal and optimal way to use RoCE is to use Priority Flow Control (PFC). To use PFC, it

must be enabled on all endpoints and switches in the flow path.

In the following section we present instructions to configure PFC on Mellanox ConnectX™

cards. There are multiple configuration steps required, all of which may be performed via Power-

Shell. Therefore, although we present each step individually, you may ultimately choose to write

a PowerShell script to do them all in one step. Note that administrator privileges are required for

these steps.

For further information, please refer to:

http://blogs.technet.com/b/josebda/archive/2012/07/31/deploying-windows-server-2012-with-

smb-direct-smb-over-rdma-and-the-mellanox-connectx-3-using-10gbe-40gbe-roce-step-by-

step.aspx

1. Standard RDMA APIs are IP based already for all existing RDMA technologies