User Manual

Rev 4.70

Mellanox Technologies

Step 5. Choose the ‘Tx Throughput Port Arbiter’ option.

Step 6. Set one of the following values:

• Best Effort (Default) - Default behavior. No precedence is given to this port over the other.

• Guaranteed - Give higher precedence to this port.

• Not Present - No configuration exists, defaults are used.

8.7 RDMA over Converged Ethernet (RoCE)

8.7.1 RoCE Overview

Remote Direct Memory Access (RDMA) is the remote memory management capability that

allows server to server data movement directly between application memory without any CPU

involvement. RDMA over Converged Ethernet (RoCE) is a mechanism to provide this efficient

data transfer with very low latencies on loss-less Ethernet networks. With advances in data center

convergence over reliable Ethernet, ConnectX® EN with RoCE uses the proven and efficient

RDMA transport to provide the platform for deploying RDMA technology in mainstream data

center application at 10GigE and 40GigE link-speed. ConnectX® EN with its hardware offload

support takes advantage of this efficient RDMA transport (InfiniBand) services over Ethernet to

deliver ultra-low latency for performance-critical and transaction intensive applications such as

financial, database, storage, and content delivery networks. RoCE encapsulates IB transport and

GRH headers in Ethernet packets bearing a dedicated ether type. While the use of GRH is

optional within InfiniBand subnets, it is mandatory when using RoCE. Applications written over

IB verbs should work seamlessly, but they require provisioning of GRH information when creat-

ing address vectors. The library and driver are modified to provide mapping from GID to MAC

addresses required by the hardware.

8.7.1.1 IP Routable (RoCEv2)

A straightforward extension of the RoCE protocol enables traffic to operate in layer 3 environ-

ments.

This capability is obtained via a simple modification of the RoCE packet format. Instead

of the GRH used in RoCE, routable RoCE packets carry an IP header which allows traversal of

IP L3 Routers and a UDP header that serves as a stateless encapsulation layer for the RDMA

Transport Protocol Packets over IP.

Figure 2: RoCEv2 and RoCE Frame Format Differences