User Manual

Rev 2.3-1.0.1

Mellanox Technologies

• Since LID is a layer 2 attribute of the InfiniBand protocol stack, it is not set for a port

and is displayed as zero when querying the port

• With RoCE, the alternate path is not set for RC QP and therefore APM is not supported

• Since the SM is not present, querying a path is impossible. Therefore, the path record

structure must be filled with the relevant values before establishing a connection. Hence,

it is recommended working with RDMA-CM to establish a connection as it takes care of

filling the path record structure

• The GID table for each port is populated with N+1 entries where N is the number of IP

addresses that are assigned to all network devices associated with the port including

VLAN devices, alias devices and bonding masters.

The only exception to this rule is a

bonding master of a slave in a DOWN state. In that case, a matching GID to the IP

address of the master will not be present in the GID table of the slave's port

• The first entry in the GID table (at index 0) for each port is always present and equal to

the link local IPv6 address of the net device that is associated with the port. Note that

even if the link local IPv6 address is not set, index 0 is still populated.

• GID format can be of 2 types, IPv4 and IPv6. IPv4 GID is a IPv4-mapped IPv6 address

while IPv6 GID is the IPv6 address itself

• VLAN tagged Ethernet frames carry a 3-bit priority field. The value of this field is

derived from the IB SL field by taking the 3 least significant bits of the SL field

• RoCE traffic is not shown in the associated Ethernet device's counters since it is

floaded by the hardware and does not go through Ethernet network driver. RoCE traf-

fic is counted in the same place where InfiniBand traffic is counted; /sys/class/infini-

band/<device>/ports/<port number>/counters/

3.1.6.1 IP Routable RoCE

RoCE v2 is supported only in ConnectX®-3 Pro adapter cards.

A straightforward extension of the RoCE protocol enables traffic to operate in IP layer 3 environ-

ments. This capability is obtained via a simple modification of the RoCE packet format. Instead

of the GRH used in RoCE, IP routable RoCE packets carry an IP header which allows traversal

of IP L3 Routers and a UDP header (RoCEv2 only) that serves as a stateless encapsulation layer

for the RDMA

Transport Protocol Packets over IP.

The proposed RoCEv2 packets use a well-known UDP destination port value that unequivocally

distinguishes the datagram. Similar to other protocols that use UDP encapsulation, the UDP

source port field is used to carry an opaque flow-identifier that allows network devices to imple-

ment packet forwarding optimizations (e.g. ECMP) while staying agnostic to the specifics of the

protocol header format.

Furthermore, since this change exclusively affects the packet format on the wire, and due to the

fact that with RDMA semantics packets are generated and consumed below the

AP, applications

can seamlessly operate over any form of RDMA service (including the routable version of RoCE

as shown the figure below), in a completely transparent way.

1. For the IPv4 address A.B.C.D the corresponding IPv4-mapped IPv6 address is ::ffff.A.B.C.D