User Manual

Rev 2.3-1.0.1

Mellanox Technologies

• Ports facing the network should be configured as trunk ports, and use Port Control Proto-

col (PCP) for priority flow control

For further information on how to configure SwitchX, please refer to SwitchX User Manual.

3.1.6.2.3 Configuring the RoCE Mode

RoCE mode is configured via the module parameter, roce_mode, in mlx4_core kernel module.

The value can be set globally and it will be applied to all HCAs on the node, or specifically to an

HCA (identified by BDF). The recognized values for

roce_modes are

• 0 - RoCE MAC based (default)

• 1 - RoCE IP based

• 2 - RoCEv2

3.1.6.2.4 Configuring DAPL over RoCE

The default dat.conf file which contains entries for the DAPL devices, does not contain entries

for the DAPL over RDMA_CM over RoCE devices.

 To add the missing entries:

Step 1. Run the ibdev2netdev utility to see all the associations between the Ethernet devices and the

IB devices/ports.

Step 2. Add a new entry line according to the format below to the dat.conf file for each output line

of the ibdev2netdev utility.

<IA Name> u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 "<ethX> <port>" ""

Parameter Description Example

<IA Name> The device's IA name. The name must be unique. ofa-v2-ethx

<ethX> The associated Ethernet device used by RoCE. eth3

<port> The port number. 1

The following is an example of the ibdev2netdev utility's output and the entries added per

each output line:

sw419:~ # ibdev2netdev mlx4_0 port 2 <===> eth2 mlx4_0 port 1 <===> eth3

ofa-v2-eth2 u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 "eth2 2" ""

ofa-v2-eth3 u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 "eth3 1" ""

The next section provides information of how to use InfiniBand over Ethernet (RoCE).

3.1.6.2.5 Installing and Loading the Driver

 To install and load the driver:

Step 1. Install MLNX_OFED (See section Section 2.3, “Installing Mellanox OFED”, on page 23

for details on installation.)

RoCE is installed as part of mlx4 and mlx4_en and other modules upon driver’s installa-

tion.

The list of the modules that will be loaded automatically upon boot can be found in the

configuration file /etc/infiniband/openib.conf.