User Manual

Rev 2.0-3.0.0

Mellanox Technologies

4.13.7.3.2Additional Ethernet VF Configuration Options

• Guest MAC configuration

By default, guest MAC addresses are configured to be all zeroes. In the mlnx_ofed guest driver,

if a guest sees a zero MAC, it generates a random MAC address for itself. If the administrator wishes

the guest to always start up with the same MAC, he/she should configure guest MACs before the

guest driver comes up.

The guest MAC may be configured by using:

For legacy guests, which do not generate random MACs, the adminstrator should always configure

their MAC addresses via ip link, as above.

• Spoof checking

Spoof checking is currently available only on upstream kernels newer than 3.1.

4.13.7.3.3RoCE Support

RoCE is supported on VFs and VLANs may be used. For RoCE, the hypervisor can support

RoCE over up to 15 vlans. There are 127 vlans available per port (for the Hypervisor all guests

together). The Hypervisor is allocated 16 GIDs, which can support 15 VLANs. The remaining

VLANs are allocated equally among the number of VFs requested in the

"num_vfs" mlx4_core

module parameter.

4.14 CORE-Direct

4.14.1 CORE-Direct Overview

CORE-Direct provides a solution for off loading the MPI collectives operations from the soft-

ware library to the network. CORE-Direct accelerates MPI applications and solves the scalability

issues in large scale systems by eliminating the issues of operating systems noise and jitter.

It addresses the collectives communication scalability problem by off loading a sequence of data-

dependent communications to the Host Channel Adapter (HCA). This solution provides the

hooks needed to support computation and communication overlap. Additionally, it provides a

means to reduce the effects of system noise and application skew on application scalability.

The relevant verbs to be used for CORE-Direct:

• ibv_create_qp_ex

• ibv_modify_cq

• ibv_query_device_ex

• ibv_post_task

Samples programs for reference:

• ibv_task_pingpong, ibv_cc_pingpong

ip link set dev <PF device> vf <NUM> mac <LLADDR>

ip link set dev <PF device> vf <NUM> spoofchk [on | off]

RoCE VLANs will not work in VST mode (packets will simply not be sent nor will they

arrive).