User Manual

Rev 2.2-1.0.1

Mellanox Technologies

This tool burns a firmware binary image to the EEPROM(s) attached to an InfiniScaleIII®

switch device. It includes query functions to the burnt firmware image and to the binary image

file. The tool accesses the EEPROM and/or switch device via an I2C-compatible interface or

via vendor-specific MADs over the InfiniBand fabric (In-Band tool).

• Debug utilities

A set of debug utilities (e.g., itrace, mstdump, isw, and i2c)

For additional details, please refer to the MFT User’s Manual docs/.

1.4 Quality of Service

Quality of Service (QoS) requirements stem from the realization of I/O consolidation over an IB

and Eth network. As multiple applications and ULPs share the same fabric, a means is needed to

control their use of network resources.

QoS over Mellanox OFED for Linux is discussed in Chapter 8, “OpenSM – Subnet Manager”.

1.5 RDMA over Converged Ethernet (RoCE)

RoCE allows InfiniBand (IB) transport applications to work over Ethernet network. RoCE

encapsulates the InfiniBand transport and the GRH headers in Ethernet packets bearing a dedi

cated ether type (0x8195). Thus, any VERB application that works in an InfiniBand fabric can

work in an Ethernet fabric as well.

RoCE is enabled only for drivers that support VPI (currently, only mlx4). When working with

RDMA applications over Ethernet link layer the following points should be noted:

• The presence of a Subnet Manager (SM) is not required in the fabric. Thus, operations

that require communication with the SM are managed in a different way in RoCE. This

does not affect the API but only the actions such as joining multicast group, that need to

be taken when using the API

• Since LID is a layer 2 attribute of the InfiniBand protocol stack, it is not set for a port

and is displayed as zero when querying the port

• With RoCE, the alternate path is not set for RC QP and therefore APM is not supported

• Since the SM is not present, querying a path is impossible. Therefore, the path record

structure must be filled with the relevant values before establishing a connection.

Hence, it is recommended working with RDMA-CM to establish a connection as it

takes care of filling the path record structure

• The GID table for each port is populated with N+1 entries where N is the number of IP

addresses that are assigned to all network devices associated with the port including

VLAN devices, alias devices and bonding masters. The only exception to this rule is a

bonding master of a slave in a DOWN state. In that case, a matching GID to the IP

address of the master will not be present in the GID table of the slave's port

• The first entry in the GID table (at index 0) for each port is always present and equal to

the link local IPv6 address of the net device that is associated with the port. Note that

even if the link local IPv6 address is not set, index 0 is still populated.