User Manual
Mellanox OFED OverviewRev 2.2-1.0.1
Mellanox Technologies
30
Notes about the output above:
The ports’ states are:
• Port 1 - InfiniBand, is in PORT_INIT state
• Port 2 - Ethernet is in PORT_ACTIVE state
The port state can also be obtained by running the
following commands:
# cat /sys/class/infiniband/mlx4_0/ports/
1/state 2: INIT
# cat /sys/class/infiniband/mlx4_0/ports/
2/state 4: ACTIVE
#
The link_layer parameter shows that:
• Port 1 is InfiniBand
• Port 2 is Ethernet
Nevertheless, Port 2 appears in the list of
the HCA's ports.
The link_layer of the two ports can also be obtained
by running the following commands:
# cat /sys/class/infiniband/mlx4_0/ports/
1/link_layer InfiniBand
# cat /sys/class/infiniband/mlx4_0/ports/
2/link_layer Ethernet
#
The fw_ver parameter shows that the firm-
ware version is 2.31.5050.
The firmware version can also be obtained by run-
ning the following commands:
# cat /sys/class/infiniband/mlx4_0/fw_ver
2.31.5050
#
Although the InfiniBand over Ethernet's
Port MTU is 2K byte at maximum, the
actual MTU cannot exceed the mlx4_en
interface's MTU. Since the mlx4_en inter-
face's MTU is typically 1560, port 2 will
run with MTU of 1K.
Please note that RoCE's MTU are subject
to InfiniBand MTU restrictions.
The
RoCE's MTU values are, 256 byte, 512
byte, 1024 byte and 2K. In general RoCE
MTU is the largest power of 2 that is still
lower than mlx4_en interface MTU.
-
1.5.2.2 Associating InfiniBand Ports to Ethernet Ports
Since both RoCE and mlx4_en use the Ethernet port of the adapter, one of the drivers must con-
trol the port state. In the example above, the mlx4_en driver controls the port’s state. The
mlx4_ib driver holds a reference to the mlx4_en net device for getting notifications about the
state of the port, as well as using the mlx4_en driver to resolve IP addresses to MAC that are
required for address vector creation. However
, RoCE traffic does not go through the mlx4_en
driver, it is completely offloaded by the hardware.
# ibdev2netdev
mlx4_0 port 2 <===> eth2
mlx4_0 port 1 <===> ib0
#