User Manual

Features Overview and ConfigurationRev 2.3-1.0.1
Mellanox Technologies
190
(e.g 1450 instead of 1500), or the uplink NIC MTU has to be incremented by 50 bytes
(e.g 1550 instead of 1500)
• From upstream 3.15-rc1 and onward, it is possible to use arbitrary UDP port for
VXLAN. Note that this requires firmware version 2.31.2800 or higher
. Additionally, you
need to enable this kernel configuration option
CONFIG_MLX4_EN_VXLAN=y.
• On upstream kernels 3.12/3.13 GRO with VXLAN is not supported
3.5 Resiliency
3.5.1 Reset Flow
Reset Flow is activated by default, once a "fatal device
1
" error is recognized. Both the HCA and
the software are reset, the ULPs and user application are notified about it, and a recovery process
is performed once the event is raised.
The "Reset Flow" is activated by the mlx4_core module
parameter
'internal_err_reset', and its default value is 1.
3.5.1.1 Kernel ULPs
Once a "fatal device" error is recognized, an
IB_EVENT_DEVICE_FATAL event is created, ULPs are
notified about the incident, and outstanding WQEs are simulated to be returned with
"flush in
error"
message to enable each ULP to close its resources and not get stuck via calling its
"remove_one" callback as part of "Reset Flow".
Once the unload part is terminated, each ULP is called with its
"add_one" callback, its resources
are re-initialized and it is re-activated.
3.5.1.2 User Space Applications (IB/RoCE)
Once a "fatal device" error is recognized an
IB_EVENT_DEVICE_FATAL event is created, applica-
tions are notified about the incident and relevant recovery actions are taken.
Applications that ignore this event enter a zombie state, where each command sent to the kernel
is returned with an error
, and no completion on outstanding WQEs is expected.
The expected behavior from the applications is to register to receive such events and recover
once the above event is raised. Same behavior is expected in case the NIC is unbounded from the
PCI and later is rebounded.
Applications running over RDMA CM should behave in the same
manner once the
RDMA_CM_EVENT_DEVICE_REMOVAL event is raised.
The below is an example of using the unbind/bind for NIC defined by "0000:04:00.0"
echo 0000:04:00.0 > /sys/bus/pci/drivers/mlx4_core/unbind
echo 0000:04:00.0 > /sys/bus/pci/drivers/mlx4_core/bind
3.5.1.3 SR-IOV
If the Physical Function recognizes the error, it notifies all the VFs about it by marking their
communication channel with that information, consequently
, all the VFs and the PF are reset.
If the VF encounters an error, only that VF is reset, whereas the PF and other VFs continue to
work unaf
fected.
1. A “fatal device” error can be a timeout from a firmware command, an error on a firmware closing command, communication channel not being
responsive in a VF. etc.