Installation guide
QLogic OFED+ Host Software Page 11 of 14
5.9 Other Information
When using Mellanox Host Channel Adapters, any changes to Virtual Fabrics (vFabrics)
in the Fabric Manager, may require a reboot of the hosts with Mellanox Host Channel
Adapters. This limitation relates to the Mellanox Host Channel Adapters not properly
responding to changes to the Fabric Manager service level (SL). For some vFabric
configuration changes, if the Fabric Manager SL changes or is mapped to a different
Virtual Lane (VL) than previously, the Mellanox Host Channel Adapter can continue to
use the previous VL. If that VL is presently disabled by the Fabric Manager, future uses
of applications which use the Fabric Manager SL may hang or timeout because there are
no VL Arbitration cycles for that VL. As a result, anytime vFabric configuration is
changed, it is recommended to reboot all hosts with Mellanox Host Channel Adapters so
that the desired Quality of Service (QoS) configuration changes fully take effect. Any
hosts with QLogic Host Channel Adapters will not need to be rebooted.
Due to Mellanox Host Channel Adapters not correctly handling changes to the Fabric
Manager SL, Qlogic recommends that all the hosts using Mellanox ConnectX or
ConnectX-2 Host Channel Adapters be rebooted when used in a virtual fabric
configuration.
When Dispersive Routing is enabled, it allows packets sent using an MPI program run
over PSM to take any one of several routes through a fabric, thus often increasing
performance. The number of routes is determined by the value of 2 to the power of the
Lid Mask Control setting (LMC). Because LMC defaults to 0, the default number of routes
through the fabric is 2
0
or 1. LMC can be set as high as 3, allowing a total number of 23
or 8 routes through the fabric. Providing these additional routes can reduce fabric
congestion, and thus improve performance. Dispersive Routing is supported when
the
Ql
ogic Fabric Manager is used in the fabric. Dispersive Routing is not supported when
using OpenSM.
The recommended tuning for nodes with dual six-core Intel Xeon 5600 Series
(Westmere) processors is to configure the Truescale Host Channel Adapters for 14
contexts, for improved message rate and small message performance. The cfgctxts
driver parameter can be specified in the /etc/modprobe.conf file to configure the
contexts appropriately. For example, add the following line to modprobe.conf:
options ib_qib pcie_caps=0x51 cfgctxts=14
Note that the current release of mvapich2 may have a memory management issue on
certain machines and certain MPI applications. If MPI applications hang when using
MVAPICH2, try using these settings with your application:
MV2_IBA_EAGER_THRESHOLD=16384 MV2_VBUF_TOTAL_SIZE=16384
When using QLogic's sample applications, these settings can be passed using the
ofed.mvapich2.params file (in /opt/iba/src/mpi_apps folder). For other applications,
they can be passed as part of the mpirun command.
When running MVAPICH2, QLogic recommends turning off RDMA fast path. To turn off
RDMA fast path, specify MV2_USE_RDMA_FAST_PATH=0 in the mpirun_rsh command line
or set this option in the parameter file for mvapich2.
Older QLogic InfiniPath 2.2.1 and prior releases will not be recognized in the installation
menus and will be treated as if no InfiniBand software is installed.