User Manual

When upgrading to MXM v0.52, Open MPI compiled with the previous versions of the
MXM should be recompiled with MXM v0.52.
HPC FeaturesRev 2.2-1.0.1
Mellanox Technologies
150
5.4.2 Enabling MXM in OpenMPI
As of MXM v2.1, MXM is automatically selected by Open MPI (up to v1.6) when the Number
of Processes (NP) is higher or equal to 128.
 To activate MXM for any NP, run:
% mpirun -mca mtl_mxm_np 0 <...other mpirun parameters ...>
From Open MPI v1.7.x, MXM is selected when the number of processes is higher or equal to 0.
i.e. by default.
5.4.3 Tuning MXM Settings
The default MXM settings are already optimized. To check the available MXM parameters and
their default values, run the
/opt/mellanox/mxm/bin/mxm_dump_config -f utility which is part
of the MXM RPM.
MXM parameters can be modified in one of the following methods:
• Modifying the default MXM parameters value as part of the mpirun:
% mpirun -x MXM_UD_RX_MAX_BUFFS=128000 <...>
• Modifying the default MXM parameters value from SHELL:
% export MXM_UD_RX_MAX_BUFFS=128000
% mpirun <...>
5.4.4 Configuring Multi-Rail Support
Multi-Rail support enables the user to use more than one of the active ports on the card, by mak-
ing a better use of the resources. It provides a combined throughput among the used ports.
Multi-Rail support in MXM v0.52 allows different processes on the same host to use different
active ports. Every process can only use one port (as opposed to MXM v1.5).
 To configure dual rail support:
• Specify the list of ports you would like to use to enable multi rail support.
-x MXM_RDMA_PORTS=cardName:portNum
or
-x MXM_IB_PORTS=cardName:portNum
For example:
-x MXM_IB_PORTS=mlx5_0:1
It is also possible to use several HCAs and ports during the run (separated by a comma):
-x MXM_IB_PORTS=mlx5_0:1,mlx5_1:1
MXM will bind a process to one of the HCA ports from the given ports list according to the
MXM_IB_MAP_MODE parameter (for load balancing).