User Manual

HPC FeaturesRev 2.0-3.0.0
Mellanox Technologies
104
 To upgrade MLNX_OFED v2.0 or later with a newer MXM:
Step 1. Remove MXM v1.1.
Step 2. Remove the pre-compiled OpenMPI.
Step 3. Install the new MXM and compile the OpenMPI with it.
5.3.2 Enabling MXM in OpenMPI
MXM Rev 2.0-3.0.0 is automatically selected by OpenMPI when the Number of Processes (NP)
is higher or equal to 128. To enable MXM for any NP use the following OpenMPI parameter
“-
mca mtl_mxm_np <number>”.
 To activate MXM for any NP, run:
5.3.3 Tuning MXM Settings
The default MXM settings are already optimized. To check the available MXM parameters and
their default values, run the
/opt/mellanox/mxm/bin/mxm_dump_config utility which is
part of the MXM RPM.
MXM parameters can be modified in one of the following methods:
• Modifying the default MXM parameters value as part of the mpirun:
• Modifying the default MXM parameters value from SHELL:
rpm -e mxm
rpm -e mlnx-openmpi_gcc
To run OpenMPI without MXM, run:
% mpirun -mca mtl ^mxm <...>
When upgrading to MXM v1.5, OMPI compiled with MXM v1.1 should be recompiled
with MXM v1.5.
% mpirun -mca mtl_mxm_np 0 <...other mpirun parameters ...>
% mpirun -x MXM_UD_RX_MAX_BUFFERS=128000 <...>
% export MXM_UD_RX_MAX_BUFFERS=128000
% mpirun <...>