User Manual

HPC FeaturesRev 2.1-1.0.6
Mellanox Technologies
120
 To upgrade MLNX_OFED v2.0 or later with a newer MXM:
Step 1. Remove MXM v1.1.
Step 2. Remove the pre-compiled OpenMPI.
Step 3. Install the new MXM and compile the OpenMPI with it.
5.3.2 Enabling MXM in OpenMPI
MXM v0.52 is automatically selected by OpenMPI (up to v1.6) when the Number of Processes
(NP) is higher or equal to 128. To enable MXM for any NP use the following OpenMPI parame-
ter
“-mca mtl_mxm_np <number>”.
From OpenMPI v1.7, MXM is selected when the number of processes is higher or equal to 0. i.e.
by default.
 To activate MXM for any NP, run:
5.3.3 Tuning MXM Settings
The default MXM settings are already optimized. To check the available MXM parameters and
their default values, run the
/opt/mellanox/mxm/bin/mxm_dump_config utility which is
part of the MXM RPM.
MXM parameters can be modified in one of the following methods:
• Modifying the default MXM parameters value as part of the mpirun:
• Modifying the default MXM parameters value from SHELL:
rpm -e mxm
rpm -e mlnx-openmpi_gcc
To run OpenMPI without MXM, run:
% mpirun -mca mtl ^mxm <...>
When upgrading to MXM v0.52, OpenMPI compiled with the previous versions of the
MXM should be recompiled with MXM v0.52.
% mpirun -mca mtl_mxm_np 0 <...other mpirun parameters ...>
% mpirun -x MXM_UD_RX_MAX_BUFFERS=128000 <...>
% export MXM_UD_RX_MAX_BUFFERS=128000
% mpirun <...>