User Manual

Rev 2.1-1.0.6
Mellanox Technologies
115
5.1.2 Running SHMEM with FCA
The Mellanox Fabric Collective Accelerator (FCA) is a unique solution for offloading collective
operations from the Message Passing Interface (MPI) or ScalableSHMEM process onto Mella-
nox InfiniBand managed switch CPUs. As a system-wide solution, FCA utilizes intelligence on
Mellanox InfiniBand switches, Unified Fabric Manager and MPI nodes without requiring addi-
tional hardware. The FCA manager creates a topology based collective tree, and orchestrates an
efficient collective operation using the switch-based CPUs on the MPI/ScalableSHMEM nodes.
FCA accelerates MPI/ScalableSHMEM collective operation performance by up to 100 times
providing a reduction in the overall job runtime. Implementation is simple and transparent during
the job runtime.
 To enable FCA by default in the ScalableSHMEM:
1. Edit the
/opt/mellanox/openshmem/2.2/etc/openmpi-mca-params.conf file.
2. Set the
scoll_fca_enable parameter to 1.
scoll_fca_enable=1
3. Set the scoll_fca_np parameter to 0.
scoll_fca_np=0
 T
o enable FCA in the shmemrun command line, add the following:
 To disable FCA:
For more details on FCA installation and configuration, please refer to the FCA User Manual
found in the Mellanox website.
5.1.3 Running ScalableSHMEM with MXM
MellanoX Messaging (MXM) library provides enhancements to parallel communication libraries
by fully utilizing the underlying networking infrastructure provided by Mellanox HCA/switch
hardware. This includes a variety of enhancements that take advantage of Mellanox networking
hardware including:
• Multiple transport support including RC, XRC and UD
• Proper management of HCA resources and memory structures
• Efficient memory registration
• One-sided communication semantics
• Connection management
• Receive side tag matching
• Intra-node shared memory communication
FCA is disabled by default and must be configured prior to using it from the Scal-
ableSHMEM.
-mca scoll_fca_enable=1
-mca scoll_fca_enable_np 0
-mca scoll_fca_enable 0 -mca coll_fca_enable 0