User guide

6–SHMEM Description and Configuration
QLogic SHMEM Relationship with MPI
IB0054606-02 A 6-7
QLogic SHMEM Relationship with MPI
QLogic SHMEM requires the QLogic PSM layer to provide the network transport
function and this runs exclusively on QLogic IB HCAs. It also requires a
compatible MPI implementation (also running over PSM) to provide program start
up and other miscellaneous services. The one-sided operations in QLogic
SHMEM are not layered on top of MPI, however, and go directly to PSM to give
low-latency, high-performance access to the HCA architecture.
Typical SHMEM programs are written using calls to the SHMEM API and do not
use MPI calls. In this case the program binary generated by shmemcc contains
references to the SHMEM dynamic library and no references at all to MPI
libraries. These binaries are portable across all MPI implementations supported
by QLogic SHMEM. This is true of the get/put micro-benchmarks provided by
QLogic SHMEM. The desired MPI can be selected at run time simply by placing
the desired mpirun on $PATH, or by using the $SHMEM_MPIRUN environment
variable.
Alternatively, it is possible to write hybrid SHMEM/MPI programs that use features
from both the SHMEM and MPI libraries. These programs must call
shmem_init() to initialize the SHMEM library state. They may also use
MPI_Init() and MPI_Finalize() if needed. There will be a direct one-to-one
correspondence between the SHMEM and MPI_COMM_WORLD rank assignments:
shmem_my_pe() will match MPI_Comm_rank() on MPI_COMM_WORLD
shmem_n_pes() will match MPI_Comm_size() on MPI_COMM_WORLD
Hybrid SHMEM/MPI programs must be linked against SHMEM libraries and the
correct MPI libraries. It is recommended that the implementation of the MPI
wrapper script(s) (mpicc) is used for compilation and that additional options are
specified to find the SHMEM include and library files. One approach is to set up
the shmemcc wrapper script to use mpicc as its compiler using the environment
variable setting:
export SHMEM_CC=mpicc
This setting needs to be adjusted if mpicc is not already on the $PATH. The
generated binary has references to both SHMEM and MPI libraries and is specific
to that MPI implementation. QLogic recommends that shmemrun is used to run
the program. The user must ensure that the correct mpirun is picked up from
$PATH or using the $SHMEM_MPIRUN environment variable.