User guide

6–SHMEM Description and Configuration
Slurm Integration
6-8 IB0054606-02 A
Slurm Integration
QLogic SHMEM relies on an MPI implementation to provide a run-time
environment for jobs. This includes job start-up, stdin/stdout/stderr routing, and
other low performance control mechanisms. QLogic SHMEM programs are
typically started using shmemrun which is a wrapper script around mpirun. The
shmemrun script takes care of setting up the environment appropriately, and also
provides a common command-line interface regardless of which underlying
mpirun is used.
Integration of QLogic SHMEM with slurm comes from the slurm integration
provided by the MPI implementation. The slurm web pages describe 3
approaches. Please refer to points 1, 2 and 3 on the following web-page:
https://computing.llnl.gov/linux/slurm/mpi_guide.html
Below are various options for integration of the QLogic SHMEM and slurm.
Full Integration
This approach fully integrates QLogic SHMEM start-up into slurm and is
available when running over MVAPICH2. The SHMEM program is executed using
srun directly. For example:
srun -N 16 shmem-test-world
To run a program on 16 nodes. slurm starts the processes using slurmd and
provides communication initialization. The implementation typically relies on
slurm provided a process management interface (PMI) library and the MPI
implementation using that so that each MPI process can hook into slurm.
The user is responsible for setting up the environment appropriately. This includes
adding QLogic SHMEM's library directory to LD_LIBRARY_PATH. See Running
SHMEM Programs for more information on the environment setup.
Two-step Integration
This approach is integrated, but is performed in 2 steps to allocate the nodes and
run the job. This is available when running over Open MPI. The run command is
now:
salloc -N 16 shmemrun shmem-test-world