Product specifications

C–Integration with a Batch Queuing System
Using SLURM for Batch Queuing
IB6054601-00 H C-3
A
Allocating Resources
When the mpirun command starts, it requires specification of the number of node
programs it must spawn (via the
-np option) and specification of an mpihosts
file listing the nodes on which the node programs may be run. (See “Environment
for Node Programs” on page 5-17 for more information.) Since performance is
usually important, a user might require that his node program be the only
application running on each node CPU. In a typical batch environment, the MPI
user would still specify the number of node programs, but would depend on the
batch system to allocate specific nodes when the required number of CPUs
become available. Thus, batch_mpirun would take at least an argument
specifying the number of node programs and an argument specifying the MPI
program to be executed. For example:
$ batch_mpirun -np n my_mpi_program
After parsing the command line arguments, the next step of batch_mpirun is to
request an allocation of n processors from the batch system. In SLURM, this uses
the command:
eval ‘srun --allocate --ntasks=$np --no-shell‘
Make sure to use back quotes rather than normal single quotes. $np is the shell
variable that your script has set from the parsing of its command line options. The
--no-shell option to srun prevents SLURM from starting a subshell. The srun
command is run with eval to set the SLURM_JOBID shell variable from the output
of the srun command.
With these specified arguments, the SLURM function srun blocks until there are
$np processors available to commit to the caller. When the requested resources
are available, this command opens a new shell and allocates the number of
processors to the requestor.
Generating the mpihosts File
Once the batch system has allocated the required resources, your script must
generate an mpihosts file, which contains a list of nodes that will be used. To do
this, the script must determine which nodes the batch system has allocated, and
how many processes can be started on each node. This is the part of the script
batch_mpirun that performs these tasks, for example:
mpihosts_file=‘mktemp -p /tmp mpihosts_file.XXXXXX‘
srun --jobid=${SLURM_JOBID} hostname -s | sort | uniq -c \
| awk ’{printf "%s:%s\n", $2, $1}’ > $mpihosts_file
The first command creates a temporary hosts file with a random name, and
assigns the name to the variable mpihosts_file it has generated.
The next instance of the SLURM srun command runs hostname -s once for
each process slot that SLURM has allocated. If SLURM has allocated two slots on
one node, hostname -s is output twice for that node.