User guide

4–Running MPI on QLogic Adapters
Open MPI
4-10 IB0054606-02 A
To explicitly disable context sharing, set this environment variable in one of the
two following ways:
PSM_SHAREDCONTEXTS=0
PSM_SHAREDCONTEXTS=NO
The default value of PSM_SHAREDCONTEXTS is 1 (enabled).
Restricting IB Hardware Contexts in a Batch Environment
If required for resource sharing between multiple jobs in batch systems, you can
restrict the number of IB hardware contexts that are made available on each node
of an MPI job by setting that number in the PSM_SHAREDCONTEXTS_MAX or
PSM_RANKS_PER_CONTEXT environment variables.
For example, if you are running two different jobs on nodes using a QDR IB HCA,
set PSM_SHAREDCONTEXTS_MAX to 8 instead of the default 16. Each job would
then have at most 8 of the 16 available hardware contexts. Both of the jobs that
want to share a node would have to set PSM_SHAREDCONTEXTS_MAX=8.
Setting PSM_SHAREDCONTEXTS_MAX=8 as a clusterwide default would
unnecessarily penalize nodes that are dedicated to running single jobs. QLogic
recommends that a per-node setting, or some level of coordination with the job
scheduler with setting the environment variable should be used.
The number of contexts can be explicitly configured with the cfgctxts module
parameter. This will override the default settings based on the number of CPUs
present on each node. See “IB Hardware Contexts on the QDR IB Adapters” on
page 4-8.
PSM_RANKS_PER_CONTEXT provides an alternate way of specifying how PSM
should use contexts. The variable is the number of ranks that will share each
hardware context. The supported values are 1, 2, 3 and 4, where 1 is no context
sharing, 2 is 2-way context sharing, 3 is 3-way context sharing and 4 is the
maximum 4-way context sharing. The same value of PSM_RANKS_PER_CONTEXT
must be used for all ranks on a node, and typically, you would use the same value
for all nodes in that job. Either PSM_RANKS_PER_CONTEXT or
PSM_SHAREDCONTEXTS_MAX would be used in a particular job, but not both. If
both are used and the settings are incompatible, then PSM will report an error and
the job will fail to start up.
NOTE
MPIs use different methods for propagating environment variables to the
nodes used for the job; See Section 7 for examples. Open MPI will
automatically propagate PSM environment variables.