User guide

4–Running MPI on QLogic Adapters
Open MPI
IB0054606-02 A 4-9
Performance can be improved in some cases by disabling IB hardware contexts
when they are not required so that the resources can be partitioned more
effectively.
To disable this behavior, explicitly configure for the number you want to use with
the cfgctxts module parameter in the modprobe configuration file (see
“Affected Files” on page 3-37 for exact file name and location).
The maximum that can be set is 18 on QDR IB Adapters.
The driver must be restarted if this default is changed. See “Managing the ib_qib
Driver” on page 3-21.
Enabling and Disabling Software Context Sharing
By default, context sharing is enabled; it can also be specifically disabled.
Context Sharing Enabled: The MPI library provides PSM the local process
layout so that IB contexts available on each node can be shared if necessary; for
example, when running more node programs than contexts. All PSM jobs assume
that they can make use of all available IB contexts to satisfy the job requirement
and try to give a context to each process.
When context sharing is enabled on a system with multiple QLogic IB adapter
boards (units) and the IPATH_UNIT environment variable is set, the number of IB
contexts made available to MPI jobs is restricted to the number of contexts
available on that unit. When multiple IB devices are present, it restricts the use to
a specific IB Adapter unit. By default, all configured units are used in round robin
order.
Context Sharing Disabled: Each node program tries to obtain exclusive access
to an IB hardware context. If no hardware contexts are available, the job aborts.
NOTE
In rare cases, setting contexts automatically on QDR IB Adapters can lead
to sub-optimal performance where one or more IB hardware contexts have
been disabled and a job is run that requires software context sharing. Since
the algorithm ensures that there is at least one IB context per online CPU,
this case occurs only if the CPUs are over-subscribed with processes (which
is not normally recommended). In this case, it is best to override the default
to use as many IB contexts as are available, which minimizes the amount of
software context sharing required.