User guide

6–SHMEM Description and Configuration
Progress Model
IB0054606-02 A 6-11
Alternatively, if $SHMEM_SHMALLOC_BASE_ADDR is specified as 0, then each
SHMEM process will independently choose its own base virtual address for the
global shared memory segment. In this case, the values for a symmetric allocation
using shmalloc() are no longer guaranteed to be identical across the PEs. The
QLogic SHMEM implementation takes care of this asymmetry by using offsets
relative to the base of the symmetric heap in its protocols. However, applications
that interpret symmetric heap pointer values or exchange symmetric heap pointer
values between PEs will not behave as expected.
It is possible for SHMEM to fail at start-up or while allocating global shared
memory due to limits placed by the operating system on the amount of *local*
shared memory that SHMEM can use. Since SHMEM programs can use very
large amounts of memory this can exceed typical OS configurations. As long as
there is sufficient physical memory for the program, the following steps can be
used to solve local shared memory allocation problems:
 Check for low ulimits on memory:
ulimit -l : max locked memory (important for PSM not SHMEM)
ulimit -v : max virtual memory
 Check the contents of these sysctl variables:
sysctl kernel.shmmax ; maximum size of a single shm allocation in
bytes
sysctl kernel.shmall ; maximum size of all shm allocations in “pages”
sysctl kernel.shmnmi ; maximum number of shm segments
 Check the size of /dev/shm:
df /dev/shm
 Check for stale files in /dev/shm:
ls /dev/shm
If any of these checks indicate a problem, ask the cluster administrator to increase
the limit.
Progress Model
QLogic SHMEM supports active and passive progress models. Active progress
means that the PE must actively call into SHMEM for progress to be made on
SHMEM one-sided operations. Passive progress means that progress on
SHMEM one-sided operations can occur without the application needing to call
into SHMEM. Active progress is the default mode of operation for QLogic
SHMEM. Passive progress can be selected using an environment variable where
required.