User guide

6–SHMEM Description and Configuration
Sizing Global Shared Memory
6-10 IB0054606-02 A
The QLogic SHMEM library pre-allocates room in the virtual address space
according to $SHMEM_SHMALLOC_MAX_SIZE (default of 4GB). It then populates
this with enough pages to cover $SHMEM_SHMALLOC_INIT_SIZE (default
16MB). The global shared memory segment can then grow dynamically from its
initial size up to its maximum size. If an allocation attempts to exceed the
maximum size allocations are no longer guaranteed to succeed, and will fail if
there is no room in the virtual memory space of the process following the global
shared memory segment. Upon failure the call to shmalloc or shrealloc
returns NULL. The only down-side of using a large maximum size is occupancy of
virtual address space (48 bits for 64-bit processes is very plentiful), and set-up of
page table entries by the OS. A reasonable limit is 4GB per process. One
side-effect of this approach is that SHMEM programs consume a large amount of
virtual memory when viewed with the "top" program. This is due to the large
maximum size setting. The RES field of top indicates the actual amount of
memory that is resident in memory (for example, in actual use).
If a SHMEM application program runs out of global shared memory, increase the
value of $SHMEM_SHMALLOC_MAX_SIZE. The value of
$SHMEM_SHMALLOC_INIT_SIZE can also be changed to pre-allocate more
memory up front rather than dynamically.
By default QLogic SHMEM will use the same base address for the symmetric
heap across all PEs in the job. This address can be changed using the
$SHMEM_SHMALLOC_BASE_ADDR environment variable. It will be rounded up to
the nearest multiple of the page size. The virtual address range specified by this
base address and the maximum size must not clash with any other memory
mapping. If any SHMEM process in a job has a memory mapping clash, the
QLogic SHMEM library will fail during shmem_init(). With 64-bit programs, a
large virtual address space (for example, 48 bits in many modern processors) and
a reasonably homogeneous cluster, it is expected that such failures will be rare.
The default value of $SHMEM_SHMALLOC_BASE_ADDR has been chosen to work
on the supported distributions and processors. In the rare event of a failure, the
value of $SHMEM_SHMALLOC_BASE_ADDR can be changed using the
environment variable.
NOTE
There is a connection between the sizing of the global shared memory and
local shared memory because of the mechanism used for accessing global
shared memory in a PE that happens to be on the same host.