User guide

6–SHMEM Description and Configuration
Implementation Behavior
6-16 IB0054606-02 A
For a put operation, these descriptions use the terms "local completion" and
“remote completion”. Once a put is locally complete, the source buffer on the
initiating PE is available for reuse. Until a put is locally complete the source buffer
must not be modified since that buffer is in use for the put operation. A blocking
put is locally complete immediately upon return from the put. A non-blocking put is
not locally complete upon return from the put—different mechanisms are used for
detecting local completion using either an explicit handle (use
shmem_test_nb() or shmem_wait_nb()) or a NULL handle (use
shmem_quiet()). Once a put is remotely complete the destination buffer on the
target PE is fully written and available for use. The mechanism provided by
SHMEM for detecting remote completion are described below.
 shmem_fence() - This function ensures that all puts issued by this PE prior
to the fence will become remotely visible before any puts issued by this PE
after the fence. The call does not necessarily imply that any of the prior puts
are actually remotely visible at the point of the fence, only that this ordering
is guaranteed.
 shmem_quiet() - This function waits for remote completion of all puts
issued by this PE prior to the quiet operation. Therefore, once the quiet
operation returns, it is guaranteed that all those puts will be remotely visible
to other PEs. This guarantee of remote completion applies to all
puts—blocking puts, non-blocking puts with handles, and non-blocking puts
with NULL handles. Additionally, this function additionally waits for local
completion of non-blocking puts and non-blocking gets that were issued with
a NULL handle.
 shmem_test_nb() and shmem_wait_nb() can be used to test and wait
for local completion of a non_blocking operation. For a non-blocking put,
this does not indicate whether remote completion has occurred.
Additional properties of the QLogic SHMEM implementation are:
 The QLogic SHMEM implementation makes no guarantees as to the
ordering in which the bytes of a put operation are delivered into the remote
memory. It is *not* a safe assumption to poll or read certain bytes of the put
destination buffer (for example, the last 8 bytes) to look for a change in value
and then infer that the entirety of the put has arrived. The correct
mechanism for this is to use the shmem_quiet() operation to force remote
completion, or to use the following type of sequence:
 Initiator side:
 Issue a batch of puts all unordered with respect to each other
 shmem_fence()