User Manual

Rev 2.1-1.0.6

Mellanox Technologies

The underlying physical pages must not be Least Recently Used (LRU) or Anonymous. To

disable that, you need to turn on the

IBV_ACCESS_ALLOCATE_MR bit as part of the

sharing bits.

Usage:

• Turns on via the ibv_reg_mr one or more of the sharing access bits. The sharing bits are part of the

ibv_reg_mr man page.

• Turns on the IBV_ACCESS_ALLOCATE_MR bit

Step 2. Request to register to a shared MR

A new verb called

ibv_reg_shared_mr is added to enable sharing an MR. To use this verb, the

application supplies the MR ID that it wants to register for and the desired access mode to that

MR. The desired access is validated against its given permissions and upon successful creation, the

physical pages of the original MR are shared by the new MR. Once the MR is shared, it can be

used even if the original MR was destroyed.

The request to share the MR can be repeated multiple times and an arbitrary number of Memory

Regions can potentially share the same physical memory locations.

Usage:

• Uses the “handle” field that was returned from the ibv_reg_mr as the mr_handle

• Supplies the desired “access mode” for that MR

• Supplies the address field which can be either NULL or any hint as the required output. The address and

its length are returned as part of the ibv_mr struct.

To achieve high performance it is highly recommended to supply an address that is aligned as the origi-

nal memory region address. Generally, it may be an alignment to 4M address.

For further information on how to use the ibv_reg_shared_mr verb, please refer to the

ibv_reg_shared_mr man page and/or to the ibv_shared_mr sample program which demonstrates

a basic usage of this verb.

Further information on the

ibv_shared_mr sample program can be found in the ibv_shared_mr

man page.

4.11 XRC - eXtended Reliable Connected Transport Service for InfiniBand

XRC allows significant savings in the number of QPs and the associated memory resources

required to establish all to all process connectivity in large clusters.

It significantly improves the scalability of the solution for large clusters of multicore end-nodes

by reducing the required resources.

For further details, please refer to the "Annex A14 Supplement to InfiniBand Architecture Speci-

fication Volume 1.2.1"

A new API can be used by user space applications to work with the XRC transport. The legacy

API is currently supported in both binary and source modes, however it is deprecated. Thus we

recommend using the new API.

The new verbs to be used are:

• ibv_open_xrcd/ibv_close_xrcd

• ibv_create_srq_ex

• ibv_get_srq_num

• ibv_create_qp_ex