User Manual
Features Overview and ConfigurationRev 2.3-1.0.1
Mellanox Technologies
160
For example:
truct ibv_exp_device_attr device_attr = {.comp_mask = IBV_EXP_DEVICE_ATTR_RESERVED -
1};
ibv_exp_query_device(context, & device_attr);
if (device_attr.exp_device_cap_flags & IBV_EXP_DEVICE_MEM_WINDOW ||
device_attr.exp_device_cap_flags & IBV_EXP_DEVICE_MW_TYPE_2B) {
/* Memory window is supported */
3.2.7.7.4 Allocating Memory Window
Allocating memory window is done by calling the ibv_alloc_mw verb.
type_mw = IBV_MW_TYPE_2/ IBV_MW_TYPE_1
mw = ibv_alloc_mw(pd, type_mw);
3.2.7.7.5 Binding Memory Windows
After allocated, memory window should be bound to a registered memory region. Memory
Region should have been registered using the
IBV_EXP_ACCESS_MW_BIND access flag.
• Binding Memory Window type 1 is done via the
ibv_exp_bind_mw verb.
struct ibv_
exp_
mw_bind mw_bind = { .comp_mask = IBV_EXP_BIND_MW_RESERVED - 1 };
ret = ibv_
exp_
bind_mw(qp, mw, &mw_bind);
• Binding memory window type 2B is done via the ibv_exp_post_send verb and a spe-
cific Work Request (WR) with
opcode = IBV_EXP_WR_BIND_MW
Prior to binding, please make sure to update the existing rkey.
ibv_inc_rkey(mw->rkey)
3.2.7.7.6 Invalidating Memory Window
Before rebinding Memory Window type 2, it must be invalidated using the ibv_exp_post_send
verb and a specific WR with opcode = IBV_EXP_WR_LOCAL_INV.
3.2.7.7.7 Deallocating Memory Window
Deallocating memory window is done using the ibv_dealloc_mw verb.
ibv_dealloc_mw(mw);
3.2.8 PeerDirect
PeerDirect uses an API between IB CORE and peer memory clients, (e.g. GPU cards) to provide
access to an HCA to read/write peer memory for data buffers. As a result, it allows RDMA-based
(over InfiniBand/RoCE) application to use peer device computing power, and RDMA intercon-
nect at the same time without copying the data between the P2P devices.
For example, PeerDirect is being used for GPUDirect RDMA.
Detailed description for that API exists under MLNX OFED installation, please see
docs/readme_and_user_manual/PEER_MEMORY_API.txt