User Manual

Rev 2.2-1.0.1
Mellanox Technologies
135
4.20 PeerDirect
PeerDirect uses an API between IB CORE and peer memory clients, (e.g. GPU cards) to provide
access to an HCA to read/write peer memory for data buffers. As a result, it allows RDMA-based
(over InfiniBand/RoCE) application to use peer device computing power, and RDMA intercon-
nect at the same time without copying the data between the P2P devices.
For example, PeerDirect is being used for GPUDirect RDMA.
Detailed description for that API exists under MLNX OFED installation, please see
docs/readme_and_user_manual/PEER_MEMORY_API.txt
4.21 Inline-Receive
When Inline-Receive is active, the HCA may write received data in to the receive WQE or CQE.
Using Inline-Receive saves PCIe read transaction since the HCA does not need to read the
scatter list, therefore it improves performance in case of short receive-messages.
On poll CQ, the driver copies the received data from WQE/CQE to the user's buffers. Therefore,
apart from querying Inline-Receive capability and Inline-Receive activation the feature is trans-
parent to user application.
When Inline-Receive is active, user application must provide a valid virtual address for
the receive buffers to allow the driver moving the inline-received message to these buf-
fers. The validity of these addresses is not checked therefore the result of providing non-
valid virtual addresses is unexpected.
Connect-IB™ supports Inline-Receive on both the requestor and the responder sides. Since data
is copied at the poll CQ verb, Inline-Receive on the requestor side is possible only if the user
chooses IB(V)_SIGNAL_ALL_WR.
4.21.1 Querying Inline-Receive Capability
User application can use the ibv_exp_query_device function to get the maximum possible
Inline-Receive size. To get the size, the application needs to set the
IBV_EXP_DEVICE_ATTR_INLINE_RECV_SZ bit in the ibv_exp_device_attr comp_mask.
4.21.2 Activating Inline-Receive
To activate the Inline-Receive, you need to set the required message size in the max_inl_recv
field in the ibv_exp_qp_init_attr struct when calling ibv_exp_create_qp function. The
value returned by the same field is the actual Inline-Receive size applied.
Setting the message size may affect the WQE/CQE size.