User Manual

Rev 2.1-1.0.6
Mellanox Technologies
107
(over InfiniBand/RoCE) application to use peer device computing power, and RDMA intercon-
nect at the same time without copying the data between the P2P devices.
For example, PeerDirect is being used for GPUDirect RDMA.
Detailed description for that API exists under MLNX OFED installation, please see
docs/readme_and_user_manual/PEER_MEMORY_API.txt
4.18 Inline-Receive
When Inline-Receive is active, the HCA may write received data in to the receive WQE or CQE.
Using Inline-Receive saves PCIe read transaction since the HCA does not need to read the
scatter list, therefore it improves performance in case of short receive-messages.
On poll CQ, the driver copies the received data from WQE/CQE to the user's buffers. Therefore,
apart from querying Inline-Receive capability and Inline-Receive activation the feature is trans-
parent to user application.
Connect-IB™ supports Inline-Receive on both the requestor and the responder sides. Since data
is copied at the poll CQ verb, Inline-Receive on the requestor side is possible only if the user
chooses IB(V)_SIGNAL_ALL_WR.
4.18.1 Querying Inline-Receive Capability
User application can use the ibv_exp_query_device function to get the maximum possible
Inline-Receive size. To get the size, the application needs to set the
IBV_EXP_DEVICE_ATTR_INLINE_RECV_SZ bit in the ibv_exp_device_attr comp_mask.
4.18.2 Activating Inline-Receive
To activate the Inline-Receive, you need to set the required message size in the max_inl_recv
field in the ibv_exp_qp_init_attr struct when calling ibv_exp_create_qp function. The
value returned by the same field is the actual Inline-Receive size applied.
4.19 Ethernet Performance Counters
Counters are used to provide information about how well an operating system, an application, a
service, or a driver is performing. The counter data helps determine system bottlenecks and fine-
tune the system and application performance. The operating system, network, and devices pro-
vide counter data that an application can consume to provide users with a graphical view of how
well the system is performing.
When Inline-Receive is active, user application must provide a valid virtual address for
the receive buffers to allow the driver moving the inline-received message to these buf-
fers. The validity of these addresses is not checked therefore the result of providing non-
valid virtual addresses is unexpected.
Setting the message size may affect the WQE/CQE size.