User Manual

Features Overview and ConfigurationRev 2.3-1.0.1
Mellanox Technologies
156
Create memory regions, which support the definition of regular non-contiguous memory
regions.
3.2.7.6 On-Demand-Paging (ODP)
On-Demand-Paging (ODP) is a technique to alleviate much of the shortcomings of memory
registration.
Applications no longer need to pin down the underlying physical pages of the
address space, and track the validity of the mappings. Rather, the HCA requests the latest trans-
lations from the OS when pages are not present, and the OS invalidates translations which are
no longer valid due to either non-present pages or mapping changes. ODP does not support the
following features:
Memory windows
Indirect MKEY
Contiguous pages
ODP can be further divided into 2 subclasses: Explicit and Implicit ODP.
Explicit ODP
In Explicit ODP, applications still register memory buffers for communication, but this opera-
tion is used to define access control for IO rather than pin-down the pages. ODP Memory
Region (MR) does not need to have valid mappings at registration time.
Implicit ODP
In Implicit ODP, applications are provided with a special memory key that represents their
complete address space.
This all IO accesses referencing this key (subject to the access rights
associated with the key) does not need to register any virtual address range.
3.2.7.6.1 Query Capabilities
On-Demand Paging is available if both the hardware and the kernel support it. To verify whether
ODP is supported, run
ibv_exp_query_device:
struct ibv_exp_device_attr dattr;
dattr.comp_mask = IBV_EXP_DEVICE_ATTR_ODP | IBV_EXP_DEVICE_ATTR_EXP_CAP_FLAGS;
ret = ibv_exp_query_device(context, &dattr);
if (dattr.exp_device_cap_flags & IBV_EXP_DEVICE_ODP)
//On-Demand Paging is supported.
Each transport has a capability field in the dattr.odp_caps structure that indicates which opera-
tions are supported by the ODP MR:
struct ibv_exp_odp_caps {
uint64_t general_odp_caps;
struct {
uint32_t rc_odp_caps;
uint32_t uc_odp_caps;
uint32_t ud_odp_caps;
uint32_t dc_odp_caps;
uint32_t xrc_odp_caps;
uint32_t raw_eth_odp_caps;
} per_transport_caps;
};