Users Guide

7–RoCE Configuration
Configuring RoCE on the Adapter for Linux
150 AH0054602-00 M
 On some older server platforms, VF devices may not be enumerated for one
of the NIC PCI functions (PF). This limitation is because of the increased
PCI base address register (BAR) requirements to support VF RDMA,
meaning that the OS/BIOS cannot assign the required BAR for each VF.
 To support the maximum number of QPs in a VM, approximately 8GB of
RAM must be available, assuming that only one VF is assigned to the VM. If
less than 8GB of RAM is assigned to VM, there can be a sudden drop in the
number of active connections due to insufficient memory and memory
allocation failures.
Configuring RoCE on the Adapter for Linux
This section describes the RoCE configuration procedure for RHEL and SLES. It
also describes how to verify the RoCE configuration and provides some guidance
about using group IDs (GIDs) with vLAN interfaces.
 RoCE Configuration for RHEL
 RoCE Configuration for SLES
 Verifying the RoCE Configuration on Linux
 vLAN Interfaces and GID Index Values
 RoCE v2 Configuration for Linux
 Configuring RoCE for SR-IOV VF Devices (VF RDMA)
RoCE Configuration for RHEL
To configure RoCE on the adapter, the Open Fabrics Enterprise Distribution
(OFED) must be installed and configured on the RHEL host.
To prepare inbox OFED for RHEL:
1. While installing or upgrading the operating system, select the InfiniBand and
OFED support packages.
2. Install the following RPMs from the RHEL ISO image:
libibverbs-devel-x.x.x.x86_64.rpm
(required for libqedr library)
perftest-x.x.x.x86_64.rpm
(required for InfiniBand bandwidth and latency applications)
or, using Yum, install the inbox OFED:
yum groupinstall "Infiniband Support"
yum install perftest
yum install tcl tcl-devel tk zlib-devel libibverbs
libibverbs-devel