Users Guide
Table Of Contents
- Table of Contents
- 1 Regulatory and Safety Approvals
- 2 Functional Description
- 3 Network Link and Activity Indication
- 4 Features
- 4.1 Software and Hardware Features
- 4.2 Virtualization Features
- 4.3 VXLAN
- 4.4 NVGRE/GRE/IP-in-IP/Geneve
- 4.5 Stateless Offloads
- 4.6 Priority Flow Control
- 4.7 Virtualization Offload
- 4.8 SR-IOV
- 4.9 Network Partitioning (NPAR)
- 4.10 Security
- 4.11 RDMA over Converged Ethernet – RoCE
- 4.12 VMWare Enhanced Networking Stack (ENS)
- 4.13 Supported Combinations
- 4.14 Unsupported Combinations
- 5 Installing the Hardware
- 6 Software Packages and Installation
- 7 Updating the Firmware
- 8 Link Aggregation
- 9 System-Level Configuration
- 10 PXE Boot
- 11 SR-IOV – Configuration and Use Case Examples
- 12 NPAR – Configuration and Use Case Example
- 13 Tunneling Configuration Examples
- 14 RoCE – Configuration and Use Case Examples
- 15 DCBX – Data Center Bridging
- 16 DPDK – Configuration and Use Case Examples
- Revision History
Broadcom NetXtreme-E-UG304-2CS
116
NetXtreme-E User Guide User Guide for Dell Platforms
Each switch port participating in NCC must be configured with the above ECN threshold for the class of service associated
with RoCEv2 traffic. Different vendors have different naming conventions that specifies the minimum and maximum marking
threshold as well as the marking percentage. The following sections provide examples of setting such parameters
aforementioned.
14.2.5.8.1 Operation
With the switches, servers, and NICs configured, RDMA applications can be deployed to run over the RoCE network. Any
libverbs-linked application can be used with appropriate changes for using IP addresses rather than GUIDs for end-point
addressing.
When using NCC, it is important to specify a RoCEv2 GID. RoCEv1 does not support NCC.
1. To see a list of GIDs supported by the device, use the following ibv_devinfo -vvv:
[root@Host1 ~]# ibv_devinfo -vvvv
hca_id:bnxt_re1
transport:InfiniBand (0)
fw_ver: 212.1.93.0
node_guid:020a:f7ff:fea6:9cf1
sys_image_guid:020a:f7ff:fea6:9cf1
vendor_id:0x14e4
vendor_part_id:5847
hw_ver: 0x1406
phys_port_cnt:1
max_mr_size:0x8000000000
page_size_cap:0x201000
…
active_width:1X (1)
active_speed:25.0 Gb/s (32)
phys_state:DISABLED (3)
GID[ 0]:fe80:0000:0000:0000:020a:f7ff:fea6:9cf0, RoCE V1 (IPV6)
GID[ 1]:fe80:0000:0000:0000:020a:f7ff:fea6:9cf0, RoCE V2 (IPV6)
GID[ 2]:0000:0000:0000:0000:0000:ffff:ac10:0102, RoCE V1 (IPV4)
GID[ 3]:0000:0000:0000:0000:0000:ffff:ac10:0102, RoCE V2 (IPV4)
2. Odd-numbered GIDs are used for RoCEv2 and this can be verified with the following command:
# cat /sys/class/infiniband/bnxt_re0/ports/1/gid_attrs/types/1
NOTE: If the above directory does not exist, this indicates the installed kernel/OFED version does not support RoCE
version 2.
A typical application of exercising the RoCE interface would be to use rping or perftest. Perftest can be obtained from github
or via most Linux package repositories such as:
RHEL/CentOS: yum install perftest
Ubuntu: apt install perftest
25 Gb/s 16/16
50 Gb/s 22/22
100 Gb/s 33/33
Link Speed ECN Min./Max. Threshold (kilobytes)