User Manual

Features Overview and ConfigurationRev 2.3-1.0.1
Mellanox Technologies
136
3.2.5.2.4 Setting Performance Tuning
• Use 4K MTU over OpenSM.
Default=0xffff, ipoib, mtu=5 : ALL=full;
• Use MTU for 4K (4092 Bytes):
• In UD mode, the maximum MTU value is 4092 Bytes
Make sure that all interfaces (including the guest interface and its virtual bridge) have the same
MTU value (MTU 4092 Bytes). For further information of MTU settings, please refer to the
Hypervisor User Manual.
• Tune the TCP/IP stack using sysctl (dom0/domu)
# /sbin/sysctl_perf_tuning
• Other performance tuning for KVM environment such as vCPU pinning and NUMA
tuning may apply. For further information, please refer to the Hypervisor User Manual.
3.2.5.3 Ethernet over IB (EoIB)
The Ethernet over IB (EoIB) mlx4_vnic module is a network interface implementation over
InfiniBand. EoIB encapsulates Layer 2 datagrams over an InfiniBand Datagram (UD) transport
service.
The InfiniBand UD datagrams encapsulates the entire Ethernet L2 datagram and its pay-
load.
To perform this operation the module performs an address translation from Ethernet layer 2
MAC addresses (48 bits long) to InfiniBand layer 2 addresses made of LID/GID and QPN.
This
translation is totally invisible to the OS and user. Thus, differentiating EoIB from IPoIB which
exposes a 20 Bytes HW address to the OS. The mlx4_vnic module is designed for Mellanox's
ConnectX® family of HCAs and intended to be used with Mellanox's BridgeX® gateway family.
Having a BridgeX gateway is a requirement for using EoIB. It performs the following operations:
• Enables the layer 2 address translation required by the mlx4_vnic module.
• Enables routing of packets from the InfiniBand fabric to a 1 or 10 GigE Ethernet subnet.
3.2.5.3.1 Ethernet over IB Topology
EoIB is designed to work over an InfiniBand fabric and requires the presence of two entities:
• Subnet Manager (SM)
The required subnet manager configuration is not unique to EoIB but rather similar to other Infini-
Band applications and ULPs.
• BridgeX gateway
The BridgeX gateway is at the heart of EoIB. On one side, usually referred to as the "internal" side, it
is connected to the InfiniBand fabric by one or more links. On the other side, usually referred to as the
"external" side, it is connected to the Ethernet subnet by one or more ports.
The Ethernet connections
on the BridgeX's external side are called external ports or eports. Every BridgeX that is in use with
EoIB needs to have one or more eports connected.
3.2.5.3.1.1 External Ports (eports) and Gateway
The combination of a specific BridgeX box and a specific eport is referred to as a gateway. The
gateway is an entity that is visible to the EoIB host driver and is used in the configuration of the
network interfaces on the host side. For example, in the host administered vNics the user will