User Manual

Rev 4.70
Mellanox Technologies
165
Suggestion 1: Go to "Device Manager", locate the Mellanox adapter that you are debugging,
right-click and go to "Information":
• PCI Gen 2: should appear as "PCI-E 5.0 GT/s
• PCI Gen 3: should appear as "PCI-E 8.0 GT/s"
• Link Speed: 40.0Gbps/10.0Gbps
Suggestion 2: To determine if the Mellanox NIC and PCI bus can achieve their maximum
speed, it's best to run ib_send_bw in a loopback. On the same machine:
1. Run "start /b /affinity 0x1 ibv_write_bw"
2. Run "start /b /affinity 0x2 ibv_write_bw 127.0.0.1"
3. Repeat for port 2 with additional -p2, and for other cards if necessary.
4. On PCI Gen3 the expected result is around 5700MB/s
On PCI Gen2 the expected result is around 3300MB/s
Any number lower than that points to bad configuration or installation on the wrong PCI slot.
Malfunctioning QoS settings and Flow Control can be the cause as well.
Suggestion 3: To determine the maximum speed between the two sides with the most basic
test:
1. Run "ib_send_bw" on machine 1
2. Run "ib_send_bw <host1>" on machine 2 where <host1> is the hostname for
machine 1.
3. Results appear in MB/s (Mega Bytes 2^20), and reflect the actual data that was
transferred, excluding headers.
4. If these results are not as expected, the problem is most probably with one or more
of the following:
• Old Firmware version.
• Misconfigured Flow-control: Global pause or PFC is configured wrong on the hosts, routers and-
switches. See Section 8.7,“RDMA over Converged Ethernet (RoCE),” on page 48
• CPU/power options are not set to "Maximum Performance".
Issue 3. QoS and Flow-control
Flow control settings can greatly affect results. In order to see configured settings for all of the
QoS options, open a PowerShell prompt and use "Get-NetAdapterQos"
To achieve maximum performance all of the following must exist:
1. All of the hosts, switches and routers should use the same matching flow control
settings. If Global-pause is used, all devices must be configured for it. If PFC (Prior-
ity Flow-control) is used all devices must have matching settings for all priorities.
2. ETS settings that limit speed of some priorities will greatly affect the output results.
3. Make sure Flow-Control is enabled on the Mellanox Interfaces (enabled by default).
Go to the device manager, right click the Mellanox interface go to "Advanced" and
make sure Flow-control is enabled for both TX and RX.
4. To eliminate QoS and Flow-control as the performance degrading factor, set all
devices to run with Global Pause and rerun the tests:
• Set Global pause on the switches, routers.
• Run "Disable-NetAdapterQos *" on all of the hosts in a PowerShell window.