User Manual

ManualsBrandsAsus ManualsServer AccessoriesPEM-FDR

www.mellanox.com

Mellanox WinOF VPI User Manual

Rev 4.70

Summary of content (192 pages)

PAGE 1
Mellanox WinOF VPI User Manual Rev 4.70 www.mellanox.
PAGE 2
Rev 4.
PAGE 3
Rev 4.70 Table of Contents Revision History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 About this Manual . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .13 Intended Audience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
PAGE 4
Rev 4.70 8.5.3 Creating a Port VLAN in Windows 2008 R2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 8.5.4 Removing a Port VLAN in Windows 2008 R2 . . . . . . . . . . . . . . . . . . . . . . . . . . 46 8.5.5 Configuring a Port to Work with VLAN in Windows 2012 and Above . . . . . . . 47 8.6 8.7 Ports TX Arbitration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 RDMA over Converged Ethernet (RoCE). . . . . . . . . . . . . . . . . . . . . . . . .
PAGE 5
Rev 4.70 Chapter 9 Booting Windows from an iSCSI Target . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 9.1 Configuring the WDS, DHCP and iSCSI Servers . . . . . . . . . . . . . . . . . . . . . . . . 77 9.1.1 Configuring the WDS Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 9.1.2 Configuring iSCSI Target . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 9.1.3 Configuring the DHCP Server . . . . . . . . . . . . .
PAGE 6
Rev 4.70 14.3.10 ibtracert . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.3.11 sminfo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.3.12 ibclearerrors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.3.13 ibstat. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
PAGE 7
Rev 4.70 List of Tables Table 1 Table 2 Table 3 Table 4 Table 5 Table 6 Table 7 Table 8 Table 9 Table 10 Table 11 Table 12 Table 13 Table 14 Table 15 Table 16 Table 17 Table 18 Table 19 Table 20 Table 21 Table 22 Table 23 Table 24 Table 25 Table 26 Table 27 Table 28 Table 29 Table 30 Table 31 Table 32 Table 33 Table 34 Table 35 Table 36 Table 37 Table 38 Table 39 Table 40 Table 41 Table 42 Table 43 Revision History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
PAGE 8
Rev 4.70 Table 44 Table 45 Table 46 Table 47 Table 48 Table 49 Table 50 Table 51 Table 52 Table 53 Table 54 Table 55 Table 56 Table 57 ibv_send_bw Flags and Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149 ibv_send_lat Flags and Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151 ibv_write_bw Flags and Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152 ibv_write_lat Flags and Options . . . . . . . . . . . . . .
PAGE 9
Rev 4.70 Revision History Table 1 - Revision History Document Revision Rev 4.70 Date June 29, 2014 Changes Updated the following section: • May 4, 2014 Section 8.7.2.2.1, “Using Global Pause Flow Control (GFC)”, on page 50 Updated the following sections: • • • • • Section 1.3, “WinOF Set of Documentation”, on page 16 Section 5.3, “Firmware Upgrade”, on page 27 Section 8.10.4.2, “Enabling SR-IOV in Mellanox WinOF Package”, on page 66 Section 10.3.
PAGE 10
Rev 4.70 Table 1 - Revision History Document Revision Date December 30, 2013 Changes Updated the following sections: Section 8.7.2.2, “Configuring Windows Host”, on page 50 - Updated the example in Step 5 • Section 11.1.4.1, “Performance Tuning Tool Application”, on page 85 - Updated the Options table • Section 11.
PAGE 11
Rev 4.70 Table 1 - Revision History Document Revision Date June 10, 2013 Changes Updated the following sections: • Section 6.2, “Downloading Mellanox Firmware Tools”, on page 27 Section 14, “InfiniBand Fabric Utilities”, on page 100 • Section 15, “Troubleshooting”, on page 162 • Section 1.3, “WinOF Set of Documentation”, on page 16 • Section , “Options”, on page 86 • Added the following sections: • “perf_tuning”Appendix ,“Synopsis,” on page 86 • Section 6.3.
PAGE 12
Rev 4.70 Table 1 - Revision History Document Revision Rev 3.0.
PAGE 13
Rev 4.70 About this Manual Scope The document describes WinOF Rev 4.70 features, performance, InfiniBand diagnostic, tools content and configuration. Additionally, this document provides information on various performance tools supplied with this version. Intended Audience This manual is intended for system administrators responsible for the installation, configuration, management and maintenance of the software and hardware of VPI (InfiniBand, Ethernet) adapter cards.
PAGE 14
Rev 4.70 Common Abbreviations and Acronyms Table 3 - Abbreviations and Acronyms Abbreviation / Acronym Whole Word / Description B (Capital) ‘B’ is used to indicate size in bytes or multiples of bytes (e.g., 1KB = 1024 bytes, and 1MB = 1048576 bytes) b (Small) ‘b’ is used to indicate size in bits or multiples of bits (e.g.
PAGE 15
Rev 4.70 Related Documents Table 4 - Related Documents Document MFT User Manual Description Describes the set of firmware management tools for a single InfiniBand node. MFT can be used for: • • WinOF Release Notes Generating a standard or customized Mellanox firmware image Querying for firmware information Burning a firmware image to a single InfiniBand node For possible software issues, please refer to WinOF Release Notes.
PAGE 16
Rev 4.70 1 Introduction This User Manual describes installation, configuration and operation of Mellanox WinOF driver Rev 4.70 package. Mellanox WinOF is composed of several software modules that contain InfiniBand and Ethernet drivers. The Mellanox WinOF driver supports 10 or 40 Gb/s Ethernet, and 40 or 56 Gb/s InfiniBand network ports. The port type is determined upon boot based on card capabilities and user settings. For more details please refer to MFT User Manual. 1.
PAGE 17
Rev 4.70 2 Downloading Mellanox WinOF Driver Follow these steps to download the .exe according to your Operating System. Step 1. Verify the machine architecture. For Windows Server 2008 R2 1. Open a CMD console (Click start-->Run and enter CMD). 2. Enter the following command. > echo %PROCESSOR_ARCHITECTURE% On an x64 (64-bit) machine, the output will be “AMD64”. For Windows Server 2012 / 2012 R2 1. To go to the Start menu.
PAGE 18
Rev 4.70 3 Extracting Files Without Running Installation To extract the files without running installation, perform the following steps. Step 1. Open a CMD console [Windows Server 2008 R2] - Click Start-->Run and enter CMD. [Windows Server 2012 / 2012 R2] - Click Start --> Task Manager-->File --> Run new task --> and enter CMD. Step 2. Extract the driver and the tools: MLNX_VPI_WinOF-_All__.exe /a • To extract only the driver files. MLNX_VPI_WinOF-_All__.
PAGE 19
Rev 4.70 Step 5. Click Install to extract this folder, or click Change to install to a different folder. Step 6. To complete the extraction, click Finish.
PAGE 20
Rev 4.70 4 Installing Mellanox WinOF Driver This section provides instructions for two types of installation procedures: • “Attended Installation” An installation procedure that requires frequent user intervention. • “Unattended Installation” An automated installation procedure that requires no user intervention. Both Attended and Unattended installations require administrator privileges. 4.1 Attended Installation The following is an example of a MLNX_WinOF_win2012 x64 installation session. Step 1.
PAGE 21
Rev 4.70 Step 6. Click Next in the Welcome screen. Step 7. Read then accept the license agreement and click Next. Step 8. Select the target folder for the installation.
PAGE 22
Rev 4.70 Step 9. The firmware upgrade screen will be displayed in the following cases: • If the user has an OEM card, in this case the firmware will not be updated. • If the user has a standard Mellanox card with an older firmware version, the firmware will be updated accordingly. However, if the user has both OEM card and Mellanox card, only Mellanox card will be updated. Step 10. Configure your system for maximum performance by checking the maximum performance box.
PAGE 23
Rev 4.70 Step 11. Select a Complete or Custom installation, follow Step a and on, on page 23. a. Select the desired feature to install: • OpenSM - installs Windows OpenSM that is required to manage the subnet from a host. OpenSM is part of the driver and installed automatically. • Performances tools - install the performance tools that are used to measure the InfiniBand performance in user environment.
PAGE 24
Rev 4.70 b. Click Install to start the installation. Step 12. Click Finish to complete the installation. • If the firmware upgrade and the restore of the network configuration failed, the following message will be displayed.
PAGE 25
Rev 4.70 4.2 Unattended Installation The following is an example of a MLNX_WinOF_win2012 x64 unattended installation session. Step 1. Open a CMD console [Windows Server 2008 R2] - Click Start-->Run and enter CMD. [Windows Server 2012 / 2012 R2] - Click Start --> Task Manager-->File --> Run new task --> and enter CMD. Step 2. Install the driver. Run: > MLNX_VPI_WinOF-4.70_All_win2012_x64.exe /S /v"/qn" Step 3. [Optional] Manually configure your setup to contain the logs option: > MLNX_VPI_WinOF-4.
PAGE 26
Rev 4.
PAGE 27
Rev 4.70 5 Uninstalling Mellanox WinOF Driver 5.1 Attended Uninstall  To uninstall MLNX_WinOF on a single node: 1. Click Start-> Control Panel-> Programs and Features-> MLNX_VPI-> Uninstall. (NOTE: This requires elevated administrator privileges – see Section 1.1, “Hardware and Software Requirements”, on page 16 for details.) 2. Double click the .exe and follow the instructions of the install wizard. 3. Click Start -> All Programs -> Mellanox Technologies -> MLNX_WinOF -> Uninstall MLNX_WinOF. 5.
PAGE 28
Rev 4.70 6 Upgrading Mellanox WinOF Driver The upgrade process differs between various Operating Systems. • Windows Server 2008 R2: When upgrading from WinOF version 3.2.0 to version 4.40 and above, the MLNX_WinOF driver upgrades the driver automatically by uninstalling the previous version and installing the new driver. The existing configuration files are not saved upon driver upgrade. • Windows Server 2012 and above: • When upgrading from WinOF version 4.2 to version 4.
PAGE 29
Rev 4.70 7 Advanced Driver Configuration Once you have installed Mellanox WinOF VPI package, you can perform various modifications to your driver to make it suitable for your system’s needs Changes made to the Windows registry happen immediately, and no backup is automatically made. Do not edit the Windows registry unless you are confident regarding the changes. 7.1 Assigning Port IP After Installation By default, your machine is configured to obtain an automatic IP address via a DHCP server.
PAGE 30
Rev 4.70 Step 2. Right-click a Mellanox Local Area Connection and left-click Properties. Step 3. Select Internet Protocol Version 4 (TCP/IPv4) from the scroll list and click Properties. Step 4. Select the “Use the following IP address:” radio button and enter the desired IP information. Step 5. Click OK. Step 6. Close the Local Area Connection dialog.
PAGE 31
Rev 4.70 Step 7. Verify the IP configuration by running ‘ipconfig’ from a CMD console. > ipconfig ... Ethernet adapter Local Area Connection 4: Connection-specific IP Address. . . . . Subnet Mask . . . . Default Gateway . . DNS .. .. .. Suffix .... .... .... . . . . : : 11.4.12.63 : 255.255.0.0 : ... 7.2 Configuring the InfiniBand Driver 7.2.1 Modifying IPoIB Configuration  To modify the IPoIB configuration after installation, perform the following steps: Step 1.
PAGE 32
Rev 4.70 7.2.2 Displaying Adapter Related Information To display a summary of network adapter software, firmware- and hardware-related information such as driver version, firmware version, bus interface, adapter identity, and network port link information, perform the following steps: Step 1. Display the Device Manager. Step 2. Select the Information tab from the Properties sheet. To save this information for debug purposes, click Save to File and provide the output file name.
PAGE 33
Rev 4.70 7.3 Configuring the Ethernet Driver The following steps describe how to configure advanced features. Step 1. Display the Device Manager. Step 2. Right-click a Mellanox network adapter (under “Network adapters” list) and left-click Properties. Select the Advanced tab from the Properties sheet.
PAGE 34
Rev 4.70 Step 3. Modify configuration parameters to suit your system. Please note the following: a. For help on a specific parameter/option, check the help button at the bottom of the dialog. b. If you select one of the entries Off-load Options, Performance Options, or Flow Control Options, you’ll need to click the Properties button to modify parameters via a pop-up dialog. 7.
PAGE 35
Rev 4.70 Step 5. [Optional] If VLANs are used, mark the egress traffic with the relevant VlanID. The NIC is referred as "Ethernet 4” in the examples below. PS $ Set-NetAdapterAdvancedProperty -Name "Ethernet 4" -RegistryKeyword "VlanID" -RegistryValue "55" Step 6. [Optional] Configure the IP address for the NIC. If DHCP is used, the IP address will be assigned automatically.
PAGE 36
Rev 4.70 8 Driver Features The Mellanox VPI WinOF driver release introduces the following capabilities: • Support for Single and Dual port Adapters • Up to 16 Rx queues per port • Rx steering mode (RSS) • Hardware Tx/Rx checksum calculation • Large Send off-load (i.e.
PAGE 37
Rev 4.70 8.2 Step 1. Enable VMQ on a specific VM: Set-VMNetworkAdapter -VmqWeight 100 Step 2. Disable VMQ on a specific VM: Set-VMNetworkAdapter -VmqWeight 0 Header Data Split The header-data split feature improves network performance by splitting the headers and data in received Ethernet frames into separate buffers. The feature is disabled by default and can be enabled in the Advanced tab (Performance Options) from the Properties window.
PAGE 38
Rev 4.70 8.4 Port Configuration 8.4.1 Auto Sensing Auto Sensing enables the NIC to automatically sense the link type (InfiniBand or Ethernet) based on the cable connected to the port and load the appropriate driver stack (InfiniBand or Ethernet). Auto Sensing is performed only when rebooting the machine or after disabling/enabling the mlx4_bus interface from the Device Manager. Hence, if you replace cables during the runtime, the NIC will not perform Auto Sensing.
PAGE 39
Rev 4.70 The figure below is an example of the displayed Port Protocol window for a dual port VPI adapter card. Step 3. In this step, you can perform the following functions: • If you choose the HW Defaults option, the port protocols will be determined according to the NIC’s hardware default values. • Choose the desired port protocol for the available port(s). If you choose IB or ETH, both ends of the connection must be of the same type (IB or ETH).
PAGE 40
Rev 4.70 8.5.1.1 Teaming (Bundle) Modes 1. Fault Tolerance Provides automatic redundancy for the server’s network connection. If the primary adapter fails, the secondary adapter (currently in a standby mode) takes over. Fault Tolerance is the basis for each of the following teaming types and is inherent in all teaming modes. 2. Switch Fault Tolerance Provides a failover relationship between two adapters when each adapter is connected to a separate switch. 3.
PAGE 41
Rev 4.70 Step 1. Display the Device Manager. Step 2. Right-click a Mellanox ConnectX 10Gb Ethernet adapter (under “Network adapters” list) and left click Properties. Select the LBFO tab from the Properties window. It is not recommended to open the Properties window of more than one adapter simultaneously. The LBFO dialog enables creating, modifying or removing a bundle. Only Mellanox Technologies adapters can be part of the LBFO.  To create a new bundle, perform the following Step 1. Click Create.
PAGE 42
Rev 4.70 Step 4. Select the adapters to be included in the bundle (that have not been associated with a VLAN). Step 5. [Optional] Select Primary Adapter. An active-passive scenario used for data transfer of link disconnecting. In such scenario, the system uses one of the other interfaces. When the primary link comes up, the LBFO interface returns to transfer data using the primary interface. If the primary adapter is not selected, the primary interface is selected randomly. Step 6.
PAGE 43
Rev 4.70  To modify an existing bundle, perform the following: a. Select the desired bundle and click Modify b. Modify the bundle name, its type, and/or the participating adapters in the bundle c. Click the Commit button  To remove an existing bundle, select the desired bundle and click Remove. You will be prompted to approve this action. Notes on this step: a. Each adapter that participates in a bundle has two properties: • Status: Connected/Disconnected/Disabled • Role: Active or Backup b.
PAGE 44
Rev 4.70 Step 1. Display the Device Manager.
PAGE 45
Rev 4.70 Step 2. Right-click a Mellanox network adapter (under “Network adapters” list) and left-click Properties. Select the VLAN tab from the Properties sheet. If a physical adapter has been added to a bundle (team), the VLAN tab will not be displayed. Step 3. Click New to open a VLAN dialog window. Enter the desired VLAN Name and VLAN ID, and select the VLAN Priority.
PAGE 46
Rev 4.70 After installing the first virtual adapter (VLAN) on a specific port, the port becomes disabled. This means that it is not possible to bind to this port until all the virtual adapters associated with it are removed. When using a VLAN, the network address is configured using the VLAN ID. Therefore, the VLAN ID on both ends of the connection must be the same. Step 4. Verify the new VLAN(s) by opening the Device Manager window or the Network Connections window.
PAGE 47
Rev 4.70 8.5.5 Step 4. Select the VLAN to be removed. Step 5. Click Remove and confirm the operation. Configuring a Port to Work with VLAN in Windows 2012 and Above In this procedure you DO NOT create a VLAN, rather use an existing VLAN ID.  To configure a port to work with VLAN using the Device Manager. 8.6 Step 1. Open the Device Manager. Step 2. Go to the Network adapters. Step 3. Right click ' Properties on Mellanox ConnectX®-3 Ethernet Adapter card. Step 4. Go to Advanced tab. Step 5.
PAGE 48
Rev 4.70 Step 5. Choose the ‘Tx Throughput Port Arbiter’ option. Step 6. Set one of the following values: • Best Effort (Default) - Default behavior. No precedence is given to this port over the other. • Guaranteed - Give higher precedence to this port. • Not Present - No configuration exists, defaults are used. 8.7 RDMA over Converged Ethernet (RoCE) 8.7.
PAGE 49
Rev 4.70 The proposed RoCEv2 packets use a well-known UDP destination port value that unequivocally distinguishes the datagram. Similar to other protocols that use UDP encapsulation, the UDP source port field is used to carry an opaque flow-identifier that allows network devices to implement packet forwarding optimizations (e.g. ECMP) while staying agnostic to the specifics of the protocol header format.
PAGE 50
Rev 4.70 8.7.2.1 Prerequisites The following are the driver’s prerequisites in order to set or configure RoCE: • ConnectX®-3 and ConnectX®-3 Pro firmware version 2.30.3000 or higher • All InfiniBand verbs applications which run over InfiniBand verbs should work on RoCE links if they use GRH headers. • Set HCA to use Ethernet protocol: Display the Device Manager and expand “System Devices”. Please refer to Section 8.4.2, “Port Protocol Configuration”, on page 38. 8.7.2.
PAGE 51
Rev 4.70 (config-if-Et10)# speed forced 40gfull (config-if-Et10)# channel-group 11 mode active Step 5. Enable PFC on ports that face the network.
PAGE 52
Rev 4.
PAGE 53
Rev 4.70 PA addresses are determined by using virtualization table. Hyper-V host retrieves the received packet, identifies recipient and forwards the original packet with the CA addresses to the desired VM. NVGRE can be implemented across an existing physical IP network without requiring changes to physical network switch architecture. Since NVGRE tunnels terminate at each Hyper-V host, the hosts handle all encapsulation and de-encapsulation of the network traffic.
PAGE 54
Rev 4.70 8.8.2 Configuring the NVGRE using PowerShell Hyper-V Network Virtualization policies can be centrally configured using PowerShell 3.0 and PowerShell Remoting. Step 1. [Windows Server 2012 Only] Enable the Windows Network Virtualization binding on the physical NIC of each Hyper-V Host (Host 1 and Host 2) Enable-NetAdapterBinding (a)-ComponentID ms_netwnv - Physical NIC name Step 2. Create a vSwitch.
PAGE 55
Rev 4.70 Please repeat steps 5 to 8 on each Hyper-V after rebooting the Hypervisor. 8.8.3 Verifying the Encapsulation of the Traffic Once the configuration using PowerShell is completed, verifying that packets are indeed encapsulated as configured is possible through any packet capturing utility. If configured correctly, an encapsulated packet should appear as a packet consisting of the following headers: Outer ETH Header, Outer IP, GRE Header, Inner ETH Header, Original Ethernet Payload. 8.8.
PAGE 56
Rev 4.70 8.9.1 Setting the DSCP in the IP Header Marking DSCP value in the IP header is done differently for IP packets constructed by the NIC (e.g. RDMA traffic) and for packets constructed by the IP stack (e.g. TCP traffic). 8.9.2 • For IP packets generated by the IP stack, the DSCP value is provided by the IP stack. The NIC does not validate the match between DSCP and Class of Service (CoS) values.
PAGE 57
Rev 4.70 8.9.5 Registry Settings The following attributes must be set manually and will be added to the miniport registry. Table 7 - DSCP Registry Keys Settings Registry Key Description TxUntagPriorityTag If 0x1, do not add 802.1Q tag to transmitted packets which are assigned 802.1p priority, but are not assigned a non-zero VLAN ID (i.e. priority-tagged). Default 0x0, for DSCP based PFC set to 0x1. RxUntaggedMapToLossless If 0x1, all untagged traffic is mapped to the lossless receive queue.
PAGE 58
Rev 4.70 Table 8 - DSCP Default Registry Keys Settings Registry Key RroceDscpMarkPriorityFlowControl_7 8.9.6 Default Value 7 DSCP Sanity Testing To verify that all QoS and DSCP settings were correct, you can capture incoming and outgoing traffic by using the ibdump tool and see the DSCP value in the captured packets as displayed in the figure below. 8.
PAGE 59
Rev 4.70 8.10.1 System Requirements • A server/blade with an SR-IOV-capable motherboard and BIOS. BIOS settings might need to be updated to enable virtualization support and SR-IOV support. • Hypervisor OS: Windows Server 2012 R2 and above • Virtual Machine (VM) OS: • The VM OS can be either Windows Server 2012 or Windows Server 2012 R2 8.10.2 8.10.3 • Mellanox ConnectX®-3/ ConnectX®-3 Pro VPI Adapter Card family with SR-IOV capability • Mellanox WinOF 4.
PAGE 60
Rev 4.70 a. b. Enable SR-IOV. Enable “Intel Virtualization Technologhy” Support For further details, please refer to the vendor's website. 8.10.3.2 Installing Hypervisor Operating System  To install Hypervisor Operating System: Step 1. Install Windows Server 2012 R2 and above. Step 2.
PAGE 61
Rev 4.70 a. Go to: Server Manager -> Manage -> Add Roles and Features -> Installation Type-> Rolebased or Feature-based Installation -> Server Selection b. Go to: Server Roles -> Hyper-V Figure 5: Hyper-V Server Roles Selection Step 3. Install Hyper-V Management Tools. a.
PAGE 62
Rev 4.70 Figure 6: Hyper-V Features Selection Step 4. Confirm the Installation. Figure 7: Hyper-V Confirming Installation Step 5. Reboot the system.
PAGE 63
Rev 4.70 8.10.3.3 Verifying SR-IOV Support within the Host Operating System  To verify that the system is properly configured for SR-IOV Step 1. Go to: Start-> Windows Powershell. Step 2. Run the following PowerShell commands. (Get-VmHost).IovSupport (Get-VmHost).IovSupportReasons In case that SR-IOV is supported by the OS, the output in the PowerShell is as in Figure 5. Figure 8: Operating System Supports SR-IOV Step 3. Update the registry configuration as described in the (Get-VmHost).
PAGE 64
Rev 4.70 Figure 10: Hyper-V Manager Step 3. Connect the virtual hard disk in the New Virtual Machine Wizard. Step 4. Go to: Connect Virtual Hard Disk -> Use an existing virtual hard disk. Step 5. Select the location of the vhd file.
PAGE 65
Rev 4.70 8.10.4 Configuring Mellanox Network Adapter for SR-IOV The following are the steps for configuring Mellanox Network Adapter for SR-IOV: 8.10.4.1 Enabling SR-IOV in Firmware SR-IOV can be enabled and managed by using one of the following methods:  To burn firmware with SR-IOV support: Step 1. Verify that HCA is configured for SR-IOV by dumping the device configuration file to userchosen location .ini.. flint -d dc > .ini Step 2.
PAGE 66
Rev 4.70 Example output: Device #1: ---------Device type: PCI device: ConnectX3 mt4099_pciconf0 Configurations: SRIOV_EN NUM_OF_VFS WOL_MAGIC_EN_P2 Step 3. Current N/A N/A N/A Enable SR-IOV with 8 VFs. mlxconfig -d mt4099_pciconf0 s SRIOV_EN=1 NUM_OF_VFS=8 Example output: Device #1: ---------Device type: PCI device: ConnectX3 mt4099_pciconf0 Configurations: SRIOV_EN NUM_OF_VFS WOL_MAGIC_EN_P2 Current N/A N/A N/A New 1 8 N/A Apply new Configuration? ? (y/n) [n] : 8.10.4.
PAGE 67
Rev 4.70 Step 4. Enable SR-IOV through Powershell. Set-MlnxPCIDeviceSriovSetting -Name "HCA 0" -SriovEnable $true SR-IOV mode configuration parameters: Parameter Name SriovEnable Values 0 = RoCE (default) 1 = SR-IOV Configures the RDMA or SR-IOV mode. Note: RDMA is not supported in SR-IOV mode. • • 0 = auto_port1 (default) 1 = auto_port2 2 = manual Configures the number of VFs to be enabled by the bus driver to each port.
PAGE 68
Rev 4.70 Figure 12: System Event Log 8.10.5 Configuring Virtual Machine Networking  To configure Virtual Machine networking: Step 1. Create an SR-IOV-enabled Virtual Switch over Mellanox Ethernet Adapter. Go to: Hyper-V Manager-> Actions -> Virtual Switch-> external-> Create virtual Switch-> Apply.
PAGE 69
Rev 4.70 Figure 13: Virtual Switch with SR-IOV Step 2. Add a VMNIC connected to a Mellanox vSwitch. Go to: Hyper-V Manager-> Settings-> Add New Hardware-> Network Adapter-> OK. In “Virtual Switch” dropdown box, choose Mellanox SR-IOV Virtual Switch.
PAGE 70
Rev 4.70 Figure 14: Adding a VMNIC to a Mellanox V-switch Step 3. Start and connect to the Virtual Machine: Select the newly created Virtual Machine and go to: Actions panel-> Connect. In the virtual machine window go to: Actions-> Start Step 4. Assign IP address to the Mellanox VMNIC. 1. To go to Network Connections, enter the following command in the command prompt: ncpa.cpl 2. Right-click the Hyper-V adapter and choose properties. 3. Mark the “Use the following IP address” checkbox. 4.
PAGE 71
Rev 4.70 Figure 15: Enable SR-IOV on VMNIC Step 9. Verify that Mellanox Virtual Function appears in the device manager. Virtual Function is configured with DHCP IP address. It can also be assigned a static IP address.
PAGE 72
Rev 4.70 Figure 16: Virtual Function in the VM To achieve best performance on SR-IOV VF, please run the following powershell commands on the host: For 10Gbe: Set-VMNetworkAdapter -Name "Network Adapter" -VMName vm1 -IovQueuePairsRequested 4 For 40Gbe: Set-VMNetworkAdapter -Name "Network Adapter" -VMName vm1 -IovQueuePairsRequested 8 8.11 Virtual Ethernet Adapter The Virtual Ethernet Adapter (VEA) provides a mechanism enabling multiple ethernet adapters on the same physical port.
PAGE 73
Rev 4.70 • Firmware version: 2.31.5050 8.11.2 VEA Feature Limitations • RoCE (RDMA) is supported only on the physical VEA • MTU (*JumboFrame registry key), QoS and, Flow Control are only configured from physical VEA • No bandwidth allocation between the two interfaces • Both interfaces share the same link speed • SR-IOV and VEA are not supported simultaneously. Only one of the features can be used at any given time. 8.11.
PAGE 74
Rev 4.70 8.12 IPoIB SR-IOV over KVM This feature is in Beta quality. For more details on how to configure IPoIB SR-IOV over KVM, please contact Mellanox support. 8.13 Lossless TCP 8.13.1 Introduction Inbound packets are stored in the data buffers. They are split into 'Lossy' and 'Lossless' according to the priority field in the 802.1Q VLAN tag. In DSCP based PFC, all traffic is directed to the 'Lossless' buffer.
PAGE 75
Rev 4.70 8.13.2 Drop Mode In this mode, a packet arriving to a descriptor ring with no free descriptors is dropped, after verifying that there are really no free descriptors. This allows isolation of the host driver execution delays from the network, as well as isolation between different SW entities sharing the adapter (e.g. SR-IOV VMs). 8.13.3 Poll Mode In this mode, a packet arriving to a descriptor ring with no free descriptors will patiently wait until a free descriptor is posted.
PAGE 76
Rev 4.70 8.13.7.1 Enabling Lossless TCP Using The Registry Key DelayDropTimeout: Registry Key location: HKLM\SYSTEM\CurrentControlSet\Control\Class\Class\{4d36e972e325-11ce-bfc1-08002be10318}\\DelayDropTimeout For instructions on how to find interface index in registry , Please refer to C.
PAGE 77
Rev 4.70 9 Booting Windows from an iSCSI Target 9.1 Configuring the WDS, DHCP and iSCSI Servers 9.1.1 Configuring the WDS Server  To configure the WDS server: 1. Install the WDS server. 2. Extract the Mellanox drivers to a local directory using the '-a' parameter. For boot over Ethernet, when using adapter cards with older firmware version than 2.30.8000, you need to extract the PXE package, otherwise use Mellanox WinOF VPI package. Example: Mellanox.msi.exe -a 3. Add the Mellanox driver to boot.
PAGE 78
Rev 4.70 4. Add to the reserved IP address the following options: Table 10 - Reserved IP Address Options Option 017 Root Path 060 066 PXEClient Boot Server Host Name Boot File Name 067 9.2 Name Value iscsi:11.4.12.65::::iqn:2011-01:iscsiboot Assuming the iSCSI target IP is: 11.4.12.65 and the Target Name: iqn:201101:iscsiboot PXEClient WDS server IP address boot\x86\wdsnbp.com Configuring the Client Machine  To configuring your client: 1.
PAGE 79
Rev 4.70 3. Choose the relevant boot image from the list of all available boot images presented. 4. Choose the Operating System you wish to install. 5. Run the Windows Setup Wizard. 6. Choose iSCSI target drive to install Windows and follow the instructions presented by the installation Wizard. Installation process will start once completing all the required steps in the Wizard, the Client will reboot and will boot from the iSCSI target.
PAGE 80
Rev 4.70 10 Deploying Windows Server 2012 and Above with SMB Direct 10.1 Overview The Server Message Block (SMB) protocol is a network file sharing protocol implemented in Microsoft Windows. The set of message packets that defines a particular version of the protocol is called a dialect. The Microsoft SMB protocol is a client-server implementation and consists of a set of data packets, each containing a request sent by the client or a response sent by the server.
PAGE 81
Rev 4.70 Get-SmbServerNetworkInterface netstat.exe -xan | ? {$_ -match "445"} 1. The NETSTAT command confirms if the File Server is listening on the RDMA interfaces.
PAGE 82
Rev 4.70 10.3.3 Verifying SMB Connection  To verify the SMB connection on the SMB client: Step 1. Copy the large file to create a new session with the SMB Server. Step 2. Open a PowerShell window while the copy is ongoing. Step 3. Verify the SMB Direct is working properly and that the correct SMB dialect is used. Get-SmbConnection Get-SmbMultichannelConnection netstat.
PAGE 83
Rev 4.70 11 Performance Tuning This section describes how to modify Windows registry parameters in order to improve performance. Please note that modifying the registry incorrectly might lead to serious problems, including the loss of data, system hang, and you may need to reinstall Windows. As such it is recommended to back up the registry on your system before implementing recommendations included in this section.
PAGE 84
Rev 4.70 Step 6. • Single port traffic - Improves performance for running single port traffic each time. • Dual port traffic - Improves performance for running traffic on both ports simultaneously. • Forwarding traffic - Improves performance for running scenarios that involve both ports (for example: via IXIA) • Multicast traffic - Improves performance when the main traffic runs on multicast. Click on “Run Tuning” button.
PAGE 85
Rev 4.70 7. Click on “Run Tuning” button. Clicking the "Run Tuning" button activates the general tuning as explained above and changes several driver registry entries for the current adapter and its sibling device once the sibling is an Ethernet device as well. It also generates a log including the applied changes. Users can view this log to restore the previous values. The log path is: %HOMEDRIVE%\Windows\System32\LogFiles\PerformanceTunning.
PAGE 86
Rev 4.70 Synopsis perf_tuning.exe -s -c1 [-c2 ] perf_tuning.exe -d -c1 -c2 perf_tuning.exe -f -c1 -c2 perf_tuning.exe -m -c1 -b -n perf_tuning -st -c1 [-c2 ] Options Flag -s Description Single port traffic scenario.
PAGE 87
Rev 4.70 Flag -f Description Forwarding traffic scenario. This option must be followed by two connection names. The tuning in this case is codependent. This option automatically sets: • • • • • SendCompletionMethod = 1 RecvCompletionMethod = 0 *ReceiveBuffers = 4096 UseRSSForRawIP = 0 UseRSSForUDP = 0 Additionally, this option chooses the best processors to assign to: • • • • DefaultRecvRingProcessor TxInterruptProcessor TxForwardingProcessor In Operating Systems support NDIS6.
PAGE 88
Rev 4.70 Flag -st Description Single stream traffic scenario. This option must be followed by one or two connection names for an Ethernet adapter. The tuning will restore the default settings on the second connection and performed on the first connection. This option automatically sets: • • • • SendCompletionMethod = 0 RecvCompletionMethod = 2 *ReceiveBuffers = 1024 In Operating Systems support NDIS6.
PAGE 89
Rev 4.70 11.1.5 SR-IOV Tuning To achieve best performance on SR-IOV VF, please run the following powershell commands on the host: Set-VMNetworkAdapter -Name "Network Adapter" -VMName vm1 -IovQueuePairsRequested 4 OR Set-VMNetworkAdapter -Name "Network Adapter" -VMName vm1 -IovQueuePairsRequested 8 for 40GbE 11.1.6 Improving Live Migration In order to improve live migration over SMB direct performance, please set the following registry key to 0 and reboot the machine: HKEY_LOCAL_MACHINE\System\CurrentContr
PAGE 90
Rev 4.70 http://www.mellanox.com/page/products_dyn?product_family=32&mtag=windows_sw_drivers  To improve performance, activate the performance tuning tool as follows: 11.3 Step 1. Start the "Device Manager" (open a command line window and enter: devmgmt.msc). Step 2. Open "Network Adapters". Step 3. Right click the relevant IPoIB adapter and select Properties. Step 4. Select the "Advanced" tab Step 5. Modify performance parameters (properties) as desired.
PAGE 91
Rev 4.70 ignated destination. RSS can significantly improve the number of transactions, the number of connections per second, and the network throughput. This parameter can be set to one of the following values: • Enabled (default): Set RSS Mode • Disabled: The hardware is configured once to use the Toeplitz hash function, and the indirection table is never changed. IOAT is not used while in RSS mode.
PAGE 92
Rev 4.70 • Offload Options Allows you to specify which TCP/IP offload settings are handled by the adapter rather than the operating system. Enabling offloading services increases transmission performance as the offload tasks are performed by the adapter hardware rather than the operating system. Thus, freeing CPU resources to work on other tasks. • IPv4 Checksums Offload Enables the adapter to compute IPv4 checksum upon transmit and/or receive instead of the CPU (default Enabled).
PAGE 93
Rev 4.70 11.4.1 Supported Standard Performance Counters 11.4.1.1 Proprietary Mellanox Adapter Traffic Counters Proprietary Mellanox adapter traffic counter set consists of global traffic statistics which gather information from ConnectX®-3 and ConnectX®-3 Pro network adapters, and includes traffic statistics, and various types of error and indications from both the Physical Function and Virtual Function.
PAGE 94
Rev 4.70 Table 11 - Mellanox Adapter Traffic Counters Mellanox Adapter Traffic Counters Description Packets Outbound Errors Shows the number of outbound packets that could not be transmitted because of errors. Packets Outbound Discarded Shows the number of outbound packets to be discarded even though no errors had been detected to prevent transmission. One possible reason for discarding packets could be to free up buffer space.
PAGE 95
Rev 4.70 Table 12 - Mellanox Adapter Diagnostics Counters Mellanox Adapter Diagnostics Counters Description Requester CQE errors Number of local CQE with errors when the local machine generates outbound traffic. Responder CQE errors Number of local CQE with errors when the local machine receives inbound traffic. Requester Invalid request errors Number of remote invalid request errors when the local machine generates outbound traffic, i.e.
PAGE 96
Rev 4.70 Table 12 - Mellanox Adapter Diagnostics Counters Mellanox Adapter Diagnostics Counters Description Discarded UD packets Number of UD packets silently discarded on the receive queue due to lack of receives descriptor. Discarded UC packets Number of UC packets silently discarded on the receive queue due to lack of receives descriptor. CQ overflows Number of CQ overflows. NOTE: this value is evaluated for the entire NIC since there are cases where CQ might be associated with both ports (i.e.
PAGE 97
Rev 4.70 Table 13 - Mellanox QoS Counters Mellanox QoS Counters Description Bytes Sent/Sec The number of bytes sent per second that are covered by this priority. The counted bytes include framing characters. Packets Sent The number of packets sent that are covered by this priority (modulo 2^64). Packets Sent/Sec The number of packets sent per second that are covered by this priority. Bytes and Packets’ TOTAL Bytes Total The total number of bytes that are covered by this priority.
PAGE 98
Rev 4.70 12 OpenSM - Subnet Manager OpenSM v3.3.11 is an InfiniBand Subnet Manager. In order to operate one host machine or more in the InfiniBand cluster., at least one Subnet Manger is required in the fabric. Please use the embedded OpenSM in the WinOF package for testing purpose in small cluster. Otherwise, we recommend using OpenSM from FabricIT EFM™ or UFM® or MLNX-OS®. OpenSM can run as a Windows service and can be started manually from the following directory: \tools.
PAGE 99
Rev 4.70 13 Software Development Kit (SDK) Software Development Kit (SDK) a set of development tools that allows the creation of InfiniBand applications for MLNX_VPI software package. The SDK package contains, header files, libraries, and code examples. To compile the examples provided with the SDK you must install Windows Driver Kit (WDK) version 8.1 and higher over Visual Studio 2013 To open the SDK package you must run the sdk.exe file and get the complete list of files.
PAGE 100
Rev 4.70 14 InfiniBand Fabric Utilities 14.1 Network Direct Interface The Network Direct Interface (NDI) architecture provides application developers with a networking interface that enables zero-copy data transfers between applications, kernel-bypass I/O generation and completion processing, and one-sided data transfer operations. NDI is supported by Microsoft and is the recommended method to write InfiniBand application.
PAGE 101
Rev 4.70 14.3.1.1 Common Configuration, Interface and Addressing Topology File (Optional) An InfiniBand fabric is composed of switches and channel adapter (HCA/TCA) devices. To identify devices in a fabric (or even in one switch system), each device is given a GUID (a MAC equivalent). Since a GUID is a non-user-friendly string of characters, it is better to alias it to a meaningful, user-given name.
PAGE 102
Rev 4.70 • Using port names defined in the topology file: (Tool option ‘-n’) This option refers to the source and destination ports by the names defined in the topology file. (Therefore, this option is relevant only if a topology file is specified to the tool.) In this mode, the tool uses the names to extract the port LIDs from the matched topology, then the tool operates as in the ‘-l’ option. 14.3.
PAGE 103
Rev 4.70 14.3.2.2 ibdiagnet Output Files Table 15 - ibdiagnet Output Files Output File Description ibdiagnet.log A dump of all the application reports generate according to the provided flags ibdiagnet.lst List of all the nodes, ports and links in the fabric ibdiagnet.fdbs A dump of the unicast forwarding tables of the fabric switches ibdiagnet.mcfdbs A dump of the multicast forwarding tables of the fabric switches ibdiagnet.
PAGE 104
Rev 4.70 14.3.2.3 ibdiagnet Error Codes 1 2 3 4 5 6 - Failed Failed Failed Failed Failed Failed to to to to to to fully discover the fabric parse command line options interact with IB fabric use local device or local port use Topology File load required Package 14.3.3 ibportstate Enables querying the logical (link) and physical port states of an InfiniBand port. It also allows adjusting the link speed that is enabled on any InfiniBand port.
PAGE 105
Rev 4.70 Table 16 - ibportstate Flags and Options (Continued) Flag Description -G/--Guid Use GUID address argument. In most cases, it is the Port GUID. Example: ‘0x08f1040023’ -s/--sm_port Use as the target lid for SM/SA queries -C/--Ca Use the specified channel adapter or router -P/--Port Use the specified port -u/--usage Usage message -t/--timeout Override the default timeout for the solicited MADs [msec] Destination’s directed path, LID, or GUID.
PAGE 106
Rev 4.70 LinkWidthEnabled:................1X or 4X LinkWidthActive:.................4X LinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps LinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps LinkSpeedActive:.................5.0 Gbps 2. Query the status of two channel adapters using directed paths. > ibportstate -C mlx4_0 -D 0 1 PortInfo: # Port info: DR path slid 65535; dlid 65535; 0 port 1 LinkState:.......................Initialize PhysLinkState:...................LinkUp LinkWidthSupported:.........
PAGE 107
Rev 4.70 LinkSpeedEnabled:................2.5 Gbps After PortInfo set: # Port info: DR path slid 65535; dlid 65535; 0 port 1 LinkSpeedEnabled:................5.0 Gbps (IBA extension) # Show the new configuration > ibportstate -C mlx4_0 -D 0 1 PortInfo: # Port info: DR path slid 65535; dlid 65535; 0 port 1 LinkState:.......................Initialize PhysLinkState:...................LinkUp LinkWidthSupported:..............1X or 4X LinkWidthEnabled:................1X or 4X LinkWidthActive:.................
PAGE 108
Rev 4.70 Table 17 - ibroute Flags and Options Flag Description -n/--no_dests Do not try to resolve destinations -D/--Direct Use directed path address arguments. The path is a comma separated list of out ports. Examples: ‘0’ – self port ‘0,1,2,1,4’ – out via port 1, then 2, ... -G/--Guid Use GUID address argument. In most cases, it is the Port GUID. Example: ‘0x08f1040023’ -M/--Multicast Show multicast forwarding tables. The parameters and specify the MLID range.
PAGE 109
Rev 4.70 Unicast lids [0x3-0x7] of switch Lid 2 guid 0x0002c902fffff00a (MT47396 InfiniscaleIII Mellanox Technologies): Lid Out Destination Port Info 0x0003 021 : (Switch portguid 0x000b8cffff004016: 'MT47396 Infiniscale-III Mellanox Technologies') 0x0006 007 : (Channel Adapter portguid 0x0002c90300001039: 'sw137 HCA-1') 0x0007 021 : (Channel Adapter portguid 0x0002c9020025874a: 'sw157 HCA-1') 3 valid lids dumped 3. Dump all Lids with valid out ports of the switch with portguid 0x000b8cffff004016.
PAGE 110
Rev 4.70 .pcap format. This file can be loaded by the Wireshark tool (www.wireshark.org) for graphical traffic analysis. This provides the ability to analyze network behavior and performance, and to debug applications that send or receive RDMA network traffic. Run "ibdump -h" to display a help message which details the tools options. 14.3.5.1 ibdump Synopsis - ibdump 14.3.5.2 ibdump Options The table below lists the various ibdump flags of the command.
PAGE 111
Rev 4.70 14.3.6.2 smpquery Synopsis smpquery [-h] [-d] [-e] [-c] [-v] [-D] [-G] [-s ] [-L] [-u] [-V] [-C ] [-P ] [-t ] [--node-name-map ] [op params] 14.3.6.3 smpquery Options The table below lists the various flags of the command. Table 19 - smpquery Flags and Options Flag Description -h/--help Print the help menu -d/--debug Raise the IB debug level.
PAGE 112
Rev 4.70 Table 19 - smpquery Flags and Options Flag Description Destination’s directed path, LID, or GUID --node-name-map Node name map file -x/--extended Use extended speeds Examples 1. Query PortInfo by LID, with port modifier. > smpquery portinfo 1 1 # Port info: Lid 1 port 1 Mkey:............................0x0000000000000000 GidPrefix:.......................0xfe80000000000000 Lid:.............................0x0001 SMLid:...........................
PAGE 113
Rev 4.70 MtuCap:..........................2048 VLStallCount:....................0 HoqLife:.........................31 OperVLs:.........................VL0-3 PartEnforceInb:..................0 PartEnforceOutb:.................0 FilterRawInb:....................0 FilterRawOutb:...................0 MkeyViolations:..................0 PkeyViolations:..................0 QkeyViolations:..................0 GuidCap:.........................128 ClientReregister:................0 SubnetTimeout:...................
PAGE 114
Rev 4.70 PartCap:.........................128 DevId:...........................0x634a Revision:........................0x000000a0 LocalPort:.......................1 VendorId:........................0x0002c9 14.3.7 perfquery Queries InfiniBand ports’ performance and error counters. Optionally, it displays aggregated counters for all ports of a node. It can also reset counters after reading them or simply reset them. 14.3.7.1 perfquery Applicable Hardware All InfiniBand devices. 14.3.7.
PAGE 115
Rev 4.
PAGE 116
Rev 4.70 # Port counters: Lid 6 port 1 PortSelect:......................1 CounterSelect:...................0x1000 SymbolErrors:....................0 LinkRecovers:....................0 LinkDowned:......................0 RcvErrors:.......................0 RcvRemotePhysErrors:.............0 RcvSwRelayErrors:................0 XmtDiscards:.....................0 XmtConstraintErrors:.............0 RcvConstraintErrors:.............0 LinkIntegrityErrors:.............0 ExcBufOverrunErrors:.............
PAGE 117
Rev 4.70 RcvErrors:.......................0 RcvRemotePhysErrors:.............0 RcvSwRelayErrors:................0 XmtDiscards:.....................3 XmtConstraintErrors:.............0 RcvConstraintErrors:.............0 LinkIntegrityErrors:.............0 ExcBufOverrunErrors:.............0 VL15Dropped:.....................0 XmtData:.........................0 RcvData:.........................0 XmtPkts:.........................0 RcvPkts:.........................0 14.3.
PAGE 118
Rev 4.70 Table 21 - ibping Flags and Options Flag Description --Guid, -G Uses GUID address argument. In most cases, it is the Port GUID. For example: "0x08f1040023" --sm_port, -s Uses 'smlid' as the target lid for SM/SA queries --Ca, -C Uses the specified ca_name --Port, -P Uses the specified ca_port --timeout, -t Overrides the default timeout for the solicited mads 14.3.
PAGE 119
Rev 4.70 Table 22 - ibnetdiscover Flags and Options Flag Description --node-name-map Specifies a node name map. The node name map file maps GUIDs to more user friendly names. See “Topology File Format” on page 120. --cache Caches the ibnetdiscover network data in the specified filename. This cache may be used by other tools for later analysis --load-cache Loads and use the cached ibnetdiscover data stored in the specified filename.
PAGE 120
Rev 4.70 14.3.9.3 Topology File Format The topology file format is largely intuitive. Most identifiers are given textual names like vendor ID (vendid), device ID (device ID), GUIDs of various types (sysimgguid, caguid, switchguid, etc.). PortGUIDs are shown in parentheses (). For switches, this is shown on the switchguid line. For CA and router ports, it is shown on the connectivity lines. The IB node is identified followed by the number of ports and the node GUID.
PAGE 121
Rev 4.
PAGE 122
Rev 4.
PAGE 123
Rev 4.70 Table 23 - ibtracert Flags and Options Flag Description --Lid, -L Uses LID address argument --errors, -e Shows send and receive errors --usage, -u Usage message --Guid, -G Uses GUID address argument. In most cases, it is the Port GUID.
PAGE 124
Rev 4.70 14.3.11.1sminfo Synopsis sminfo [-d(ebug)] [-e(rr_show)] [-s state] [-p prio] [-a activity] [-D(irect)] [-L(id)] [-u(sage)] [-G(uid)] [-C ca_name] [-P ca_port] [-t(imeout) timeout_ms] [V(ersion)] [-h(elp)] sm_lid | sm_dr_path [modifier] 14.3.11.2sminfo Options The table below lists the various flags of the command. Most OpenIB diagnostics take the following common flags. The exact list of supported flags per utility can be found in the usage message and can be shown using the util_name -h syntax.
PAGE 125
Rev 4.70 Examples sminfo # local ports sminfo sminfo 32 # show sminfo of lid 32 sminfo -G 0x8f1040023 # same but using guid address 14.3.12 ibclearerrors ibclearerrors is a script which clears the PMA error counters in PortCounters by either waking the InfiniBand subnet topology or using an already saved topology file. 14.3.12.1ibclearerrors Synopsis ibclearerrors [-h] [-N | -nocolor] [ | -C ca_name -P ca_port -t(imeout) timeout_ms] 14.3.12.
PAGE 126
Rev 4.
PAGE 127
Rev 4.70 • It verifies the existing inventory, with all the object fields, and matches it to a pre-saved one. • A Multicast Compliancy test. • An Event Forwarding test. • A Service Record registration test. • An RMPP stress test. • A Small SA Queries stress test. It is recommended that after installing opensm, the user should run "osmtest -f c" to generate the inventory file, and immediately afterwards run "osmtest -f a" to test OpenSM.
PAGE 128
Rev 4.70 Table 28 - osmtest Flags and Options Flag Description -m, --max_lid This option specifies the maximal LID number to be searched for during inventory file build (default to 100) -g, --guid This option specifies the local port GUID value with which OpenSM should bind. OpenSM may be bound to 1 port at a time. If GUID given is 0, OpenSM displays a list of possible port GUIDs and waits for user input.
PAGE 129
Rev 4.70 Table 28 - osmtest Flags and Options Flag Description -vf This option sets the log verbosity level. A flags field must follow the -D option.
PAGE 130
Rev 4.70 Table 29 - ibaddr Flags and Options Flags Description Debugging Flags Description NOTE: Most OpenIB diagnostics take the following common flags. The exact list of supported flags per utility can be found in the usage message and can be shown using the util_name -h syntax. -d Raises the IB debugging level. Can be used several times (-ddd or -d -d -d). -e shows send and receive errors (timeouts and others) -h shows the usage message -v Increases the application verbosity level.
PAGE 131
Rev 4.70 Examples ibaddr ibaddr ibaddr ibaddr ibaddr ibaddr 32 -G -l -L -g 0x8f1040023 32 # 32 # 32 # # local port´s address # show lid range and gid of lid 32 # same but using guid address show lid range only show decimal lid range only show gid address only 14.3.17 ibcacheedit ibcacheedit allows users to edit an ibnetdiscover cache created through the --cache option in ibnetdiscover(8). 14.3.17.
PAGE 132
Rev 4.70 14.3.18 iblinkinfo iblinkinfo reports link info for each port in an IB fabric, node by node. Optionally, iblinkinfo can do partial scans and limit its output to parts of a fabric. 14.3.18.1iblinkinfo Synopsis [-hcdl -C -P -p -S -G -D --load-cache ] 14.3.18.
PAGE 133
Rev 4.70 Table 31 - iblinkinfo Flags and Options Flags Description --diffcheck Specifies what diff checks should be done in the--diffoption above. Comma separate multiple diff check key(s). The available diff checks are:port = port connections,state = port state, lid = lids, nodedesc = node descriptions. If port is specified alongside lid or nodedesc, remote port lids and node descriptions will also be compared.
PAGE 134
Rev 4.70 Table 32 - ibqueryerrors Flags and Options Flags Description -r Reports the port information. This includes LID, port, external port (if applicable), link speed setting, remote GUID, remote port, remote external port (if applicable), and remote node description information. --data Includes the optional transmit and receive data counters. --threshold-file Specifies an alternate threshold file.
PAGE 135
Rev 4.70 Table 32 - ibqueryerrors Flags and Options Flags -t Description Overrides the default timeout for the solicited mads. 14.3.19.3ibqueryerrors Exit Status If a failure to scan the fabric occurs return -1. If the scan succeeds without errors beyond thresholds return 0. If errors are found on ports beyond thresholds return 1. 14.3.19.4ibqueryerrors Files /opt/ufm/files/conf/infiniband-diags/error_thresholds Define threshold values for errors. File format is simple "name=val".
PAGE 136
Rev 4.70 Table 33 - ibsysstat Flags and Options Flags Description NOTE: Most OpenIB diagnostics take the following common flags. The exact list of supported flags per utility can be found in the usage message and can be shown using the util_name -h syntax. -d Raises the IB debugging level. Can be used several times (-ddd or -d -d -d). -e Shows send and receive errors (timeouts and others) -h Shows the usage message -v Increases the application verbosity level.
PAGE 137
Rev 4.70 14.3.21 saquery saquery issues the selected SA query. Node records are queried by default. 14.3.21.
PAGE 138
Rev 4.70 14.3.21.2saquery Options Table 34 - saquery Flags and Options Flags Description -p Gets PathRecord info. -N Gets NodeRecord info. --list | -D Gets NodeDescriptions of CAs only. -S Gets ServiceRecord info. -I Gets InformInfoRecord (subscription) info.
PAGE 139
Rev 4.70 Table 34 - saquery Flags and Options Flags Description -t, -timeout Specifies SA query response timeout in milliseconds. Default is 100 milliseconds. You may want to use this option if IB_TIMEOUT is indicated. --node-name-map Specifies a node name map.The node name map file maps GUIDs to more user friendly names. See ibnetdiscover(8) for node name map file format.Only used with the -O and -U options.
PAGE 140
Rev 4.70 14.3.22.2smpdump Options Table 35 - smpdump Flags and Options Flags Description attr IBA attribute ID for SM attribute mod IBA modifier for SM attribute Debugging Flags Description NOTE: Most OpenIB diagnostics take the following common flags. The exact list of supported flags per utility can be found in the usage message and can be shown using the util_name -h syntax. -d Raises the IB debugging level. Can be used several times (-ddd or -d -d -d).
PAGE 141
Rev 4.70 Examples Direct Routed Examples: smpdump -D 0,1,2,3,5 16 # NODE DESC smpdump -D 0,1,2 0x15 2 # PORT INFO, port 2 LID Routed Examples: smpdump 3 0x15 2 smpdump 0xa0 0x11 14.4 # PORT INFO, lid 3 port 2 # NODE INFO, lid 0xa0 InfiniBand Fabric Performance Utilities The performance utilities described in this chapter are intended to be used as a performance micro-benchmark. 14.4.1 ib_read_bw ib_read_bw calculates the BW of RDMA read between a pair of machines.
PAGE 142
Rev 4.70 Table 36 - ib_read_bw Flags and Options Flag Description -b, --bidirectional Measures bidirectional bandwidth (default unidirectional) -V, --version Displays version number -g, --grh Use GRH with packets (mandatory for RoCE) 14.4.2 ib_read_lat ib_read_lat calculates the latency of RDMA read operation of message_size between a pair of machines. One acts as a server and the other as a client.
PAGE 143
Rev 4.70 Table 37 - ib_read_lat Flags and Options Flag -g, --grh Description Use GRH with packets (mandatory for RoCE) 14.4.3 ib_send_bw ib_send_bw calculates the BW of SEND between a pair of machines. One acts as a server and the other as a client. The server receive packets from the client and they both calculate the throughput of the operation.
PAGE 144
Rev 4.70 which you send packet only if you receive one. Each of the sides samples the CPU each time they receive a packet in order to calculate the latency. 14.4.4.1 ib_send_lat Synopsis ib_send_lat [-i(b_port) ib_port] [-c(onnection_type) RC\UC\UD] [-m(tu) mtu_size] [s(ize) message_size] [-t(x-depth) tx_size] [-n iteration_num] [-p(ort) PDT_port] [-a(ll)] [-V(ersion)] [-C report cycles] [-H report histogram] [-U report unsorted] 14.4.4.
PAGE 145
Rev 4.70 14.4.5.1 ib_write_bw Synopsis ib_write_bw [-q num of qps] [-c(onnection_type) RC\UC] [-i(b_port) ib_port] [-m(tu) mtu_size] [-s(ize) message_size] [-t(x-depth) tx_size] [-n iteration_num] [-p(ort) PDT_port] [-b(idirectional)] [-a(ll)] [-V(ersion)] 14.4.5.2 ib_write_bw Options The table below lists the various flags of the command.
PAGE 146
Rev 4.70 14.4.6.1 ib_write_lat Synopsis ib_write_lat [-i(b_port) ib_port] [-c(onnection_type) RC\UC] [-m(tu) mtu_size] [s(ize) message_size] [-t(x-depth) tx_size] [-n iteration_num] [-p(ort) PDT_port] [-a(ll)] [-V(ersion)] [-C report cycles] [-H report histogram] [-U report unsorted] 14.4.6.2 ib_write_lat Options The table below lists the various flags of the command.
PAGE 147
Rev 4.70 14.4.7.1 ibv_read_bw Synopsis ibv_read_bw [-i(b_port) ib_port] [-d ib device] [-o(uts) outstanding reads] [-m(tu) mtu_size] [-s(ize) message_size] [-t(x-depth) tx_size] [-n iteration_num] [-p(ort) PDT_port] [-u qp timeout] [-S(l) sl type] [-x gid index] [-e(vents) use events] [-F CPU freq fail] [-b(idirectional)] [-a(ll)] [-V(ersion)] 14.4.7.2 ibv_read_bw Options The table below lists the various flags of the command.
PAGE 148
Rev 4.70 Table 42 - ibv_read_bw Flags and Options Flag Description -Q, --cq-mod Generate Cqe only after <--cq-mod> completion -N, --no peak-bw Cancel peak-bw calculation (default with peak) 14.4.8 ibv_read_lat This is a more advanced version of ib_read_lat ,and contains more flags and features than the older version and also improved algorithms. ibv_read_lat calculates the latency of RDMA read operation of message_size between a pair of machines. One acts as a server and the other as a client.
PAGE 149
Rev 4.
PAGE 150
Rev 4.
PAGE 151
Rev 4.70 They perform a ping pong benchmark on which you send packet only after you receive one. Each of the sides samples the CPU clock each time they receive a send packet, in order to calculate the latency. 14.4.10.1ibv_send_lat Synopsis ibv_send_lat [-i(b_port) ib_port] [-c(onnection_type) RC\UC\UD] [-d ib_device name] [-m(tu) mtu_size] [-s(ize) message_size] [-t(x-depth) tx_size] [I(nline_size) inline size] [-u qp timeout] [-S(L) sl type] [-x gid index] [-e(events) use events] [-n iteration_num] [-g n
PAGE 152
Rev 4.70 Table 45 - ibv_send_lat Flags and Options Flag Description -g, --post= The number of posts for each qp in the chain (default tx_depth) -I, --inline_size= The maximum size of message to be sent in “inline mode” (default 0) -e, --events Inactive during CQ events (default poll) -g, --mcg= Sends messages to multicast group with qps attached to it. -M, --MGID= In case of multicast, uses as the group MGID.
PAGE 153
Rev 4.70 Table 46 - ibv_write_bw Flags and Options Flag Description -c, --connection= Connection type RC/UC(default RC) -s, --size= The size of message to exchange (default 65536) -a, --all Runs sizes from 2 till 2^23 -t, --tx-depth= The size of tx queue (default 100) -n, --iters= The number of exchanges (at least 2, default 1000) -u, --qp-timeout= QP timeout.
PAGE 154
Rev 4.70 14.4.12.1ibv_write_lat Synopsis ibv_write_lat [-i(b_port) ib_port] [-c(onnection_type) RC\UC\UD][-m(tu) mtu_size] [-s(ize) message_size] [-t(x-depth) tx_size] [-I(nline_size) inline size] [-u qp timeout] [-S(L) sl type] [-d ib_device name] [-x gid index] [-n iteration_num] [-p(ort) PDT_port] [-a(ll)] [-V(ersion)] [-C report cycles] [-H report histogram] [-U report unsorted] 14.4.12.2ibv_write_lat Options The table below lists the various flags of the command.
PAGE 155
Rev 4.70 14.4.13 nd_write_bw This test is used for performance measuring of RDMA-Write requests in Microsoft Windows Operating Systems. nd_write_bw is performance oriented for RDMA-Write with maximum throughput, and runs over Microsoft's NetworkDirect standard. The level of customizing for the user is relatively high. User may choose to run with a customized message size, customized number of iterations, or alternatively, customized test duration time.
PAGE 156
Rev 4.70 latency, and runs over Microsoft's NetworkDirect standard. The level of customizing for the user is relatively high. User may choose to run with a customized message size, customized number of iterations, or alternatively, customized test duration time. nd_write_lat runs with all message sizes from 1B to 4MB (powers of 2), message inlining, CQ moderation. 14.4.14.1nd_write_lat Synopsis Server side: start /b /affinity 0X1 nd_write_lat -s1048576 -D10 -S 11.137.53.
PAGE 157
Rev 4.70 14.4.15.1nd_read_bw Synopsis Server side: start /b /affinity 0X1 nd_read_bw -s1048576 -D10 -S 11.137.53.1 Client side: start /b /wait /affinity 0X1 nd_read_bw -s1048576 -D10 -C 11.137.53.1 14.4.15.2nd_read_bw Options The table below lists the various flags of the command. Table 50 - nd_read_bw Options Flags Description -h Shows the Help screen. -v Shows the version number. -p Connects to the port .
PAGE 158
Rev 4.70 14.4.16.1nd_read_lat SynopsisSynopsis Server side: start /b /affinity 0X1 nd_read_lat -s1048576 -D10 -S 11.137.53.1 Client side: start /b /wait /affinity 0X1 nd_read_lat -s1048576 -D10 -C 11.137.53.1 14.4.16.2nd_read_lat Options The table below lists the various flags of the command. Table 51 - nd_read_lat Options Flags Description -h Shows the Help screen. -v Shows the version number. -p Connects to the port .
PAGE 159
Rev 4.70 14.4.17.1nd_send_bw Synopsis Server side: start /b /affinity 0X1 nd_send_bw -s1048576 -D10 -S 11.137.53.1 Client side: start /b /wait /affinity 0X1 nd_send_bw -s1048576 -D10 -C 11.137.53.1 14.4.17.2nd_send_bw Options The table below lists the various flags of the command. Table 52 - nd_send_bw Flags and Options Flag Description -h Shows the Help screen. -v Shows the version number. -p Connects to the port .
PAGE 160
Rev 4.70 14.4.18.1nd_send_lat Synopsis Server side: start /b /affinity 0X1 nd_send_lat -s1048576 -D10 -S 11.137.53.1 Client side: start /b /wait /affinity 0X1 nd_send_lat -s1048576 -D10 -C 11.137.53.1 14.4.18.2nd_send_lat Options The table below lists the various flags of the command. Table 53 - nd_send_lat Options Flag Description -h Shows the Help screen. -v Shows the version number. -p Connects to the port .
PAGE 161
Rev 4.70 14.4.19.1NTttcp Synopsis Server: ntttcp_x64.exe -r -t 15 -m 16,*, Client: ntttcp_x64.exe -s -t 15 -m 16,*, 14.4.19.2NTttcp Options The table below lists the various flags of the command.
PAGE 162
Rev 4.70 15 Troubleshooting 15.1 InfiniBand Troubleshooting Issue 1. The InfiniBand interfaces are not up after the first reboot after the installation process is completed. Suggestion: To troubleshoot this issue, follow the steps below: 1. Check that the InfiniBand driver is running on all nodes by using “vstat”. The vstat utility located at \tools, displays the status and capabilities of the network adaptor card(s). 2.
PAGE 163
Rev 4.70 Suggestion: The error message indicates that the wrong firmware image has been programmed on the adapter card. See Section 2,“Downloading Mellanox WinOF Driver,” on page 17. Issue 5. The Ethernet driver fails to start. A yellow sign appears near the "Mellanox ConnectX 10Gb Ethernet Adapter" in the Device Manager display. Suggestion: This can happen due to a hardware error. Try to disable and re-enable "Mellanox ConnectX Adapter" from the Device Manager display. Issue 6.
PAGE 164
Rev 4.70 Issue 10. • Mellanox ConnectX EN 10Gbit Ethernet Adapter device detected that the link connected to port is up, and has initiated normal operation. • Mellanox ConnectX EN 10Gbit Ethernet Adapter device detected that the link connected to port is down. This can occur if the physical link is disconnected or damaged, or if the other end-port is down. • Mismatch in the configurations between the two ports may affect the performance.
PAGE 165
Rev 4.70 Suggestion 1: Go to "Device Manager", locate the Mellanox adapter that you are debugging, right-click and go to "Information": • PCI Gen 2: should appear as "PCI-E 5.0 GT/s • PCI Gen 3: should appear as "PCI-E 8.0 GT/s" • Link Speed: 40.0Gbps/10.0Gbps Suggestion 2: To determine if the Mellanox NIC and PCI bus can achieve their maximum speed, it's best to run ib_send_bw in a loopback. On the same machine: 1. Run "start /b /affinity 0x1 ibv_write_bw" 2.
PAGE 166
Rev 4.70 15.4 General Troubleshooting Issue 1. Running Windows as VM over ESX with Mellanox HCAs Virtual machines with Windows 2008 and later guest OS fail to power on ConnectX adapter network cards are connected as Direct I/O PCI devices. To solve this issue, perform the following steps: 1. Right-click the virtual machine and select Edit Settings. 2. Click the Options tab and expand Advanced. 3. Click Edit Configuration. 4. Click Add Row. 5. Add the parameter to the new row: a.
PAGE 167
Rev 4.70 15.5 Installation Error Codes and Troubleshooting 15.5.1 Setup Return Codes Table 55 - Setup Return Codes Error Code 1603 1633 Description Troubleshooting Fatal error during installation The installation package is not supported on this platform. Contact support Make sure you are installing the right package for your platform For additional details on Windows installer return codes, please refer to: http://support.microsoft.com/kb/229683 15.5.
PAGE 168
Rev 4.70 Appendix A: Windows MPI (MS-MPI) A.1 Overview Message Passing Interface (MPI) is meant to provide virtual topology, synchronization, and communication functionality between a set of processes. With MPI you can run one process on several hosts. • Windows MPI run over the following protocols: • Sockets (Ethernet) • Network Direct (ND) A.1.1 A.2 Prerequisites • Install HPC (Build: 4.0.3906.0). • Validate traffic (ping) between the whole MPI Hosts.
PAGE 169
Rev 4.70 Step 3. [Recommended] Direct ALL TCP/UDP traffic to a lossy priority by using the “IPProtocolMatchCondition”. TCP is being used for MPI control channel (smpd), while UDP is being used for other services such as remote-desktop. Arista switches forwards the pcp bits (e.g. 802.1p priority within the vlan tag) from ingress to egress to enable any two End-Nodes in the fabric as to maintain the priority along the route.
PAGE 170
Rev 4.70 A.5.2 • New-NetQosPolicy “UDP" -IPProtocolMatchCondition UDP PriorityValue8021Action 1 • Enable-NetQosFlowControl 3 • Disable-NetQosFlowControl 0,1,2,4,5,6,7 • Enable-netadapterqos -Name Running MPI Command Examples • Running MPI pallas test over ND. mpiexec.exe -p 19020 -hosts 4 11.11.146.101 11.21.147.101 11.21.147.51 11.11.145.101 -env MPICH_NETMASK 11.0.0.0/ 255.0.0.0 -env MPICH_ND_ZCOPY_THRESHOLD -1 -env MPICH_DISABLE_ND 0 -env MPICH_DISABLE_SOCK 1 -affinity c:\\test1.
PAGE 171
Rev 4.70 Appendix B: NVGRE Configuration Scrips Examples The setup is as follow for both examples below: Hypervisor VM on VM on Hypervisor VM on VM on B.1 mtlae14 mtlae14 mtlae14 mtlae15 mtlae15 mtlae15 = = = = = = "Port1", 192.168.20.114/24 mtlae14-005, 172.16.14.5/16, mtlae14-006, 172.16.14.6/16, "Port1", 192.168.20.115/24 mtlae15-005, 172.16.15.5/16, mtlae15-006, 172.16.15.
PAGE 172
Rev 4.70 # Step 3. Configure the Provider Address and Route records on Hyper-V Host 1 (Host 1 Only) mtlae14 $NIC = Get-NetAdapter "Port1" New-NetVirtualizationProviderAddress -InterfaceIndex $NIC.InterfaceIndex -ProviderAddress 192.168.20.114 -PrefixLength 24 New-NetVirtualizationProviderRoute -InterfaceIndex $NIC.InterfaceIndex -DestinationPrefix "0.0.0.0/0" -NextHop 192.168.20.1 # Step 5.
PAGE 173
Rev 4.70 # ------- The commands from Step 2 - 4 are not persistent, Its suggested to create script is running after each OS reboot # Step 2. Configure a Subnet Locator and Route records on each Hyper-V Host (Host 1 and Host 2) mtlae14 & mtlae15 New-NetVirtualizationLookupRecord -CustomerAddress 172.16.14.5 -ProviderAddress 192.168.20.114 -VirtualSubnetID 5001 -MACAddress "00155D720100" -Rule "TranslationMethodEncap" New-NetVirtualizationLookupRecord -CustomerAddress 172.16.14.6 -ProviderAddress 192.168.20.
PAGE 174
Rev 4.70 Appendix C: Registry Keys Mellanox IPoIB and Ethernet drivers use registry keys to control the NIC operations. The registry keys receive default values during the installation of the Mellanox adapters. Most of the parameters are visible in the registry by default, however, certain parameters must be created in order to modify the default behavior of the Mellanox driver.
PAGE 175
Rev 4.70 C.2 Finding the Index Value of the Network Interface To find the index value of your Network Interface from the Device Manager please perform the following steps: Step 1. Open Device Manager, and go to Network Adapters. Step 2. Right click ->Properties on Mellanox Connect-X® Ethernet Adapter. Step 3. Go to Details tab. Step 4. Select the Driver key, and obtain the nn number.
PAGE 176
Rev 4.70 C.3 Basic Registry Keys This group contains the registry keys that control the basic operations of the NIC Value Name *JumboPacket Default Value 1500 Description The maximum size of a frame (or a packet) that can be sent over the wire. This is also known as the maximum transmission unit (MTU). The MTU may have a significant impact on the network's performance as a large packet can cause high latency. However, it can also reduce the CPU utilization and improve the wire efficiency.
PAGE 177
Rev 4.70 Value Name *ReceiveBuffers Default Value 1024 Description The number of packets each ring receives. This parameter affects the memory consumption and the performance. Increasing this value can enhance receive performance, but also consumes more system memory. In case of lack of received buffers (dropped packets or out of order received packets), you can increase the number of received buffers. The valid values are 256 up to 4096. Note: On 32-bit systems, the non-pageable memory is limited.
PAGE 178
Rev 4.70 Value Name C.4 Default Value Description RxIntModerationProfile 1 Enables the assignment of different interrupt moderation profiles for receive completions. Interrupt moderation can have a great effect on optimizing network throughput and CPU utilization. The valid values are: • 0: Low Latency Implies higher rate of interrupts to achieve better latency, or to handle scenarios where only a small number of streams are used.
PAGE 179
Rev 4.70 Value Name Default Value Description *LsoV1IPv4 1 Large Send Offload Version 2 (IPv4). The valid values are: • 0: disable • 1: enable *LsoV1IPv4 1 Large Send Offload Version 2 (IPv6). The valid values are: • 0: disable • 1: enable LSOSize 32000 The maximum number of bytes that the TCP/IP stack can pass to an adapter in a single packet. This value affects the memory consumption and the NIC performance. The valid values are MTU+1024 up to 64000.
PAGE 180
Rev 4.70 Value Name C.5 Default Value Description *TCPUDPChecksumO ffloadIPv4 3 Specifies whether the device performs the calculation of TCP or UDP checksum over IPv4. The valid values are: • 0: (disable) • 1: (Tx Enable) • 2: (Rx Enable) • 3: (Tx and Rx enable) *TCPUDPChecksumO ffloadIPv6 3 Specifies whether the device performs the calculation of TCP or UDP checksum over IPv6. The valid values are: • 0: (disable) • 1: (Tx Enable) • 2: (Rx Enable) • 3: (Tx and Rx enable) ParentBusRegPath HKLM\SY
PAGE 181
Rev 4.70 Value Name Default Value Description *InterruptModeration 1 Sets the rate at which the controller moderates or delays the generation of interrupts, making it possible to optimize network throughput and CPU utilization. When disabled, the interrupt moderation of the system generates an interrupt when the packet is received. In this mode, the CPU utilization is increased at higher data rates, because the system must handle a larger number of interrupts.
PAGE 182
Rev 4.70 Value Name *RSS Default Value 1 Description Sets the driver to use Receive Side Scaling (RSS) mode to improve the performance of handling incoming packets. This mode allows the adapter port to utilize the multiple CPUs in a multi-core system for receiving incoming packets and steering them to their destination. RSS can significantly improve the number of transactions per second, the number of connections per second, and the network throughput.
PAGE 183
Rev 4.70 Value Name ReturnPacketThreshold Default Value 0 Description The allowed number of free received packets on the rings. Any number above it will cause the driver to return the packet to the hardware immediately. When the value is set to 0, the adapter uses 2/3 of the received ring size. The valid values are: 0 to 4096. Note: This registry value is not exposed via the UI. NumTcb 16 The number of send buffers that the driver allocates for sending purposes.
PAGE 184
Rev 4.70 Value Name *HeaderDataSplit Default Value 0 Description Enables the driver to use header data split. In this mode, the adapter uses two buffers to receive the packet. The first buffer holds the header, while the second buffer holds the data. This method reduces the cache hits and improves the performance. The valid values are: • 0: disable • 1: enable Note: This registry value is not exposed via the UI. VlanId 0 Enables packets with VlanId.
PAGE 185
Rev 4.70 Value Name BlueFlame Default Value 1 Description The latency-critical Send WQEs to the device. When a BlueFlame is used, the WQEs are written directly to the PCI BAR of the device (in addition to memory), so that the device may handle them without having to access memory, thus shortening the execution latency. For best performance, it is recommended to use the BlueFlame when the HCA is lightly loaded. For high-bandwidth scenarios, it is recommended to use regular posting (without BlueFlame).
PAGE 186
Rev 4.70 Value Name PromiscuousVlan Default Value 0 Description Specifies whether a promiscuous VLAN is enabled or not. When this parameter is set, all the packets with VLAN tags are passed to an upper level without executing any filtering. The valid values are: • 0: disable • 1: enable Note: This registry value is not exposed via the UI. UseRSSForRawIP 1 The execution of RSS on UDP and Raw IP packets.
PAGE 187
Rev 4.70 This group of registry keys allows the administrator to control the TCP/IP traffic by pausing frame transmitting and/or receiving operations. By enabling the Flow Control mechanism, the adapters can overcome any TCP/IP issues and eliminate the risk of data loss. Value Name Default Value Description *FlowControl 3 When Rx Pause is enabled, the receiving adapter generates a flow control frame when its received queue reaches a pre-defined limit.
PAGE 188
Rev 4.70 Value Name *RssOrVmqPreference Default Value 0 Description Specifies whether VMQ capabilities should be enabled instead of receive-side scaling (RSS) capabilities. The valid values are: • 0: Report RSS capabilities • 1: Report VMQ capabilities Note: This registry value is not exposed via the UI. *VMQLookaheadSplit 1 Specifies whether the driver enables or disables the ability to split the receive buffers into lookahead and post-lookahead buffers.
PAGE 189
Rev 4.70 This section describes the registry keys that are used to control RoCE mode. Value Name roce_mode Default Value 0 - RoCE Description The RoCE mode. The valid values are: • 0 - RoCE • 4 - No RoCE Note: The default value depends on the WinOF package used. C.7 IPolB Registry Keys The following section describes the registry keys that are unique to IPoIB. Value Name GUIDMask Default Value 0 Description Controls the way the MAC is generated for IPoIB interface.
PAGE 190
Rev 4.70 Value Name LocalEndpointMaxAge Default Value 5 Description The maximum number of runs of the local end point DB monitor, before an unused local endpoint is removed. The endpoint age is zeroed when it is used as a source in the send flow or a destination in the receive flow. Each monitor run will increment the age of all non VMQ local endpoints. When LocalEndpointMaxAge is reached - the endpoint will be removed. The valid values are 1 up to 20.
PAGE 191
Rev 4.70 Value Name C.9 Default Value Description RssBaseCpu 1 The CPU number of the first CPU that the RSS can use. NDIS uses the default value of 0 for the base CPU number, however this value is configurable and can be changed. The Mellanox adapter reads this value from registry and sets it to NDIS on driver start-up. Value Type: DWORD The valid values are 0 up to the number of processors on the system.
PAGE 192
Rev 4.70 For more information on how to find device index nn, please refer to C.1 “Finding the Index Value of the HCA,” on page 174 . Key Name SriovEnable Key Type REG_DWORD Values 0 = RoCE (default) 1 = SR-IOV Configures the RDMA or SR-IOV mode. Note: RDMA is not supported in SR-IOV mode. • 0= auto_port1 (default) 1= auto_port2 2 = manual Configures the number of VFs to be enabled by the bus driver to each port.