Deployment guide VMware vSphere Bitfusion on Dell EMC PowerEdge servers Abstract VMware vSphere Bitfusion is a software solution that you can deploy on Dell EMC PowerEdge R740xd and C4140 servers. The solution virtualizes hardware resources to provide a pool of shared resources that are accessible to any virtual machine in the network.
Revisions Revisions Date Description September 2020 Initial release March 2021 Document updated with support information for VMware vSphere Bitfusion 2.5 May 2021 Document updated with support information for VMware vSphere Bitfusion 3.0 Acknowledgements Authors: Jay Engh, Chris Gully Support: Gurupreet Kaushik, Sherry Keller The information in this publication is provided “as is.” Dell Inc.
Table of contents Table of contents Revisions.............................................................................................................................................................................2 Acknowledgements .............................................................................................................................................................2 Table of contents .................................................................................................
Table of contents 7.3 Prepare and configure the GPU hosts and client cluster for PVRDMA ............................................................20 7.3.1 Tag a VMkernel adapter for PVRDMA .............................................................................................................21 7.3.2 Add and manage hosts on the Bitfusion distributed switch ..............................................................................22 7.
Executive summary Executive summary Modern computational requirements have evolved and diverged with the wide acceptance and use of containers and virtualization. Today’s workloads are presenting challenges to the CPU-centric paradigm which has been a key determinant of customers' server performance and utility needs. General purpose CPUs no longer adequately handle these new workloads, whether they are artificial intelligence, machine learning or virtual desktops.
New and enhanced in VMware vSphere Bitfusion 3.0 1 New and enhanced in VMware vSphere Bitfusion 3.0 Following are some of the important features that are included with the release of VMware vSphere Bitfusion 3.0: • • • New installation experience for upcoming VMware vSphere Bitfusion servers. Improved support for VMware vSphere Bitfusion servers with multiple networks. Seamless installation of the NVIDIA System Management Interface (nvidia-smi) application on VMware vSphere Bitfusion client servers.
Audience and scope 2 Audience and scope This deployment guide includes step-by-step instructions for deployment and configuration of the VMware vSphere Bitfusion appliance on Dell EMC PowerEdge R740xd and C4140 rack servers. This deployment guide makes certain assumptions about the prerequisite knowledge of the deployment personnel and the hardware they are using.
Overview 3 Overview With the new VMware vSphere Bitfusion software, graphics processing units (GPU) are no longer isolated from other resources. GPUs are now shared in a virtualized pool of resources and you can access them through any virtual machine in the infrastructure as shown in Figure 1. Similar to processors and storage resources, GPU deployments can now benefit from optimized utilization, reduced Capex and Opex, and accelerated development and deployment of R&D resources.
Overview Bitfusion GPU sharing model Deployment of VMware Bitfusion on the Dell EMC PowerEdge servers provide an infrastructure solution incorporating the best-in class hardware from Dell EMC with core VMware products. Virtualization of computation, storage, networking and accelerators is delivered on a cluster of PowerEdge servers. The combination of VMware vSphere Bitfusion software on the Dell EMC PowerEdge hardware described in this document has been validated in Dell EMC labs.
Component overview 4 Component overview This section briefly describes the components that support VMware vSphere Bitfusion and their key capabilities to help you deploy the software. 4.1 DELL EMC PowerEdge R740xd server The PowerEdge R740xd server provides the benefit of scalable storage performance and data set processing. This 2U, 2-socket platform brings you scalability and performance to adapt to a variety of applications.
Component overview Key capabilities: • • • Unthrottled performance and superior thermal efficiency with patent-pending interleaved GPU system design* No-compromise (CPU + GPU) acceleration technology up to 500 TFLOPS / U+ using the NVIDIA® Tesla™V100 with NVLink™ 2.4KW PSUs help future-proof for next generation GPUs Front view of a Dell EMC PowerEdge C4140 Rear view of a Dell EMC PowerEdge C4140 Internal view of a Dell EMC PowerEdge R740xd displaying a NVIDIA T4 GPU card 4.
Component overview 4.4 Dell EMC Networking S5248F-ON Switch The S5200F-ON series introduces optimized 25GbE and 100GbE open networking connectivity for servers/storage in demanding web and cloud environments. Innovative next-generation top-of-rack family of 25GbE switches providing optimized performance both in-rack and between-racks, cost-effective 50/100GbE leaf/spine fabric, and migration capabilities for future connectivity needs.
Component overview NVIDIA V100 for PCIE NVIDIA V100 for NVLink 4.7 Mellanox ConnextX-5 Dual Port 10/25GbE Adapter ConnectX-5 EN supports two ports of 25Gb Ethernet connectivity, sub-600 ns latency, and very high message rate, plus PCIe switch and NVMe over Fabric offloads, providing the highest performance and most flexible solution for the most demanding applications and markets: Machine Learning, Data Analytics, and more.
Pre-deployment requirements and introduction to new features 5 Pre-deployment requirements and introduction to new features This section describes the pre-deployment requirements for configuring VMware vSphere Bitfusion. Figure 13 shows an illustration of the components involved in creating the Bitfusion client-server cluster. Bitfusion sample client-server cluster 5.1 GPU hosts The Bitfusion OVA is deployed on the GPU hosts. Below are the Dell EMC customized images for VMware ESXi 7.
Pre-deployment requirements and introduction to new features and vMotion traffic can be enabled. For information on certified and supported vSAN ReadyNode, see the VMware Compatibility Guide. 5.3 Introduction to new features 5.3.1 Remote clients VMware vSphere Bitfusion 2.0 allows you to run client servers that are a part of the same vCenter software as of the Bitfusion server. With VMware vSphere Bitfusion 2.
Pre-deployment requirements and introduction to new features Improved management of bare-metal server workloads 5.3.3 Improved health checks In earlier releases of vSphere Bitfusion, health checks are performed with a simple command-line parameter. With the integration of vCenter, a new graphical user interface is introduced to display the command-line outputs within a dialog box.
Pre-deployment requirements and introduction to new features 5.6 Client virtual machine The client cluster has a virtual machine with CentOS installed with the required NVIDIA tools and drivers. You can use this virtual machine to access the GPUs remotely. Install the following components on the CentOS virtual machine to set up the client cluster: • • • • • Python 3 and pip3 package manager Compute Unified Device Architecture 10.
Solution overview 6 Solution overview Following is the solution overview for the deployment instructions provided in the rest of the document. 6.1 Architecture The GPU hosts and client cluster architecture shown in Figure 14 is the reference architecture for the use case described in the forthcoming section.
Solution overview 6.2 Component information Component PowerEdge 740xd PowerEdge C4140 vSAN RN 740xd NVIDIA T4 NVIDIA V100 PCIe NVIDIA V100 SXM2 S5248F Client VM Hypervisor Hypervisor 6.3 VLANs and IP subnet information. VLAN ID 96 90 10 20 19 Hardware information that is used in this deployment guide. Description Specification GPU Host 2 * Intel(R) Xeon(R) Gold 6242 CPU @ 2.
Deployment and configuration 7 Deployment and configuration The following section describes the step-by-step instructions to deploy Bitfusion appliances on the GPU hosts and a quick test to showcase the GPU accessibility by a remote VM assuming all the pre-requisites are met. 7.1 Verify the GPU host hardware configuration Follow the steps: 1. Login to iDRAC and browse to the Configuration > BIOS settings > System Profile Settings and verify that the System Profile is set to Performance. 2.
Deployment and configuration 3. Create pvrdma as a new distributed portgroup pvrdma using the New Distributed Port Group wizard under the Bitfusion distributed switch created in the previous step. Set the Port Binding to ephemeral and VLAN to 90. 4. Create VMKernel port for PVRDMA on all hosts by clicking on Configure > Networking > VMkernel Adapters > Add Networking wizard. Create vmk1 on the GPU hosts and then create vmk3 on the client cluster. 5.
Deployment and configuration Tagging the VMkernel adapter 7.3.2 Add and manage hosts on the Bitfusion distributed switch Follow the steps: 1. Attach the GPU hosts and client cluster hosts. 2. Assign the two RDMA NICs (vmnic6 & vmnic7) from each host. 3. Assign the vmkernel port (vmk3 on client cluster hosts and vmk1 on GPU hosts) on each host created in the earlier step to pvrdma port group.
Deployment and configuration Adding and managing hosts on the Bitfusion distributed switch. 7.4 Deploy the Open Virtual Appliance (OVA) to create the bitfusionserver-1 virtual machine Follow the steps: 1. 2. 3. 4. Use Deploy OVF Template action on the C4140 to deploy the first appliance. Select the appliance OVA file, bitfusion-server-2.0.0-11.ova. Select the folder and provide the name for the first appliance bitfusion-server-1.
Deployment and configuration Enter the vCenter URL and the vCenter administrator account credentials 9. Extract the TLS certificate from the browser navigation pane. The hexadecimal is case sensitive. a. For Google Chrome: click on the lock icon or not secure icon to the left of the URL bar in the browser and then click Certificate > Details > Thumbprint. b.
Deployment and configuration Extracting the TLS certificate from Google Chrome web browser navigation pane 10. Provide credentials for user customer. This account is used to login to the appliance for any troubleshooting. 11. Select the checkbox for NVIDIA driver license agreement. The appliance has connectivity to the internet to download the NVIDIA software. Select the checkbox for NVIDIA driver license agreement 12. Provide information for Network Adapter 1 settings.
Deployment and configuration • • • • • MTU – 9000 Gateway – 100.71.x.x DNS – 100.71.x.x Search domain – oseadc.local NTP – 100.71.x.x Configuring the network adapter 1 settings 13. Provide information for Network Adapter 2 settings. This provides the appliance access to the data plane (pvrdma) for GPU traffic. Select the checkbox for configuring network adapter 2 settings. • • • 26 IPv4 address – 172.16.6.
Deployment and configuration Configure the network adapter 2 settings and select the checkbox 14. Click Next to complete the deployment configuration and wait for the task to complete. Refrain from powering on the virtual machine. 7.5 Edit the bitfusion-server-1 hardware settings Follow the steps: 1. Under the Virtual Hardware tab, verify if the number of vCPUs is 8 • Minimum No. of vCPUs = 4x No. of GPU devices attached to the appliance. In this case, 4 x 2 GPUs i.e. 8 2.
Deployment and configuration Editing the bitfusion-server-1 hardware settings 3. Click on Add New Device and add two PCIe devices. Select the PCI devices from the drop-down menu. • • 28 0000:18:00.0 | GV100GL V100 SXM2 16GB 0000:3b:00.
Deployment and configuration Adding PCI devices from the drop-down menu 4. Click on Add New Device and select Network adapter. • • 29 Browse and select the network pvrdma that is created on the Bitfusion distributed switch earlier.
Deployment and configuration Setting the adapter type as PVRDMA the device protocol to RoCE v2 5. Under the Virtual Machine Options tab, select Advanced > Configuration parameters and then select Edit Configuration. Edit the pciPassthru.64bitMMIOSizeGB parameter to 64. • pciPassthru.64bitMMIOSizeGB= , where n equals (num-cards * size-of-card-in-GB) rounded up to NEXT power of 2. In this case, 2x 16 → 32, rounded to next power of 2 i.e. 64 6.
Deployment and configuration Notification on the top of the window indicating successful deployment of Bitfusion 7. Select Menu > Bitfusion and wait for the Bitfusion plug-in GUI to load. Bitfusion plug-in GUI bitfusion-server-1 virtual machine 8. Login to the bitfusion-1 appliance using the same credentials used during deploying the appliance. Run the following command to reload, restart and edit the bitfusion-manager service file: sudo vi /usr/lib/systemd/system/bitfusion-manager.
Deployment and configuration Editing the bitfusion-manager service file 9. Use the following commands to first save the file, and then restart the bitfusion-manager service for the changes to take effect: sudo systemctl daemon-reload sudo systemctl restart bitfusion-manager 7.6 Deploy the Open Virtual Appliance (OVA) to create the bitfusionserver-2 virtual machine Follow the steps: 1. Select the appliance OVA file, bitfusion-server-2.0.0-11.ova. 2.
Deployment and configuration • • 7.7 CIDR–24 MTU–9000 Edit the bitfusion-server-2 virtual machine hardware settings Follow the steps: 1. Under the Virtual Hardware tab, Verify the number of vCPUs is 8 • Minimum No. of vCPUs = 4x No. of GPU devices attached to the appliance. In this case, 4 x 2 GPUs i.e. 8 2. Verify the memory is set to 48GB and select the checkbox Reserve all guest memory. • Minimum GB of memory = 1.5x aggregate total of GPU memory on all GPU cards passed through. In this case, 1.
Deployment and configuration Select Enable Bitfusion from the Actions menu 8. A window pops up listing the options to enable as a client or server. Select the For a server, this will allow it to be used as a GPU server radio button and click on ENABLE. This adds guest variables informing the server it is not the first GPU server in the Bitfusion cluster. 9. Power on the virtual machine and wait for the Bitfusion plugin UI to show the additional appliance and additional GPUs added to the cluster.
Deployment and configuration Bitfusion plug-in GUI for bitfusion-server-2 virtual machine 10. Login to the bitfusion-2 appliance using the same credentials used during deploying the appliance. Run the following command to reload, restart and edit the bitfusion-manager service file: sudo vi /usr/lib/systemd/system/bitfusion-manager.service Add the line Environment=BF_IB_GID_INDEX=1 at the end of the [Service] section 11.
Deployment and configuration 6. Verify the network adapter 2 settings: • • • 7.9 IPv4 address–172.16.6.22 CIDR–24 MTU–9000 Edit the bitfusion-server-3 virtual machine hardware settings Follow the steps: 1. Under the Virtual Hardware tab, verify the number of vCPUs is 8 • Minimum No. of vCPUs = 4x No. of GPU devices attached to the appliance. In this case, 4 x 2 GPUs i.e. 8 2. Verify that the memory is set to 48GB and select the checkbox Reserve all guest memory. • Minimum GB of memory = 1.
Deployment and configuration Bitfusion plug-in GUI for bitfusion-server-3 virtual machine 10. In the Bitfusion user interface, verify that the total available GPUs is 6 and the allocation is 0 in the Cluster GPU Allocation graph. All six available GPUs in the cluster 11. Login to the bitfusion-3 appliance using the same credentials used during deploying the appliance.
Deployment and configuration sudo systemctl daemon-reload sudo systemctl restart bitfusion-manager 7.10 Alternate method of installing an additional Bitfusion server Alternatively, you can add a new server to your vSphere Bitfusion cluster using the deploy procedure on the primary server.
Deployment and configuration 7.11 Improved support for multiple networks During the deployment process of a vSphere Bitfusion server, you must configure at least Network Adapter 1 as it is used for management and data traffic. Network Adapter 2, Network Adapter 3 and Network Adapter 4 are all optional and are only used for data traffic. With vSphere Bitfusion 3.0, you can now add network interfaces for data traffic management after the deployment process of a server. Follow the steps: 1.
Deployment and configuration 6. Run Bitfusion commands to list all the GPUs available on the GPU server cluster by using the command: bitfusion – list_gpus List all available GPUs in the cluster 7. Run Bitfusion commands over the PVRDMA network targeting each GPU server and verify that the available resources are listed. Use the following command: bitfusion list_gpus -l 172.16.6.x 8.
Deployment and configuration Client cluster access is successfully provided to the GPU resources 7.13 Support for remote clients and bare-metal server VMware vSphere Bitfusion 2.5 allows you to create tokens to enable clients that are installed on different vCenter server instances or on a Tanzu Kubernetes Grid (TKG) container or on a bare-metal client server. Follow the steps: 1. 2. 3. 4. 5. 6. From the vSphere Client software Menu tab, select Bitfusion. From the Tokens tab, select New Token.
Deployment and configuration 7.14 Support for backup and restore Note: To upgrade your cluster, back up the VMware environment, deploy new server virtual machines with vSphere Bitfusion 2.5 and restore the backup. To upgrade the operating system of a client, install the latest version of CentOS, Red Hat Enterprise Linux or Ubuntu Server package. To upgrade the VMware vSphere Bitfusion software from version 2.0 to 2.5, follow the steps: 1.
Deployment and configuration a. Deploy a new server virtual machine using a vSphere Bitfusion 2.5 OVF Template. b. During deployment, enter the same hostname as of the primary vSphere Bitfusion 2.0 server. c. While setting up the new virtual machine, add the same number of GPUs as of the vSphere Bitfusion 2.0 server. d. Add a guestinfo.bitfusion.server.host-id configuration parameter. The value of the parameter should match the host ID of your primary server with vSphere Bitfusion 2.
Getting help 8 Getting help 8.1 Contacting Dell EMC Dell EMC provides several online and telephone-based support and service options. Availability varies by country, region, and product, and some services may not be available in your area. To contact Dell EMC for sales, technical assistance, or customer service issues, see https://www.dell.com/contactdell.
