Performance Best Practices for VMware vSphere® 4.0 VMware® ESX 4.0 and ESXi 4.0 vCenter Server 4.
Performance Best Practices for VMware vSphere® 4.0 You can find the most up-to-date technical documentation on the VMware Web site at: http://www.vmware.com/support/ The VMware Web site also provides the latest product updates. If you have comments about this documentation, submit your feedback to: docfeedback@vmware.com © 2007-2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws.
Contents About This Book 7 1 Hardware for Use with VMware vSphere 11 Validate Your Hardware 11 Hardware CPU Considerations 11 Hardware-Assisted Virtualization 11 Hardware-Assisted CPU Virtualization (Intel VT-x and AMD AMD-V) 11 Hardware-Assisted MMU Virtualization (Intel EPT and AMD RVI) 11 Hardware Storage Considerations 13 Hardware Networking Considerations 14 Hardware BIOS Settings 15 2 ESX and Virtual Machines 17 ESX General Considerations 17 ESX CPU Considerations 19 Hyper-Threading 20 Non-Uniform
Performance Best Practices for VMware vSphere 4.0 Glossary 43 Index 51 4 VMware, Inc.
Tables Table 1. Conventions Used in This Manual 8 Table 2-1. Symptoms of Insufficient Resources for the ESX Service Console VMware, Inc.
Performance Best Practices for VMware vSphere 4.0 6 VMware, Inc.
About This Book This book, Performance Best Practices for VMware® vSphere 4.0, provides a list of performance tips that cover the most performance-critical areas of VMware® vSphere 4.0. It is not intended as a comprehensive guide for planning and configuring your deployments. Chapter 1, “Hardware for Use with VMware vSphere,” on page 11, provides guidance on selecting hardware for use with vSphere.
Performance Best Practices for VMware vSphere 4.0 Conventions Table 1 illustrates the typographic conventions used in this manual. Table 1.
About This Book Issue Publication Idle loop KB article 1730 CPU utilization KB article 2032 Network throughput between virtual machines KB article 1428 Queue depths on QLogic cards KB article 1267 Guest storage drivers KB article 9645697 Disk outstanding commands parameter KB article 1268 VMotion CPU Compatibility Requirements for Intel Processors KB article 1991 VMotion CPU Compatibility KB article 1992 Requirements for AMD Processors Migrations with VMotion Prevented Due to CPU Mismatch—H
Performance Best Practices for VMware vSphere 4.0 10 VMware, Inc.
1 Hardware for Use with VMware vSphere 1 This chapter provides guidance on selecting and configuring hardware for use with VMware vSphere. Validate Your Hardware Test system memory for 72 hours, checking for hardware errors. Verify that all hardware in the system is on the hardware compatibility list for the VMware ESX version you will be running. Make sure that your hardware meets the minimum configuration supported by the VMware ESX version you will be running.
Performance Best Practices for VMware vSphere 4.0 Hardware-assisted MMU virtualization allows an additional level of page tables that map guest physical memory to host physical memory addresses, eliminating the need for ESX to intervene to virtualize the MMU in software. For information about configuring the way ESX uses hardware virtualization, see “Configuring ESX for Hardware-Assisted Virtualization” on page 21. 12 VMware, Inc.
Chapter 1 Hardware for Use with VMware vSphere Hardware Storage Considerations Back-end storage configuration can greatly affect performance. Refer to the following sources for more information: For SAN best practices: “Using ESX Server with SAN: Concepts” in the SAN Configuration Guide. For NFS best practices: “Advanced Networking” in the Server Configuration Guide. For iSCSI best practices: “Configuring Storage” in the Server Configuration Guide.
Performance Best Practices for VMware vSphere 4.0 Hardware Networking Considerations 14 Before undertaking any network optimization effort, you should understand the physical aspects of the network. The following are just a few aspects of the physical layout that merit close consideration: Consider using server-class network interface cards (NICs) for the best performance.
Chapter 1 Hardware for Use with VMware vSphere Hardware BIOS Settings The default hardware BIOS settings on servers might not always be the best choice for optimal performance. This section lists some of the BIOS settings you might want to check. NOTE ESX 4.0 supports Enhanced Intel SpeedStep® and Enhanced AMD PowerNow!™ CPU power management technologies that can save power when a host is not fully utilized.
Performance Best Practices for VMware vSphere 4.0 16 VMware, Inc.
2 ESX and Virtual Machines 2 This chapter provides guidance regarding ESX software itself and the virtual machines that run in it. ESX General Considerations This subsection provides guidance regarding a number of general performance considerations in ESX. Plan the deployment. Allocate enough resources, especially (in the case of ESX, but not ESXi) for the service console. Table 2-1 lists symptoms of insufficient resources for the service console. Table 2-1.
Performance Best Practices for VMware vSphere 4.0 18 ESX 4.0 introduces virtual hardware version 7. By creating virtual machines using this hardware version, or upgrading existing virtual machines to this version, a number of additional capabilities become available. Some of these, such as VMXNET3 and PVSCSI, can improve performance for many workloads.
Chapter 2 ESX and Virtual Machines ESX CPU Considerations This subsection provides guidance regarding CPU considerations in ESX. CPU virtualization adds varying amounts of overhead depending on the percentage of the virtual machine’s workload that can be executed on the physical processor as is and the cost of virtualizing the remaining workload: For many workloads, CPU virtualization adds only a very small amount of overhead, resulting in performance essentially comparable to native.
Performance Best Practices for VMware vSphere 4.0 Although some recent operating systems (including Windows Vista, Windows Server 2008, and Windows 7) use the same HAL (hardware abstraction layer) or kernel for both UP and SMP installations, many operating systems can be configured to use either a UP HAL/kernel or an SMP HAL/kernel.
Chapter 2 ESX and Virtual Machines For nearly all workloads, custom hyper-threading settings are not necessary. In cases of unusual workloads that interact badly with hyper-threading, however, choosing the None or Internal hyper-threading option might help performance. For example, an application with cache-thrashing problems might slow down an application sharing its physical CPU core.
Performance Best Practices for VMware vSphere 4.0 Hardware-assisted MMU is supported for both AMD and Intel processors beginning with ESX 4.0 (AMD processor support started with ESX 3.5 Update 1). On processors that support it, ESX 4.0 by default uses hardware-assisted MMU virtualization for virtual machines running certain guest operating systems and uses shadow page tables for others (see http://communities.vmware.com/docs/DOC-9882 for a detailed list).
Chapter 2 ESX and Virtual Machines ESX Memory Considerations This subsection provides guidance regarding memory considerations in ESX. Memory Overhead Virtualization causes an increase in the amount of physical memory required due to the extra memory needed by ESX for its own code and for data structures.
Performance Best Practices for VMware vSphere 4.0 While ESX allows significant memory overcommitment, usually with little or no impact on performance, you should avoid overcommitting memory to the point that it results in heavy memory reclamation. If the memory savings provided by transparent page sharing reach their limit and the total memory demands exceed the amount of machine memory in the system, ESX will need to use additional memory reclamation techniques.
Chapter 2 ESX and Virtual Machines Large Memory Pages for Hypervisor and Guest Operating System In addition to the usual 4KB memory pages, ESX also makes 2MB memory pages available (commonly referred to as “large pages”). By default ESX assigns these 2MB machine memory pages to guest operating systems that request them, giving the guest operating system the full advantage of using large pages.
Performance Best Practices for VMware vSphere 4.0 ESX Storage Considerations This subsection provides guidance regarding storage considerations in ESX. ESX supports raw device mapping (RDM), which allows management and access of raw SCSI disks or LUNs as VMFS files. An RDM is a special file on a VMFS volume that acts as a proxy for a raw device. The RDM file contains meta data used to manage and redirect disk accesses to the physical device.
Chapter 2 ESX and Virtual Machines Before performing an alignment, carefully evaluate the performance impact of the unaligned VMFS partition on your particular workload. The degree of improvement from alignment is highly dependent on workloads and array types. You might want to refer to the alignment recommendations from your array vendor for further information.
Performance Best Practices for VMware vSphere 4.0 ESX Networking Considerations This subsection provides guidance regarding networking considerations in ESX. 28 In a native environment, CPU utilization plays a significant role in network throughput. To process higher levels of throughput, more CPU resources are needed. The effect of CPU resource availability on the network throughput of virtualized applications is even more significant.
3 Guest Operating Systems 3 This chapter provides guidance regarding the guest operating systems running in virtual machines. Guest Operating System General Considerations Use guest operating systems that are supported by ESX. See the Guest Operating System Installation Guide for a list. NOTE VMware Tools might not be available for unsupported guest operating systems. Install the latest version of VMware Tools in the guest operating system. Make sure to update VMware Tools after each ESX upgrade.
Performance Best Practices for VMware vSphere 4.0 ESX VMI support can be enabled for a virtual machine through the vSphere Client by selecting Edit virtual machine settings, choosing the Options tab, selecting Paravirtualization, then marking the box next to Support VMI Paravirtualization. Enabling VMI for a virtual machine reduces the number of available PCI slots in the guest operating system running in that virtual machine by one.
Chapter 3 Guest Operating Systems Guest Operating System CPU Considerations In SMP guests the guest operating system can migrate processes from one vCPU to another. This migration can incur a small CPU overhead. If the migration is very frequent it might be helpful to pin guest threads or processes to specific vCPUs. (Note that this is another reason not to configure virtual machines with more vCPUs than they need.) Many operating systems keep time by counting timer interrupts.
Performance Best Practices for VMware vSphere 4.0 Guest Operating System Storage Considerations The default storage adapter in ESX 4.0 is either BusLogic or LSILogic, depending on the guest operating system. However, ESX 4.0 also includes a new virtual storage adapter, paravirtualized SCSI (PVSCSI, also called VMware Paravirtual).
Chapter 3 Guest Operating Systems Guest Operating System Networking Considerations The default virtual network adapter emulated inside a guest is either an AMD PCnet32 device (vlance) or an Intel E1000 device (E1000). VMware also offers the VMXNET family of paravirtualized network adapters, however, that provide better performance than these default adapters and should be used for optimal performance within any guest operating system for which they are available.
Performance Best Practices for VMware vSphere 4.0 For VMXNET, the default number of receive and transmit buffers is 100 each, with the maximum possible being 128. For Enhanced VMXNET, the default number of receive and transmit buffers are 150 and 256, respectively, with the maximum possible receive buffers being 512. You can alter these settings by changing the buffer size defaults in the .vmx (configuration) files for the affected virtual machines.
4 Virtual Infrastructure Management 4 This chapter provides guidance regarding resource management best practices. Most of the suggestions included in this section can be implemented using the vSphere Client connected to a VMware vCenter server. Some can also be implemented using the vSphere Client connected to an individual ESX host.
Performance Best Practices for VMware vSphere 4.0 VMware vCenter Best Practices This section lists VMware vCenter practices and configurations recommended by VMware for optimal performance. This section also includes a few features that are controlled or accessed through vCenter. Large numbers of managed hosts, managed virtual machines, and connected VMware vSphere Clients can affect the performance of a vCenter server.
Chapter 4 Virtual Infrastructure Management VMware VMotion and Storage VMotion ESX 4.0 introduces virtual hardware version 7. Because virtual machines running on hardware version 7 can’t run on prior versions of ESX, they can be moved using VMware VMotion only to other ESX 4.0 hosts. Before using VMware Storage VMotion make sure you have sufficient storage bandwidth between the ESX host where the virtual machine is running and both the source and destination storage arrays.
Performance Best Practices for VMware vSphere 4.0 VMware Distributed Resource Scheduler (DRS) Best Practices Clustering configurations can have a significant impact on performance. This section lists Distributed Resource Scheduler (DRS) practices and configurations recommended by VMware for optimal performance. When deciding which hosts to group into DRS clusters, try to choose hosts that are as homogeneous as possible in terms of CPU and memory.
Chapter 4 Virtual Infrastructure Management 300 The default frequency is 300 seconds, but it can be set to anything between 60 seconds and 3600 seconds. Users are discouraged from changing the default value. This is recommended only in specific cases where the user would like to invoke the algorithm less frequently at the cost of potential loss in throughput.
Performance Best Practices for VMware vSphere 4.0 VMware Distributed Power Management (DPM) Best Practices Distributed Power Management (DPM) conserves power by migrating virtual machines to fewer hosts when utilizations are low. DPM is most appropriate for clusters in which composite virtual machine demand varies greatly over time; for example, clusters in which overall demand is higher during the day and significantly lower at night.
Chapter 4 Virtual Infrastructure Management VMware Fault Tolerance Best Practices VMware Fault Tolerance (FT) provides continuous virtual machine availability in the event of a server failure. For each virtual machine there are two FT-related actions that can be taken: turning FT on or off and enabling or disabling FT.
Performance Best Practices for VMware vSphere 4.0 42 Make sure that power management scheme settings (both in the BIOS and in ESX) that cause CPU frequency scaling are consistent between the hosts on which the primary and secondary virtual machines run. Enabling CPU reservations for the primary virtual machine (which will be duplicated for the secondary virtual machine) to ensure that the secondary gets CPU cycles when it requires them.
Glossary A AMD Virtualization (AMD-V) AMD’s version of virtualization assist, included in some 64-bit AMD processors. See also Virtualization Assist. B Ballooning A technique used in VMware ESX to reclaim the guest memory pages that are considered the least valuable by the guest operating system. This is accomplished using the vmmemctl driver, which is installed as part of the VMware Tools suite.
Performance Best Practices for VMware vSphere 4.0 EPT (Extended Page Tables) Intel’s implementation of hardware virtual MMU. F Fault Tolerance (FT) A feature in vSphere 4.0 that runs a secondary copy of a virtual machine on a secondary host and seamlessly switches to that secondary copy in the event of failure of the primary host. Fibre Channel A networking technology used for storage. See also iSCSI, NAS, NFS, and SAN.
Glossary iSCSI A protocol allowing SCSI commands to be transmitted over TCP/IP (typically using ordinary Ethernet cabling). An iSCSI client is called an initiator (can be software or hardware); an iSCSI server is called a target. J Jumbo frames Ethernet frames with a payload of more than 1,500 bytes. Because there is a CPU cost per network packet, larger packets can reduce the CPU cost of network traffic. Not all Gigabit Ethernet cards or switches support jumbo frames.
Performance Best Practices for VMware vSphere 4.0 NIC Historically meant “network interface card.” With the recent availability of multi-port network cards, as well as the inclusion of network ports directly on system boards, the term NIC is now sometimes used to mean “network interface controller” (of which there may be more than one on a physical network card or system board).
Glossary R RAID (Redundant Array of Inexpensive Disks) A technology using multiple hard disks to improve performance, capacity, or reliability. Raw Device Mapping (RDM) The use of a mapping file in a VMFS volume to point to a raw physical device. RVI (Rapid Virtualization Indexing) AMD’s implementation of hardware virtual MMU. Also called NPT. S SAN See Storage Area Network. Secure Virtual Machine (SVM) Another name for AMD’s version of virtualization assist, included in some 64-bit AMD processors.
Performance Best Practices for VMware vSphere 4.0 Thrashing A situation that occurs when virtual or physical memory is not large enough to hold the full working set of a workload. This mismatch can cause frequent reading from and writing to a paging file, typically located on a hard drive, which can in turn severely impact performance. TLB (Translation Lookaside Buffer) A CPU cache used to hold page table entries.
Glossary vlance One of the virtual network adapters available in a virtual machine running in ESX. The vlance adapter emulates an AMD PCnet32 device. Note that in some cases NIC morphing can automatically convert a vlance device into a VMXNET device. See also NIC Morphing, E1000, and VMXNET. VMFS (Virtual Machine File System) A high performance cluster file system. VMI (Virtual Machine Interface) A paravirtualization interface supported by ESX. See also Paravirtualization.
Performance Best Practices for VMware vSphere 4.0 50 VMware, Inc.
Index Numerics 10 Gigabit Ethernet and NetQueue 14 64-bit DMA addresses 14 CPU affinity and hyper-threading 20 CPU virtualization hardware-assisted 11 A D active/active storage arrays policy 27 active/passive storage arrays policy 27 alignment file system partitions 26 AMD PCnet32 device 33 AMD PowerNow! 15 AMD-V 15, 21 anti-virus programs scheduling 29 disks eager-zeroed 26 independent nonpersistent 26 independent persistent 26 lazy-zeroed 26 snapshot 26 thick 26 thin 26 Distributed Power Management
Performance Best Practices for VMware vSphere 4.0 file system partitions alignment 26 Floppy drives 17 FT (Fault Tolerance) 41 lazy-zeroed disks 26 limits and DRS 39 LPT ports 17 LSILogic virtual storage adapter 32 H HA (High Availability) 40 HAL UP vs.
Index P page sharing 23 paravirtual timer 30 paravirtualization 29 PCI bus architecture 14 PCI Express bus architecture 14 PCIe bus architecture 14 PCI-X bus architecture 14 PCnet32 device 33 PVSCSI virtual storage adapter 18, 32 Q queue depth 13 driver 27 virtual SCSI driver 32 R rapid virtualization indexing 11, 15 raw device mapping 26 RDM 26 receive buffers insufficient 33 reservation use of 35 reservations and DRS 39 resource pools 35, 39 RVI (rapid virtualization indexing) 11, 15 S shadow page tab
Performance Best Practices for VMware vSphere 4.
