vSphere Resource Management Guide Update 1 ESX 4.0 ESXi 4.0 vCenter Server 4.0 This document supports the version of each product listed and supports all subsequent versions until the document is replaced by a new edition. To check for more recent editions of this document, see http://www.vmware.com/support/pubs.
vSphere Resource Management Guide You can find the most up-to-date technical documentation on the VMware Web site at: http://www.vmware.com/support/ The VMware Web site also provides the latest product updates. If you have comments about this documentation, submit your feedback to: docfeedback@vmware.com Copyright © 2006–2011 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws.
Contents Updated Information 5 About This Book 7 1 Getting Started with Resource Management 9 What Is Resource Management? 9 Configuring Resource Allocation Settings 10 Viewing Resource Allocation Information 13 Admission Control 16 2 Managing CPU Resources 17 CPU Virtualization Basics 17 Administering CPU Resources 18 3 Managing Memory Resources 25 Memory Virtualization Basics 25 Administering Memory Resources 28 4 Managing Resource Pools 37 Why Use Resource Pools? 38 Create Resource Pools 39 Add V
vSphere Resource Management Guide 7 Viewing DRS Cluster Information 67 Viewing the Cluster Summary Tab Using the DRS Tab 69 67 8 Using NUMA Systems with ESX/ESXi 73 What is NUMA? 73 How ESX/ESXi NUMA Scheduling Works 74 VMware NUMA Optimization Algorithms and Settings 75 Resource Management in NUMA Architectures 76 Specifying NUMA Controls 77 A Performance Monitoring Utilities: resxtop and esxtop 81 Using the esxtop Utility 81 Using the resxtop Utility 81 Using esxtop or resxtop in Interactive Mode 82
Updated Information This vSphere Resource Management Guide is updated with each release of the product or when necessary. This table provides the update history of the vSphere Resource Management Guide. Revision Description EN-000264-03 Included a point in “Multicore Processors,” on page 19 section. EN-000264-02 Removed references to CPU.MachineClearThreshold since this advanced CPU attribute is not available through the vSphere Client.
vSphere Resource Management Guide 6 VMware, Inc.
About This Book ® The vSphere Resource Management Guide describes resource management for VMware ESX™, ESXi, and VMware vCenter™ Server environments. This guide focuses on the following topics.
vSphere Resource Management Guide Technical Support and Education Resources The following technical support resources are available to you. To access the current version of this book and other books, go to http://www.vmware.com/support/pubs. Online and Telephone Support To use online support to submit technical support requests, view your product and contract information, and register your products, go to http://www.vmware.com/support.
1 Getting Started with Resource Management To understand resource management, you must be aware of its components, its goals, and how best to implement it in a cluster setting. Resource allocation settings for a virtual machine (shares, reservation, and limit) are discussed, including how to set them and how to view them. Also, admission control, the process whereby resource allocation settings are validated against existing resources is explained.
vSphere Resource Management Guide Resource Consumers Virtual machines are resource consumers. The default resource settings assigned during creation work well for most machines. You can later edit the virtual machine settings to allocate a share-based percentage of the total CPU and memory of the resource provider or a guaranteed reservation of CPU and memory.
Chapter 1 Getting Started with Resource Management Resource Allocation Shares Shares specify the relative priority or importance of a virtual machine (or resource pool). If a virtual machine has twice as many shares of a resource as another virtual machine, it is entitled to consume twice as much of that resource when these two virtual machines are competing for resources. Shares are typically specified as High, Normal, or Low and these values specify share values with a 4:2:1 ratio, respectively.
vSphere Resource Management Guide For example, assume you have 2GHz available and specify a reservation of 1GHz for VM1 and 1GHz for VM2. Now each virtual machine is guaranteed to get 1GHz if it needs it. However, if VM1 is using only 500MHz, VM2 can use 1.5GHz. Reservation defaults to 0. You can specify a reservation if you need to guarantee that the minimum required amounts of CPU or memory are always available for the virtual machine.
Chapter 1 Getting Started with Resource Management Figure 1-1. Single Host with Two Virtual Machines ESX/ESXi host VM-QA VM-Marketing In the following example, assume that VM-QA is memory intensive and accordingly you want to change the resource allocation settings for the two virtual machines to: n Specify that, when system memory is overcommitted, VM-QA can use twice as much memory and CPU as the Marketing virtual machine.
vSphere Resource Management Guide Table 1-2. CPU Resource Allocation Field Description Total Capacity Guaranteed CPU allocation, in megahertz (MHz), reserved for this object. Reserved Capacity Number of megahertz (MHz) of the reserved allocation that this object is using. Available Capacity Number of megahertz (MHz) not reserved. Memory Section The following information about memory resource allocation is shown: Table 1-3.
Chapter 1 Getting Started with Resource Management Virtual Machine Resource Allocation Tab A Resource Allocation tab is available when you select a virtual machine from the inventory panel. This Resource Allocation tab displays information about the CPU and memory resources for the selected virtual machine. CPU Section These bars display the following information about host CPU usage: Table 1-5. Host CPU Field Description Consumed Actual consumption of CPU resources by the virtual machine.
vSphere Resource Management Guide Table 1-9. Resource Settings Field Description Reservation Guaranteed memory allocation for this virtual machine. Limit Upper limit for this virtual machine’s memory allocation. Shares Memory shares for this virtual machine. Configured User-specified guest physical memory size.
Managing CPU Resources 2 ESX/ESXi hosts support CPU virtualization. When you utilize CPU virtualization, you should understand how it works, its different types, and processorspecific behavior. Also, you need to be aware of the performance implications of CPU virtualization.
vSphere Resource Management Guide Hardware-Assisted CPU Virtualization Certain processors (such as Intel VT and AMD SVM) provide hardware assistance for CPU virtualization. When using this assistance, the guest can use a separate mode of execution called guest mode. The guest code, whether application code or privileged code, runs in the guest mode. On certain events, the processor exits out of guest mode and enters root mode.
Chapter 2 Managing CPU Resources View Processor Information You can access information about current CPU configuration through the vSphere Client or using the vSphere SDK. Procedure 1 In the vSphere Client, select the host and click the Configuration tab. 2 Select Processors. You can view the information about the number and type of physical processors and the number of logical processors. NOTE In hyperthreaded systems, each hardware thread is a logical processor.
vSphere Resource Management Guide The ESX CPU scheduler can interpret processor topology, including the relationship between sockets, cores, and logical processors. The scheduler uses topology information to optimize the placement of virtual CPUs onto different sockets to maximize overall cache utilization, and to improve cache affinity by minimizing virtual CPU migrations. In undercommitted systems, the ESX CPU scheduler spreads load across all sockets by default.
Chapter 2 Managing CPU Resources Hyperthreading and ESX/ESXi Hosts An ESX/ESXi host enabled for hyperthreading should behave similarly to a host without hyperthreading. You might need to consider certain factors if you enable hyperthreading, however. ESX/ESXi hosts manage processor time intelligently to guarantee that load is spread smoothly across processor cores in the system.
vSphere Resource Management Guide 3 Select a hyperthreading mode for this virtual machine from the Mode drop-down menu. Hyperthreaded Core Sharing Options You can set the hyperthreaded core sharing mode for a virtual machine using the vSphere Client. Table 2-1 shows the available choices for this mode. Table 2-1. Hyperthreaded Core Sharing Modes Option Description Any The default for all virtual machines on a hyperthreaded system.
Chapter 2 Managing CPU Resources Using CPU Affinity By specifying a CPU affinity setting for each virtual machine, you can restrict the assignment of virtual machines to a subset of the available processors in multiprocessor systems. By using this feature, you can assign each virtual machine to processors in the specified affinity set. In this context, the term CPU refers to a logical processor on a hyperthreaded system and refers to a core on a non-hyperthreaded system.
vSphere Resource Management Guide n When you move a virtual machine from one host to another, affinity might no longer apply because the new host might have a different number of processors. n The NUMA scheduler might not be able to manage a virtual machine that is already assigned to certain processors using affinity. n Affinity can affect an ESX/ESXi host's ability to schedule virtual machines on multicore or hyperthreaded processors to take full advantage of resources shared on such processors.
Managing Memory Resources 3 All modern operating systems provide support for virtual memory, allowing software to use more memory than the machine physically has. Similarly, the ESX/ESXi hypervisor provides support for overcommitting virtual machine memory, where the amount of guest memory configured for all virtual machines might be larger than the amount of physical host memory. If you intend to use memory virtualization, you should understand how ESX/ESXi hosts allocate, tax, and reclaim memory.
vSphere Resource Management Guide For example, consider a virtual machine with a configured size of 1GB. When the guest operating system boots, it detects that it is running on a dedicated machine with 1GB of physical memory. The actual amount of physical host memory allocated to the virtual machine depends on its memory resource settings and memory contention on the ESX/ESXi host. In some cases, the virtual machine might be allocated the full 1GB. In other cases, it might receive a smaller allocation.
Chapter 3 Managing Memory Resources The amount of memory saved by memory sharing depends on workload characteristics. A workload of many nearly identical virtual machines might free up more than thirty percent of memory, while a more diverse workload might result in savings of less than five percent of memory. Software-Based Memory Virtualization ESX/ESXi virtualizes guest physical memory by adding an extra level of address translation.
vSphere Resource Management Guide Figure 3-1. ESX/ESXi Memory Mapping virtual machine 1 virtual machine 2 a b c b guest virtual memory a b b c guest physical memory c machine memory a b b n The boxes represent pages, and the arrows show the different memory mappings. n The arrows from guest virtual memory to guest physical memory show the mapping maintained by the page tables in the guest operating system.
Chapter 3 Managing Memory Resources View Memory Allocation Information You can use the vSphere Client to view information about current memory allocations. You can view the information about the total memory and memory available to virtual machines. In ESX, you can also view memory assigned to the service console. Procedure 1 In the vSphere Client, select a host and click the Configuration tab. 2 Click Memory. You can view the information shown in “Host Memory Information,” on page 29.
vSphere Resource Management Guide ESX/ESXi memory virtualization adds little time overhead to memory accesses. Because the processor's paging hardware uses page tables (shadow page tables for software-based approach or nested page tables for hardware-assisted approach) directly, most memory accesses in the virtual machine can execute without address translation overhead. The memory space overhead has two components. n A fixed, system-wide overhead for the VMkernel and (for ESX only) the service console.
Chapter 3 Managing Memory Resources An ESX/ESXi host determines allocations for each virtual machine based on the number of shares allocated to it and an estimate of its recent working set size. n Shares — ESX/ESXi hosts use a modified proportional-share memory allocation policy. Memory shares entitle a virtual machine to a fraction of available physical memory.
vSphere Resource Management Guide Figure 3-2. Memory Ballooning in the Guest Operating System 1 memory 2 swap space memory 3 memory swap space NOTE You must configure the guest operating system with sufficient swap space. Some guest operating systems have additional limitations. If necessary, you can limit the amount of memory vmmemctl reclaims by setting the sched.mem.maxmemctl parameter for a specific virtual machine.
Chapter 3 Managing Memory Resources Swap File Location By default, the swap file is created in the same location as the virtual machine's configuration file. A swap file is created by the ESX/ESXi host when a virtual machine is powered on. If this file cannot be created, the virtual machine cannot power on. Instead of accepting the default, you can also: n Use per-virtual machine configuration options to change the datastore to another shared storage location.
vSphere Resource Management Guide If you are overcommitting memory with ESX/ESXi, to support the intra-guest swapping induced by ballooning, ensure that your guest operating systems also have sufficient swap space. This guest-level swap space must be greater than or equal to the difference between the virtual machine’s configured memory size and its Reservation.
Chapter 3 Managing Memory Resources To determine the effectiveness of memory sharing for a given workload, try running the workload, and use resxtop or esxtop to observe the actual savings. Find the information in the PSHARE field of the interactive mode in the Memory page. Measuring and Differentiating Types of Memory Usage The Performance tab of the vSphere Client displays a number of metrics that can be used to analyze memory usage.
vSphere Resource Management Guide A similar result is obtained when determining Memory Shared and Memory Shared Common for the ESX/ESXi host. n Memory Shared for the host is the sum of each virtual machine's Memory Shared. Calculate this by looking at each virtual machine's guest physical memory and counting the number of blocks that have arrows to machine memory blocks that themselves have more than one arrow pointing at them.
Managing Resource Pools 4 A resource pool is a logical abstraction for flexible management of resources. Resource pools can be grouped into hierarchies and used to hierarchically partition available CPU and memory resources. Each standalone host and each DRS cluster has an (invisible) root resource pool that groups the resources of that host or cluster. The root resource pool is not displayed because the resources of the host (or cluster) and the root resource pool are always the same.
vSphere Resource Management Guide Why Use Resource Pools? Resource pools allow you to delegate control over resources of a host (or a cluster), but the benefits are evident when you use resource pools to compartmentalize all resources in a cluster. Create multiple resource pools as direct children of the host or cluster and configure them. You can then delegate control over the resource pools to other individuals or organizations. Using resource pools can result in the following benefits.
Chapter 4 Managing Resource Pools Create Resource Pools You can create a child resource pool of any ESX/ESXi host, resource pool, or DRS cluster. NOTE If a host has been added to a cluster, you cannot create child resource pools of that host. You can create child resource pools of the cluster if the cluster is enabled for DRS. When you create a child resource pool, you are prompted for resource pool attribute information.
vSphere Resource Management Guide Resource Pool Creation Example This procedure example demonstrates how you can create a resource pool with the ESX/ESXi host as the parent resource. Assume that you have an ESX/ESXi host that provides 6GHz of CPU and 3GB of memory that must be shared between your marketing and QA departments. You also want to share the resources unevenly, giving one department (QA) a higher priority.
Chapter 4 Managing Resource Pools n The information displayed in the Resource Allocation tab about the resource pool’s reserved and unreserved CPU and memory resources changes to reflect the reservations associated with the virtual machine (if any). NOTE If a virtual machine has been powered off or suspended, it can be moved but overall available resources (such as reserved and unreserved CPU and memory) for the resource pool are not affected.
vSphere Resource Management Guide Table 4-2. Reservation Types Reservation Type Description Fixed The system checks whether the selected resource pool has sufficient unreserved resources. If it does, the action can be performed. If it does not, a message appears and the action cannot be performed. Expandable (default) The system considers the resources available in the selected resource pool and its direct parent resource pool.
Chapter 4 Managing Resource Pools Figure 4-3. Admission Control with Expandable Resource Pools: Successful Power-On 6GHz RP-MOM VM-M1, 1GHz 2GHz VM-K1, 2GHz RP-KID VM-K2, 2GHz Now, consider another scenario with VM-M1 and VM-M2 (shown in Figure 4-4): n Power on two virtual machines in RP-MOM with a total reservation of 3GHz. n You can still power on VM-K1 in RP-KID because 2GHz are available locally. n When you try to power on VM-K2, RP-KID has no unreserved CPU capacity so it checks its parent.
vSphere Resource Management Guide 44 VMware, Inc.
Creating a DRS Cluster 5 A DRS cluster is a collection of ESX/ESXi hosts and associated virtual machines with shared resources and a shared management interface. Before you can obtain the benefits of cluster-level resource management you must create a DRS cluster. When you add a host to a DRS cluster, the host’s resources become part of the cluster’s resources.
vSphere Resource Management Guide Admission Control and Initial Placement When you attempt to power on a single virtual machine or a group of virtual machines in a DRS-enabled cluster, vCenter Server performs admission control. It checks that there are enough resources in the cluster to support the virtual machines. If the cluster does not have sufficient resources to power on a single virtual machine, or any of the virtual machines in a group, a message appears.
Chapter 5 Creating a DRS Cluster Group Power-On Example The user selects three virtual machines in the same datacenter for a group power-on attempt. The first two virtual machines (VM1 and VM2) are in the same DRS cluster (Cluster1), while the third virtual machine (VM3) is on a standalone host. VM1 is in automatic mode and VM2 is in manual mode.
vSphere Resource Management Guide n If the cluster and virtual machines involved are all fully automated, vCenter Server migrates running virtual machines between hosts as needed to ensure efficient use of cluster resources. NOTE Even in an automatic migration setup, users can explicitly migrate individual virtual machines, but vCenter Server might move those virtual machines to other hosts to optimize cluster resources. By default, automation level is specified for the whole cluster.
Chapter 5 Creating a DRS Cluster DRS Cluster Prerequisites Any host that is added to a DRS cluster must meet certain prerequisites to use cluster features successfully. Shared Storage Ensure that the managed hosts use shared storage. Shared storage is typically on a storage area network (SAN) but can also be implemented using NAS shared storage.
vSphere Resource Management Guide Configure EVC from the Cluster Settings dialog box. The hosts in a cluster must meet certain requirements for the cluster to use EVC. For more information on EVC and EVC requirements, see Basic System Administration. n CPU compatibility masks – vCenter Server compares the CPU features available to a virtual machine with the CPU features of the destination host to determine whether to allow or disallow migrations with VMotion.
Chapter 5 Creating a DRS Cluster 12 Select a location for the swapfiles of your virtual machines. You can either store a swapfile in the same directory as the virtual machine itself or a datastore specified by the host (host-local swap). 13 Click Next. 14 Review the summary page that lists the options you selected. 15 Click Finish to complete cluster creation, or click Back to go back and make modifications to the cluster setup. A new cluster does not include any hosts or virtual machines.
vSphere Resource Management Guide 52 VMware, Inc.
Using DRS Clusters to Manage Resources 6 After you create a DRS cluster, you can customize it and use it to manage resources. To customize your DRS cluster and the resources it contains you can configure DRS rules and you can add and remove hosts and virtual machines. When a cluster’s settings and resources have been defined, you should ensure that it is and remains a valid cluster. You can also use a valid DRS cluster to manage power resources and interoperate with VMware HA.
vSphere Resource Management Guide 2 Right-Click and select Edit Settings. 3 In the left panel under VMware DRS select Rules. 4 Click Add. 5 In the Virtual Machine Rule dialog box, name the rule. 6 Select one of the options from the pop-up menu: n Keep Virtual Machines Together One virtual machine cannot be part of more than one such rule. n Separate Virtual Machines This type of rule cannot contain more than two virtual machines. 7 Click Add and click OK. The rule is created.
Chapter 6 Using DRS Clusters to Manage Resources 4 Select the rule you want to remove and click Remove. The rule is deleted. Adding Hosts to a Cluster The procedure for adding hosts to a cluster is different for hosts managed by the same vCenter Server (managed hosts) than for hosts not managed by that server. After a host has been added, the virtual machines deployed to the host become part of the cluster and DRS can recommend migration of some virtual machines to other hosts in the cluster.
vSphere Resource Management Guide vCenter Server removes all existing resource pools of the host and the virtual machines in the host’s hierarchy are all attached to the root. Because share allocations are relative to a resource pool, you might have to manually change a virtual machine’s shares after selecting this option, which destroys the resource pool hierarchy.
Chapter 6 Using DRS Clusters to Manage Resources Procedure 1 Select the host and select Enter Maintenance Mode from the right-click menu. 2 After the host is in maintenance mode, drag it to a different inventory location, either the top-level datacenter or a different cluster. When you move the host, its resources are removed from the cluster. If you grafted the host’s resource pool hierarchy onto the cluster, that hierarchy remains with the cluster.
vSphere Resource Management Guide DRS Cluster Validity The vSphere Client indicates whether a DRS cluster is valid, overcommitted (yellow), or invalid (red). DRS clusters become overcommitted or invalid for several reasons. n A cluster might become overcommitted if a host fails. n A cluster becomes invalid if vCenter Server is unavailable and you power on virtual machines using a vSphere Client connected directly to an ESX/ESXi host.
Chapter 6 Using DRS Clusters to Manage Resources Figure 6-1. Valid Cluster with Fixed Resource Pools cluster Total Capacity: 12G Reserved Capacity: 11G Available Capacity: 1G RP1 Reservation: 4G Reservation Used: 4G Unreserved: 0G VM1, 2G VM6, 2G RP2 Reservation: 4G Reservation Used: 3G Unreserved: 1G VM2, 2G VM7, 2G RP3 Reservation: 3G Reservation Used: 3G Unreserved: 0G VM3, 3G VM4, 1G VM5, 2G VM8, 2G The cluster has the following characteristics: n A cluster with total resources of 12GHz.
vSphere Resource Management Guide Figure 6-2.
Chapter 6 Using DRS Clusters to Manage Resources Figure 6-3.
vSphere Resource Management Guide A cluster can also turn red if you reconfigure a resource pool while a virtual machine is failing over. A virtual machine that is failing over is disconnected and does not count toward the reservation used by the parent resource pool. You might reduce the reservation of the parent resource pool before the failover completes. After the failover is complete, the virtual machine resources are again charged to the parent resource pool.
Chapter 6 Using DRS Clusters to Manage Resources VMware DPM can use one of three power management protocols to bring a host out of standby mode: Intelligent Platform Management Interface (IPMI), Hewlett-Packard Integrated Lights-Out (iLO), or Wake-OnLAN (WOL). Each protocol requires its own hardware support and configuration. If a host does not support any of these protocols it cannot be put into standby mode by VMware DPM.
vSphere Resource Management Guide Test Wake-on-LAN for VMware DPM The use of Wake-on-LAN (WOL) for the VMware DPM feature is fully supported, if you configure and successfully test it according to the VMware guidelines. You must perform these steps before enabling VMware DPM for a cluster for the first time or on any host that is being added to a cluster that is using VMware DPM. Prerequisites Before testing WOL, ensure that your cluster meets the prerequisites.
Chapter 6 Using DRS Clusters to Manage Resources Automation Level Whether the host power state and migration recommendations generated by VMware DPM are executed automatically or not depends upon the power management automation level selected for the feature. The automation level is configured under Power Management in the cluster’s Settings dialog box. The options available are: n Off – The feature is disabled and no recommendations will be made.
vSphere Resource Management Guide After enabling and running VMware DPM, you can verify that it is functioning properly by viewing each host’s Last Time Exited Standby information displayed on the Host Options page in the cluster Settings dialog box and on the Hosts tab for each cluster. This field shows a timestamp and whether vCenter Server Succeeded or Failed the last time it attempted to bring the host out of standby mode. If no such attempt has been made, the field displays Never.
Viewing DRS Cluster Information 7 You can view information about a DRS cluster using the cluster Summary and DRS tabs in the vSphere Client. You can also apply the DRS recommendations that appear in the DRS tab. This chapter includes the following topics: n “Viewing the Cluster Summary Tab,” on page 67 n “Using the DRS Tab,” on page 69 Viewing the Cluster Summary Tab You can access a cluster’s Summary tab from the inventory panel of the vSphere Client.
vSphere Resource Management Guide Cluster Summary Tab VMware DRS Section The VMware DRS section appears in the cluster's Summary tab only if VMware DRS is enabled. Table 7-2. VMware DRS Section Field Description Migration Automation Level Manual, Partially Automated, Fully Automated. Power Management Automation Level Off, Manual, Automatic. DRS Recommendations Number of DRS migration recommendations awaiting user confirmation.
Chapter 7 Viewing DRS Cluster Information Using the DRS Tab The DRS tab is available when you select a DRS cluster object from the inventory panel in the vSphere Client. This tab displays information about the DRS recommendations made for the cluster, faults that have occurred in applying such recommendations, and the history of DRS actions. You can access three pages from this tab. These pages are named Recommendations, Faults, and History.
vSphere Resource Management Guide Table 7-4. DRS Recommendations Information Column Description Priority Priority level (1-5) for the recommendation. Priority one, the highest, indicates a mandatory move because of a host entering maintenance or standby mode or DRS rule violations. Other priority ratings denote how much the recommendation would improve the cluster’s performance; from priority two (significant improvement) to priority five (slight). Prior to ESX/ESXi 4.
Chapter 7 Viewing DRS Cluster Information DRS History Page The History page of the DRS tab displays recent actions taken as a result of DRS recommendations. You can reach this page by clicking the History button on the DRS tab. For each action, DRS provides the information shown in Table 7-6. Table 7-6. DRS History Page Field Description DRS Actions Details of the action taken. Time Timestamp of when the action occurred.
vSphere Resource Management Guide 72 VMware, Inc.
Using NUMA Systems with ESX/ESXi 8 ESX/ESXi supports memory access optimization for Intel and AMD Opteron processors in server architectures that support NUMA (non-uniform memory access). After you understand how ESX/ESXi NUMA scheduling is performed and how the VMware NUMA algorithms work, you can specify NUMA controls to optimize the performance of your virtual machines.
vSphere Resource Management Guide There are many disadvantages to using such an operating system on a NUMA platform. The high latency of remote memory accesses can leave the processors under-utilized, constantly waiting for data to be transferred to the local node, and the NUMA connection can become a bottleneck for applications with high-memory bandwidth demands. Furthermore, performance on such a system can be highly variable.
Chapter 8 Using NUMA Systems with ESX/ESXi VMware NUMA Optimization Algorithms and Settings This section describes the algorithms and settings used by ESX/ESXi to maximize application performance while still maintaining resource guarantees. Home Nodes and Initial Placement When a virtual machine is powered on, ESX/ESXi assigns it a home node. A virtual machine runs only on processors within its home node, and its newly allocated memory comes from the home node as well.
vSphere Resource Management Guide When a virtual machine moves to a new node, the ESX/ESXi host immediately begins to migrate its memory in this fashion. It manages the rate to avoid overtaxing the system, particularly when the virtual machine has little remote memory remaining or when the destination node has little free memory available. The memory migration algorithm also ensures that the ESX/ESXi host does not move memory needlessly if a virtual machine is moved to a new node for only a short period.
Chapter 8 Using NUMA Systems with ESX/ESXi The IBM Enterprise X-Architecture supports servers with up to four nodes (also called CECs or SMP Expansion Complexes in IBM terminology). Each node can contain up to four Intel Xeon MP processors for a total of 16 CPUs. The next generation IBM eServer x445 uses an enhanced version of the Enterprise X-Architecture, and scales to eight nodes with up to four Xeon MP processors for a total of 32 CPUs.
vSphere Resource Management Guide An administrator can also manually move a virtual machine to another node after the virtual machine has started running. In this case, the page migration rate of the virtual machine must be set manually, so that memory from the virtual machine’s previous node can be moved to its new node. Manual NUMA placement might interfere with the ESX/ESXi resource management algorithms, which try to give each virtual machine a fair share of the system’s processor resources.
Chapter 8 Using NUMA Systems with ESX/ESXi Then, you want this virtual machine to run only on node 1. 1 In the vSphere Client inventory panel, select the virtual machine and select Edit Settings. 2 Select Options and click Advanced. 3 Click the Configuration Parameters button. 4 In the vSphere Client, set memory affinity for the NUMA node to 1. Completing these two tasks ensures that the virtual machine runs only on NUMA node 1 and, when possible, allocates memory from the same node. VMware, Inc.
vSphere Resource Management Guide 80 VMware, Inc.
A Performance Monitoring Utilities: resxtop and esxtop The resxtop and esxtop command-line utilities provide a detailed look at how ESX/ESXi uses resources in real time. You can start either utility in one of three modes: interactive (default), batch, or replay. The fundamental difference between resxtop and esxtop is that you can use resxtop remotely (or locally), whereas esxtop can be started only through the service console of a local ESX host.
vSphere Resource Management Guide After it is set up, start resxtop from the command line. For remote connections, you can either connect to an ESX/ESXi host either directly or through vCenter Server. The command-line options are the same as for esxtop (except for the R option) with additional connection options. NOTE resxtop does not use all the options shared by other vSphere CLI commands. Table A-1.
Appendix A Performance Monitoring Utilities: resxtop and esxtop Table A-2. Interactive Mode Command-Line Options (Continued) Option Description a Show all statistics. This option overrides configuration file setups and shows all statistics. The configuration file can be the default ~/.esxtop4rc configuration file or a user-defined configuration file. c Load a user-defined configuration file. If the -c option is not used, the default configuration filename is ~/.esxtop4rc.
vSphere Resource Management Guide Table A-3. Interactive Mode Single-Key Commands (Continued) Key Description # Prompts you for the number of statistics rows to display. Any value greater than 0 overrides automatic determination of the number of rows to show, which is based on window size measurement. If you change this number in one resxtop (or esxtop) panel, the change affects all four panels. s Prompts you for the delay between updates, in seconds.
Appendix A Performance Monitoring Utilities: resxtop and esxtop Table A-4. CPU Panel Statistics (Continued) Line Description CCPU(%) Percentages of total CPU time as reported by the ESX service console. This field is not displayed if you are using ESXi. n us — Percentage user time. n sy — Percentage system time. n id — Percentage idle time. n wa — Percentage wait time. n cs/sec — Context switches per second recorded by the service console.
vSphere Resource Management Guide Table A-4. CPU Panel Statistics (Continued) Line Description AFFINITY BIT MASK Bit mask showing the current scheduling affinity for the world. HTSHARING Current hyperthreading configuration. CPU The physical or logical processor on which the world was running when resxtop (or esxtop) obtained this information. HTQ Indicates whether the world is currently quarantined or not. N means no and Y means yes. TIMER/s Timer rate for this world.
Appendix A Performance Monitoring Utilities: resxtop and esxtop Table A-6. Memory Panel Statistics Field Description PMEM (MB) Displays the machine memory statistics for the server. All numbers are in megabytes. VMKMEM (MB) COSMEM (MB) total Total amount of machine memory in the server. cos Amount of machine memory allocated to the ESX service console. vmk Amount of machine memory being used by the ESX/ESXi VMkernel.
vSphere Resource Management Guide Table A-6. Memory Panel Statistics (Continued) Field Description SWAP (MB) Displays the ESX/ESXi swap usage statistics. All numbers are in megabytes. MEMCTL (MB) 88 curr Current swap usage. target Where the ESX/ESXi system expects the swap usage to be. r/s Rate at which memory is swapped in by the ESX/ESXi system from disk. w/s Rate at which memory is swapped to disk by the ESX/ESXi system. Displays the memory balloon statistics. All numbers are in megabytes.
Appendix A Performance Monitoring Utilities: resxtop and esxtop Table A-6. Memory Panel Statistics (Continued) Field Description SWTGT (MB) Target where the ESX/ESXi host expects the swap usage by the resource pool or virtual machine to be. SWR/s (MB) Rate at which the ESX/ESXi host swaps in memory from disk for the resource pool or virtual machine. SWW/s (MB) Rate at which the ESX/ESXi host swaps resource pool or virtual machine memory to disk. CPTRD (MB) Amount of data read from checkpoint file.
vSphere Resource Management Guide Table A-8. Storage Adapter Panel Statistics (Continued) Column Description NCHNS Number of channels. NTGTS Number of targets. NLUNS Number of LUNs. NWDS Number of worlds. BLKSZ Block size in bytes. This statistic is applicable only to LUNs. AQLEN Storage adapter queue depth. Maximum number of ESX/ESXi VMkernel active commands that the adapter driver is configured to support. LQLEN LUN queue depth.
Appendix A Performance Monitoring Utilities: resxtop and esxtop Table A-9. Storage Adapter Panel Interactive Commands Command Description e Toggles whether storage adapter statistics are displayed expanded or unexpanded. Allows viewing storage resource utilization statistics broken down by individual channels belonging to an expanded storage adapter. You are prompted for the adapter name. P Toggles whether storage adapter statistics are displayed expanded or unexpanded.
vSphere Resource Management Guide Table A-10. Storage Device Panel Statistics (Continued) Column Description WQLEN World queue depth. This is the maximum number of ESX/ESXi VMkernel active commands that the world is allowed to have. This is a per device maximum for the world. It is valid only if the corresponding device is expanded to worlds. ACTV Number of commands in the ESX/ESXi VMkernel that are currently active. This statistic is applicable only to worlds and devices.
Appendix A Performance Monitoring Utilities: resxtop and esxtop Table A-11. Storage Device Panel Interactive Commands Command Description e Expand or roll up storage world statistics. This command allows you to view storage resource utilization statistics separated by individual worlds belonging to an expanded storage device. You are prompted for the device name. The statistics are per world per device. This is not applicable to NFS. p Expand or roll up storage path statistics.
vSphere Resource Management Guide Table A-12. Virtual Machine Storage Panel Statistics (Continued) Column Description BLKSZ Block size in bytes. It is valid only if the corresponding world is expanded to devices. NUMBLKS Number of blocks of the device. It is valid only if the corresponding world is expanded to devices. DQLEN Storage device queue depth. This is the maximum number of ESX/ESXi VMkernel active commands that the device is configured to support.
Appendix A Performance Monitoring Utilities: resxtop and esxtop Table A-13. Virtual Machine Storage Panel Interactive Commands Command Description e Expand or roll up storage world statistics. Allows you to view storage resource utilization statistics separated by individual worlds belonging to a group. You are prompted to enter the group ID. The statistics are per world. l Expand or roll up storage device, that is LUN, statistics.
vSphere Resource Management Guide Table A-15. Network Panel Interactive Commands Command Description T Sorts by Mb Tx column. R Sorts by Mb Rx column. t Sorts by Packets Tx column. r Sorts by Packets Rx column. N Sorts by PORT-ID column. This is the default sort order. L Changes the displayed length of the DNAME column. Interrupt Panel The interrupt panel displays information about the use of interrupt vectors. Table A-16.
Appendix A Performance Monitoring Utilities: resxtop and esxtop Use esxtop or resxtop in Batch Mode After you have prepared for batch mode, you can use esxtop or resxtop in this mode. Procedure 1 Start resxtop (or esxtop) to redirect the output to a file. For example: esxtop -b > my_file.csv The filename must have a .csv extension. The utility does not enforce this, but the post-processing tools require it. 2 Process statistics collected in batch mode using tools such as Microsoft Excel and Perfmon.
vSphere Resource Management Guide Prepare for Replay Mode To run in replay mode, you must prepare for replay mode. Procedure 1 Run vm-support in snapshot mode on the ESX service console. Use the following command. vm-support -S -d duration -I interval 2 Unzip and untar the resulting tar file so that esxtop can use it in replay mode. You can now use esxtop in replay mode. Use esxtop in Replay Mode You can use esxtop in replay mode. You do not have to run replay mode on the ESX service console.
Advanced Attributes B You can set advanced attributes for hosts or individual virtual machines to help you customize resource management. In most cases, adjusting the basic resource allocation settings (reservation, limit, shares) or accepting default settings results in appropriate resource allocation. However, you can use advanced attributes to customize resource management for a host or a specific virtual machine.
vSphere Resource Management Guide Advanced Memory Attributes You can use the advanced memory attributes to customize memory resource usage. Table B-2. Advanced Memory Attributes 100 Attribute Description Default Mem.CtlMaxPercent Limits the maximum amount of memory reclaimed from any virtual machine using vmmemctl, based on a percentage of its configured memory size. Specify 0 to disable reclamation using vmmemctl for all virtual machines. 65 Mem.
Appendix B Advanced Attributes Advanced NUMA Attributes You can use the advanced NUMA attributes to customize NUMA usage. Table B-3. Advanced NUMA Attributes Attribute Description Default Numa.RebalanceEnable Set this option to 0 to disable all NUMA rebalancing and initial placement of virtual machines, effectively disabling the NUMA scheduling system. 1 Numa.PageMigEnable If you set this option to 0, the system does not automatically migrate pages between nodes to improve memory locality.
vSphere Resource Management Guide 3 Click the Configuration Parameters button. 4 In the dialog box that appears, click Add Row to enter a new parameter and its value. Advanced Virtual Machine Attributes You can use the advanced virtual machine attributes to customize virtual machine configuration. Table B-4. Advanced Virtual Machine Attributes 102 Attribute Description sched.mem.maxmemctl Maximum amount of memory reclaimed from the selected virtual machine by ballooning, in megabytes (MB).
Index A D admission control CPU 23 resource pools 41 with expandable resource pools 42 advanced attributes CPU 99 hosts 99 memory 100 NUMA 101 virtual machines 101 alarms 66 AMD Opteron-based systems 49, 73, 76, 101 applications CPU-bound 18 single-threaded 18 automation modes, virtual machines 51 DPM and admission control 16 automation level 65 enabling 64 Last Time Exited Standby 65 monitoring 66 overrides 65 threshold 65 DRS disabling 51 fully automated 50 group power on 46 information 68 initial pla
vSphere Resource Management Guide DRS rules creating 53 deleting 54 disabling 54 editing 54 DRS tab Faults page 70 History page 71 Recommendations page 69 using 69 DRS Troubleshooting Guide 70 dual-processor virtual machine 17 dynamic load balancing, NUMA 75 Dynamic Voltage and Frequency Scaling (DVFS) 24 E educational support 7 Enhanced VMotion Compatibility (EVC) 18, 49, 50, 67 entering maintenance mode 57 ESX/ESXi memory allocation 30 memory reclamation 31 esxtop batch mode 97 common statistics descrip
Index Mem.
vSphere Resource Management Guide interactive mode 82 interactive mode command-line options 82 interactive mode single-key commands 83 interrupt panel 96 memory panel 86 network panel 95 options 81 order pages 83 performance monitoring 81 statistics column 83 storage adapter panel 89 storage device panel 91 virtual machine storage panel 93 root resource pool 37 S sched.mem.maxmemctl 31, 102 sched.mem.pshare.enable 102 sched.swap.dir 102 sched.swap.
