6.0.1

ESXi 5.0 and later includes support for exposing virtual NUMA topology to guest operating systems. For
more information about virtual NUMA control, see “Using Virtual NUMA,” on page 110.
VMware NUMA Optimization Algorithms and Settings
This section describes the algorithms and settings used by ESXi to maximize application performance while
still maintaining resource guarantees.
Home Nodes and Initial Placement
When a virtual machine is powered on, ESXi assigns it a home node. A virtual machine runs only on
processors within its home node, and its newly allocated memory comes from the home node as well.
Unless a virtual machine’s home node changes, it uses only local memory, avoiding the performance
penalties associated with remote memory accesses to other NUMA nodes.
When a virtual machine is powered on, it is assigned an initial home node so that the overall CPU and
memory load among NUMA nodes remains balanced. Because internode latencies in a large NUMA system
can vary greatly, ESXi determines these internode latencies at boot time and uses this information when
initially placing virtual machines that are wider than a single NUMA node. These wide virtual machines are
placed on NUMA nodes that are close to each other for lowest memory access latencies.
Initial placement-only approaches are usually sufficient for systems that run only a single workload, such as
a benchmarking configuration that remains unchanged as long as the system is running. However, this
approach is unable to guarantee good performance and fairness for a datacenter-class system that supports
changing workloads. Therefore, in addition to initial placement, ESXi 5.0 does dynamic migration of virtual
CPUs and memory between NUMA nodes for improving CPU balance and increasing memory locality.
Dynamic Load Balancing and Page Migration
ESXi combines the traditional initial placement approach with a dynamic rebalancing algorithm.
Periodically (every two seconds by default), the system examines the loads of the various nodes and
determines if it should rebalance the load by moving a virtual machine from one node to another.
This calculation takes into account the resource settings for virtual machines and resource pools to improve
performance without violating fairness or resource entitlements.
The rebalancer selects an appropriate virtual machine and changes its home node to the least loaded node.
When it can, the rebalancer moves a virtual machine that already has some memory located on the
destination node. From that point on (unless it is moved again), the virtual machine allocates memory on its
new home node and it runs only on processors within the new home node.
Rebalancing is an effective solution to maintain fairness and ensure that all nodes are fully used. The
rebalancer might need to move a virtual machine to a node on which it has allocated little or no memory. In
this case, the virtual machine incurs a performance penalty associated with a large number of remote
memory accesses. ESXi can eliminate this penalty by transparently migrating memory from the virtual
machine’s original node to its new home node:
1 The system selects a page (4KB of contiguous memory) on the original node and copies its data to a
page in the destination node.
2 The system uses the virtual machine monitor layer and the processor’s memory management hardware
to seamlessly remap the virtual machine’s view of memory, so that it uses the page on the destination
node for all further references, eliminating the penalty of remote memory access.
When a virtual machine moves to a new node, the ESXi host immediately begins to migrate its memory in
this fashion. It manages the rate to avoid overtaxing the system, particularly when the virtual machine has
little remote memory remaining or when the destination node has little free memory available. The memory
migration algorithm also ensures that the ESXi host does not move memory needlessly if a virtual machine
is moved to a new node for only a short period.
Chapter 14 Using NUMA Systems with ESXi
VMware, Inc. 109