Locality-Optimized Resource Alignment

ManualsBrandsHP ManualsSoftwareHP-UX 11i v3 Enterprise OE

Table of Contents

Executive summary ............................................................................................................................... 3

Background and motivation .................................................................................................................. 3

Structure of HP servers ...................................................................................................................... 3

Interleaved memory .......................................................................................................................... 3

Local memory .................................................................................................................................. 4

LORA scope ........................................................................................................................................ 4

Hardware platforms ......................................................................................................................... 4

Operating system ............................................................................................................................. 4

Virtual partitioning ........................................................................................................................... 4

Application workload ....................................................................................................................... 5

Variability in hardware resources ....................................................................................................... 5

When to use LORA........................................................................................................................... 5

LORA configuration rules ...................................................................................................................... 5

nPartitions ....................................................................................................................................... 6

vPars .............................................................................................................................................. 6

Integrity Virtual Machines .................................................................................................................. 6

LORA system administration .................................................................................................................. 6

Server Tunables product in the Tune-N-Tools bundle ............................................................................. 6

loratune command ........................................................................................................................... 6

Benefits .............................................................................................................................................. 7

Performance .................................................................................................................................... 7

Cost ............................................................................................................................................... 7

Power management .......................................................................................................................... 7

Summary ............................................................................................................................................ 7

Glossary ............................................................................................................................................. 8

Technical details .................................................................................................................................. 8

Configuring nPartitions for LORA ........................................................................................................... 8

Converting an existing nPartition for LORA .......................................................................................... 8

Creating a new nPartition for LORA ................................................................................................... 9

Determine the size of the nPartition ................................................................................................. 9

Create the nPartition ..................................................................................................................... 9

Configuring vPars for LORA ................................................................................................................ 10

Considerations for memory granule size ........................................................................................... 10

Creating new vPars instances .......................................................................................................... 10

Locality-Optimized Resource Alignment

Version 3.5 for HP-UX 11i v3 Update 5

Summary of content (20 pages)

PAGE 1
Locality-Optimized Resource Alignment Version 3.5 for HP-UX 11i v3 Update 5 Table of Contents Executive summary............................................................................................................................... 3 Background and motivation .................................................................................................................. 3 Structure of HP servers .......................................................................................................
PAGE 2
Dividing the nPartition into vPars instances of equal size ..................................................................... 11 Establishing vPars instances by processor and memory requirements .................................................... 14 Modifying existing vPars instances ................................................................................................... 16 Summary of vPars configuration rules ...................................................................................
PAGE 3
Executive summary Locality-Optimized Resource Alignment (hereinafter LORA) is a framework for increasing system performance by exploiting the locality domains in HP servers with Non-Uniform Memory Architecture. LORA consists of configuration rules and tuning recommendations, plus HP-UX 11i v3 enhancements and tools to support them. LORA introduces a new mode to supplement the Symmetric Multiprocessing (SMP) mode originally implemented in HP-UX.
PAGE 4
Sometimes interleaved memory is the best technique. ILM yields good performance when memory references are spread across the entire address space with equal probability. This is the case for applications using large global data sets with no spatial locality. Local memory If memory is not interleaved, then the natural localities inherent in the structure of the server complex are evident. The processor cores in each locality enjoy fast access to their local memory.
PAGE 5
Application workload Memory reference patterns are key to system performance on NUMA platforms. Some applications exhibit strong locality of memory reference, others scatter accesses to global data, and some fall in between. Most commercial applications exhibit sufficient reference locality to gain significant benefit by running with LORA. Technical applications that access extremely large data sets in a uniform manner are better suited to interleaved memory and SMP mode.
PAGE 6
HP-UX 11i supports a variety of partitioning models. The sections that follow explain which of these models are sensitive to the LORA configuration rules. nPartitions At the nPartition level, each base cell should be configured with ⅞ths local memory to comply with the LORA configuration rules. The floating cells, if any, always have 100% local memory. The appendix Configuring nPartitions for LORA contains more detail.
PAGE 7
• terminating a major application or completing a backup • dynamic platform operation such as online cell activation The simplest way to use the command is to invoke it with no arguments: loratune More details are available in the man page. Benefits Performance LORA reduces average memory access latency times in comparison to the interleaved memory mode. The magnitude of the reduction depends on the memory reference pattern and the number of localities in the partition.
PAGE 8
Glossary Term Definition Socket Receptacle on a motherboard for the physical package of processing resources. Processor core If the physical package of processing resources includes multiple independent functional entities, each of them is called a processor core. Core Same as processor core. The keyword ‘cpu’ in the vPars commands refers to a core. CPU Acronym for Central Processing Unit. The term processor core is preferred. Cell The basic physical building block of a system complex.
PAGE 9
The parmodify command is used to change the local memory ratio. For example, for an nPartition built from cells 1, 2, and 3: parmodify -p 2 -m 1:base:::87.5% -m 2:base:::87.5% -m 3:base:::87.5% It is necessary to reboot the partition for the change to take effect. The man page for the parmodify(1m) command has the full details.
PAGE 10
Configuring vPars for LORA Considerations for memory granule size Memory for vPars instances is allocated only in multiples of the memory granule size. The granule size can be different for interleaved memory and local memory. Default granule sizes are established when a server complex is shipped from the factory. The defaults can be changed by use of the vparenv command, or by adding options to the command that creates the very first vPars instance.
PAGE 11
Figure 1. Resources available in the example nPartition Dividing the nPartition into vPars instances of equal size Suppose that it is desired to create six vPars instances of equal size in this partition. Since the partition contains 24 cores, each vPars instance should have 4 cores. Since the partition contains 24 GB of interleaved memory and 168 GB of local memory, each vPars instance should have 4 GB of interleaved memory and 28 GB of local memory.
PAGE 12
Figure 2. Six equal vPars instances configured according to LORA rules To achieve alignment between processors and memory with the vPars version delivered in Update 3, it was necessary to specify the cell identifiers for the processors to match the local memory. This had the drawback of disallowing gWLM to perform processor deletion operations upon the cell local processors.
PAGE 13
This first example was extremely simple: the resources on each cell were divided in half. But it shows quite clearly the power behind the LORA concept: the alignment between processor cores and local memory guarantees that the preponderance of memory references will be satisfied through the fastest hardware path. A slightly more complicated example involves dividing the nPartition into four equal vPars instances.
PAGE 14
Establishing vPars instances by processor and memory requirements Suppose it is stipulated to divide the example partition into vPars of the following sizes: Table 2. Specification for size of vPars instances Name Number of cores Amount of memory cow 12 48 GB dog 4 88 GB elk 2 16 GB fox 2 16 GB The first vPars instance in the table, cow, requires 12 cores, so it will not fit within a single locality.
PAGE 15
The diagram in Figure 4 shows the example rx8640 with processor and memory resources allocated to the first two vPars instances from Table 5: Figure 4.
PAGE 16
Figure 5. Allocations for the custom vPars instances CPU 0 CPU 1 CPU 2 CPU 3 CPU 0 CPU 1 CPU 2 CPU 3 CPU 0 CPU 1 CPU 2 CPU 3 CPU 4 CPU 5 CPU 6 CPU 7 CPU 4 CPU 5 CPU 6 CPU 7 CPU 4 CPU 5 CPU 6 CPU 7 elk cow dog fox Modifying existing vPars instances The vparmodify command can be used to change the composition of existing vPars instances. When the number of processor cores allocated is changed, this is referred to as "CPU migration".
PAGE 17
Figure 6. Allocations after the CPU migration operations CPU 0 CPU 1 CPU 2 CPU 3 CPU 0 CPU 1 CPU 2 CPU 3 CPU 0 CPU 1 CPU 2 CPU 3 CPU 4 CPU 5 CPU 6 CPU 7 CPU 4 CPU 5 CPU 6 CPU 7 CPU 4 CPU 5 CPU 6 CPU 7 elk cow dog fox It is also possible to migrate memory along with the processor cores, but only for the memory that was specifically designated as floating memory.
PAGE 18
Advanced tuning An important part of the LORA value proposition is to deliver ease-of-use along with performance. Our goal is that LORA should work out-of-the-box, without the need for system administrators to perform explicit tuning. Several factors make the goal impossible to reach in every single case. The range of applications deployed across the HP-UX customer base is extremely diverse.
PAGE 19
Table 4.
PAGE 20
For more information For a discussion of SMP versus NUMA, see http://en.wikipedia.org/wiki/Symmetric_multiprocessing Detailed information about the capabilities and configuration of the Superdome platform is available at http://h18000.www1.hp.com/products/quickspecs/archives_Division/11717_div_v1/11717_div. HTML The definitive manual for nPartition administration is the nPartition Administrator's Guide at http://docs.hp.com/en/5991-1247B_ed2/index.