Locality-Optimized Resource Alignment for Superdome 2

ManualsBrandsHP ManualsSoftwareHP-UX 11i v3 Enterprise OE

Table Of Contents

Locality-Optimized Resource Alignment for

Superdome 2

Version 1.0

Technical white paper

Table of contents

Executive summary ............................................................................................................................... 2

Background and motivation of LORA ..................................................................................................... 2

Structure of Superdome 2 servers ....................................................................................................... 2

Interleaved memory .......................................................................................................................... 3

Local memory .................................................................................................................................. 3

Operating system mode .................................................................................................................... 4

Application workload ....................................................................................................................... 4

LORA configuration rules ...................................................................................................................... 4

Partitioning ...................................................................................................................................... 4

LORA system administration .................................................................................................................. 5

Server Tunables product in the Tune-N-Tools bundle ............................................................................. 5

loratune command ........................................................................................................................... 5

Benefits .............................................................................................................................................. 6

Performance .................................................................................................................................... 6

Cost ............................................................................................................................................... 6

Power management .......................................................................................................................... 6

Summary ............................................................................................................................................ 6

Glossary ............................................................................................................................................. 7

Technical details .................................................................................................................................. 7

Configuring nPartitions for LORA ........................................................................................................... 8

Creating a new nPartition for LORA ................................................................................................... 8

Converting an existing nPartition for LORA .......................................................................................... 8

Helpful hints .................................................................................................................................... 8

Configuring vPars for LORA .................................................................................................................. 9

Considerations for memory granule size ............................................................................................. 9

Creating new virtual partitions ........................................................................................................... 9

Dividing the nPartition into virtual partitions of equal size .................................................................... 10

Establishing virtual partitions by processor and memory requirements ................................................... 13

Helpful hints .................................................................................................................................. 15

Factory preconfiguration ................................................................................................................. 16

Advanced tuning ............................................................................................................................... 16

numa_mode kernel tunable parameter ............................................................................................. 16

numa_sched_launch kernel tunable parameter ............................................................................. 17

numa_policy kernel tunable parameter ......................................................................................... 17

mpsched command ........................................................................................................................ 17

For more information / Call to action .................................................................................................. 19

Summary of content (19 pages)

PAGE 1
Locality-Optimized Resource Alignment for Superdome 2 Version 1.0 Technical white paper Table of contents Executive summary............................................................................................................................... 2 Background and motivation of LORA ..................................................................................................... 2 Structure of Superdome 2 servers ........................................................................................
PAGE 2
Executive summary Locality-Optimized Resource Alignment (LORA) is a framework for increasing system performance by exploiting the locality domains in HP servers with Non-Uniform Memory Architecture, such as Superdome 2. LORA consists of configuration rules and tuning recommendations, plus HP-UX 11i v3 enhancements and tools to support them. LORA introduces a new mode to supplement the Symmetric Multiprocessing (SMP) mode originally implemented in HP-UX.
PAGE 3
Figure 1. Conceptual diagram of a Superdome 2 server Interleaved memory Interleaved memory (ILM) is a technique for masking the NUMA properties of a system. Successive cache lines in the memory address space are drawn from different localities, making the average memory access latency time more-or-less uniform. Whether memory is accessed sequentially or randomly, every locality in the system will be used with equal probability. Sometimes interleaved memory is the best technique.
PAGE 4
Operating system mode The first versions of HP-UX were designed for Uniform Memory Architecture platforms, and so treated each page of memory the same as every other page. This approach to memory management has been termed Symmetric Multiprocessor (SMP) mode. LORA introduced a new operating system mode that takes processor and memory locality into consideration for all process scheduling and memory management activities.
PAGE 5
vPars The virtualization model of vPars is particularly well-suited to LORA. The first version of the T1335DC product tailored for Superdome 2, version A.05.07, contains many optimizations to gain additional benefit from local memory. Each virtual partition should be configured with ⅞ths local memory and ⅛th ILM. Since the underlying nPartition has this memory ratio, it is straightforward to reflect the same ratio in the virtual partitions.
PAGE 6
The simplest way to use the command is to invoke it with no arguments: loratune More details are available in the man page. Benefits Performance LORA reduces average memory access latency times in comparison to the interleaved memory mode. The magnitude of the reduction depends on the memory reference pattern and the number of localities in the partition. When processors spend less time waiting for memory references to be satisfied, all aspects of application performance improve.
PAGE 7
Glossary Term Socket Definition A socket on Superdome 2 represents the field-replaceable processing unit on a blade. Each socket is associated with a set of memory DIMMs and local I/O resources. Processor core When a socket contains multiple independent processing entities, each of them is called a processor core. Core Same as processor core. The keyword ‘cpu’ in the vPars commands refers to a core. CPU Acronym for Central Processing Unit. The term ‘processor core’ is preferred.
PAGE 8
Configuring nPartitions for LORA This section discusses the case of creating a new LORA nPartition and of converting an existing 100% interleaved nPartition for use with LORA. This is relevant when virtual partitions will not be used, that is, the nPartition will be used to run a single instance of HP-UX or will act as the host for Integrity Virtual Machines. Memory interleaving is a property of the nPartition and the blades that compose it.
PAGE 9
fabric. Using a mixture of even-numbered and odd-numbered blades will still work, but some interblade communication will use slower paths through the interconnect fabric. Configuring vPars for LORA A nice thing about Superdome 2 is that all of the vPars configuration commands can be entered directly to the Onboard Administrator, even if none of the virtual partitions are booted. Of course, the commands are also available with the same syntax in any of the virtual partitions running in the nPartition.
PAGE 10
The LORA configuration procedure is a refinement of the general vPars configuration procedure described in the Superdome 2 Partitioning Administrator Guide. That reference should be consulted for details not discussed in this paper. We will illustrate the procedure with an nPartition consisting of blades 1, 3, and 5 in enclosure 1 of a Superdome 2 server with 64 GB of memory on each blade.
PAGE 11
vparcreate -N 2 -p vp1 -a socket:1/1/0:mem::28672 vparcreate -N 2 -p vp2 -a socket:1/1/1:mem::28672 vparcreate -N 2 -p vp3 -a socket:1/3/0:mem::28672 vparcreate -N 2 -p vp4 -a socket:1/3/1:mem::28672 vparcreate -N 2 -p vp5 -a socket:1/5/0:mem::28672 vparcreate -N 2 -p vp6 -a socket:1/5/1:mem::28672 socket:1/1/0:cpu::4 -a mem::4096 -a socket:1/1/1:cpu::4 -a mem::4096 -a socket:1/3/0:cpu::4 -a mem::4096 -a socket:1/3/1:cpu::4 -a mem::4096 -a socket:1/5/0:cpu::4 -a mem::4096 -a socket:1/5/1:cpu::4 -a mem::409
PAGE 12
is interleaved memory. The first three instances can be built easily, with each taking three quarters of the resources on a blade. The fourth instance must be built from the remaining resources and is therefore split across all three blades. The splintering of a virtual partition should be avoided when possible, but sometimes it is unavoidable. For this example, the processor and memory allocations would be as shown in the following diagram: Figure 4.
PAGE 13
vparcreate -N 2 -p q3 -a mem::6144 vparmodify -p 2:q3 -a socket:1/5/0:mem::21504 -a socket:1/5/1:mem::21504 vparmodify -p 2:q3 -a socket:1/5/0:cpu::3 -a socket:1/5/0:cpu::3 vparcreate vparmodify vparmodify vparmodify vparmodify vparmodify vparmodify -N -p -p -p -p -p -p 2 -p 2:q4 2:q4 2:q4 2:q4 2:q4 2:q4 q4 -a -a -a -a -a -a -a mem::6144 socket:1/1/0:mem::7168-a socket:1/1/1:mem::7168 socket:1/3/0:mem::7168-a socket:1/3/1:mem::7168 socket:1/5/0:mem::7168-a socket:1/5/1:mem::7168 socket:1/1/0:cpu::1 -a s
PAGE 14
as evenly as possible, while still respecting the memory granule size: 20 GB from one socket and 21 GB from two others. The second virtual partition in the table, dog, requires 70 GB of memory, so it also will not fit within one or even two localities. Once again, the strategy is to confine it within the minimum number of localities, which is three, and divide the resources as evenly as possible among the three localities.
PAGE 15
along with three processor cores. It is necessary to split each instance across three sockets, which is generally undesirable; but, that cannot be avoided in this particular case. The diagram in Figure 6 shows the result: Figure 6. The last two virtual partitions added to the LORA configuration If we had started at the bottom of the table, and configured the instances elk and fox first, each one could have been confined to one single socket, which would have given them superb performance.
PAGE 16
partitions of equal size. It would be less ideal to have 8 virtual partitions, because you would be forced to split some of them across two different sockets. A minor optimization, and one which also applies for 100% interleaved nPartitions, is to keep the cores on a socket in the same virtual partition, because they share a common cache. Keeping such cores working together on the same application can give a small performance benefit, which is worth realizing if the choice of cores is otherwise arbitrary.
PAGE 17
numa_sched_launch kernel tunable parameter The numa_sched_launch parameter controls the default process launch policy. The launch policy refers to the preferred locality for processes forked as children of a parent process. In LORA mode, the default launch policy is PACKED, which places child processes in the same locality as their parent. Setting the parameter to the value 0 forces the default launch policy to be the same as it is in SMP mode.
PAGE 18
Using mpsched to enforce good alignment for Java HP-UX is usually able to achieve good alignment when running Java. Here are some guidelines for explicit tunes to enforce good alignment when running multiple instances of the Java virtual machine. We will assume that each Java instance is small enough in terms of processor and memory resources to fit into a single locality. It is best to subdivide the Java instance if it exceeds the capacity of a single locality.
PAGE 19
For more information For a discussion of SMP versus NUMA, see http://en.wikipedia.org/wiki/Symmetric_multiprocessing Detailed information about the capabilities and configuration of the Superdome 2 platform is available at the following URLs: http://h18006.www1.hp.com/storage/pdfs/4AA1-7762ENW.pdf http://h18004.www1.hp.com/products/quickspecs/13682_div/13682_div.html Superdome 2 Partitioning Administrator Guide: http://bizsupport1.austin.hp.com/bc/docs/support/SupportManual/c02471028/c02471028.