Locality-Optimized Resource Alignment for Superdome 2

4
Operating system mode
The first versions of HP-UX were designed for Uniform Memory Architecture platforms, and so treated
each page of memory the same as every other page. This approach to memory management has
been termed Symmetric Multiprocessor (SMP) mode. LORA introduced a new operating system mode
that takes processor and memory locality into consideration for all process scheduling and memory
management activities. HP-UX 11i v3 chooses the proper mode at boot time based on the quantity
and distribution of local memory.
In general, a system operating in SMP mode should be configured with predominantly interleaved
memory. It would be unusual to see good performance if the operating system is treating memory in
a symmetric fashion while the hardware is exposing the memory localities. Conversely, when the
operating system is running in LORA mode, it needs a lot of local memory to perform effectively.
Application workload
Application memory reference patterns are key to system performance on NUMA platforms. Some
applications exhibit strong locality of memory reference, others scatter accesses to global data, and
some fall in between. Most commercial applications exhibit sufficient reference locality to gain
significant benefit by running with LORA. Technical applications that access extremely large data sets
in a uniform manner are better suited to interleaved memory and SMP mode.
In some cases, the type of the application is not as important as the size of the data set that it
references. If an nPartition is devoted to running one single application, and the working set of that
application consumes the bulk of the available physical memory, then there are few opportunities to
exploit locality. Database management systems are often run in this manner, and so will generally
exhibit more predictable behavior when run in SMP mode. By contrast, if an nPartition is running
multiple applications, or application instances, each of which has a working set smaller than the
amount of physical memory, then LORA mode will usually be able to show a benefit by aligning
processing resources.
LORA configuration rules
The fundamental configuration rule for LORA is to establish ⅞
ths
of the memory in the partition as local
memory (leaving ⅛
th
as interleaved memory). While some circumstances might see a slight
improvement with a few megabytes more or less than ⅞
ths
, the recommended value is an excellent
general choice. It simplifies configuration and management to converge on a single value. The
design point for the HP-UX 11i v3 kernel is to tune for ⅞
ths
local memory.
The processor scheduling algorithms have no flexibility if there is only one processor core available in
any given locality. Therefore, the LORA configuration rules suggest that there be a minimum of two
processor cores in each locality.
At the nPartition level, each blade should be configured with ⅞
ths
local memory to comply with the
LORA configuration rules. The appendix
Configuring nPartitions for LORA contains more detail.
Partitioning
The two partitioning solutions provided by HP are Virtual Partitions (vPars) and Integrity Virtual
Machines. Because these solutions subdivide the physical resources of an nPartition, they present
opportunities to exploit locality.