ccNUMA Overview

ManualsBrandsHP ManualsSoftwareHP General Developer Tools Software

ccNUMA overview

Table of contents

Introduction...............................................................................................................................................3

Executive summary.....................................................................................................................................3

Problem statement......................................................................................................................................3

Introduction to ccNUMA.............................................................................................................................3

Physical memory characteristics ...............................................................................................................3

Cache coherency...................................................................................................................................4

Interleaved memory................................................................................................................................4

Local memory ........................................................................................................................................4

HP high-end SMP servers and the cellular architecture....................................................................................4

Past implementations ..............................................................................................................................5

Locality domains ....................................................................................................................................5

Comparison of existing HP 9000 high-end SMP servers to the new HP Integrity SMP servers...........................6

Competitive situation..................................................................................................................................7

SGI IRIX ................................................................................................................................................7

SGI Altix 64-bit Linux systems ..................................................................................................................7

Tru64 ...................................................................................................................................................7

Linux.....................................................................................................................................................8

IBM NUMA-Q .......................................................................................................................................8

IBM pSeries...........................................................................................................................................8

Sun ......................................................................................................................................................8

Impact on applications ...............................................................................................................................9

Memory topology ......................................................................................................................................9

Process management..................................................................................................................................9

Launch policies ......................................................................................................................................9

Default launch policies..........................................................................................................................11

Discovering the execution domain topology ............................................................................................12

Utilizing the execution domain...............................................................................................................12

Discovering a process’s locality bindings ................................................................................................12

Inheritance of launch policies ................................................................................................................13

Summary of content (17 pages)

PAGE 1
ccNUMA overview Table of contents Introduction...............................................................................................................................................3 Executive summary.....................................................................................................................................3 Problem statement......................................................................................................................................
PAGE 2
Memory allocation...................................................................................................................................15 malloc() ..............................................................................................................................................15 mmap() and shmget()............................................................................................................................15 Fortran/common/ .............................................
PAGE 3
Introduction ccNUMA (cache coherent Non-Uniform Memory Architecture) systems offer programmers and users the simplicity and flexibility of symmetric multiprocessing (SMP) with the memory scalability of clusters. In a ccNUMA system, processors, memory, and I/O are grouped together into cells. The latency and bandwidth characteristics of communication within a cell are “fast,” while going outside a cell is “slow.
PAGE 4
Cache coherency All modern multiprocessor systems are cache coherent. This is necessary since processor caches have copies of data that is in memory. When one processor modifies data in memory location, it does so in its cache, on a cache line basis. The process of modifying the data must also include the step of notifying any other processors that may have that cache-line that the data in their cache is now invalid.
PAGE 5
Past implementations With HP-UX 11i v1, all memory is interleaved over all the cells. The interleaving is done on a cache-line basis. This makes the memory latency appear uniform. It should be noted that latency is greater for large systems than for small systems since there are more memory references across more crossbars, on average. For example, for a 16 CPU, 4-cell system, a memory reference is satisfied ¼ of the time locally and ¾ of the time remotely.
PAGE 6
Comparison of existing HP 9000 high-end SMP servers to the new HP Integrity SMP servers The primary physical difference between the old and current high-end SMP servers from HP is the improvement of the cell memory bandwidth. The primary logical difference between the two systems is the segregation of memory into local memory and interleave memory, as shown in Figure 3. Figure 3. HP Integrity server memory logical diagram Table 1.
PAGE 7
The HP 9000 Superdome running HP-UX 11i v1.1 acts as UMA with an effective latency closer to 2-hop performance. For example, an 8-cell configuration has an effective latency of 323 ns. The HP Integrity server delivers 243 ns latency and 7.2 GB/s memory bandwidth if the dataset can be contained in local memory. This is a 33% improvement in memory latency, if the applications can take advantage of the performance features of the memory system.
PAGE 8
The NUMA application programming interfaces (APIs) allow applications to make scheduling and resource allocation decisions based on advance knowledge of the application’s resource needs and behavior.
PAGE 9
Impact on applications Many applications will not be adversely affected by ccNUMA, but some key types will. Applications that need to be aware of ccNUMA include: • Pthreads applications • SystemV shared memory applications • Message passing applications. MPI is ccNUMA-aware with version 1.8.
PAGE 10
HP-UX thread-launch and process-launch policies are independent of one another. When a process creates another process (via fork() or vfork()), the child process inherits the parent process’s launch policy. The initial thread in the child process inherits the launch policy of the creating thread (and not that of its process). Other threads in a multi-threaded process inherit their launch policy from the creating thread.
PAGE 11
Fill First is the default policy for programs that are gang-scheduled. The launch policy tree only extends from the parent to its immediate children. The children then become the root of a new launch policy tree. Fill Tree mpsched policy name: FILL_TREE This request establishes a tree-based Fill First launch policy for the specified thread. This request differs from PTHREAD_POLICY_FILL_NP in which threads become part of the launch tree.
PAGE 12
Discovering the execution domain topology The mpctl() system call is used to discover the system or the processor set (pset) topology. It can also be used to work backward to discover which pset a processor or LDOM belongs on. Table 3.
PAGE 13
Inheritance of launch policies Every thread and process has a Launch Policy associated with it. If no explicit policy is assigned, then, in general, the child process or thread inherits its policy from its parent. There is no explicit relationship between the launch policy of a process and its child threads. HP-UX maintains which thread-launch and process-launch policies are in effect for threads and processes by using policy trees.
PAGE 14
Figure 4. Sample HP-UX launch policy trees The first process in Figure 4 is launched with a Round Robin Tree process-launch policy and a Fill First Tree threadlaunch policy. As a result, the root of the process policy tree (labeled A) establishes a Round Robin Tree processlaunch policy, and the root of the thread policy tree (B) establishes a Fill First Tree thread-launch policy.
PAGE 15
Memory allocation The overall strategy of the memory system is to keep private data local and shared data global. Therefore, the default behavior will be adequate for most applications. Application developers who understand the data structures of the application and how the data is accessed can use the tools presented here to optimize performance.
PAGE 16
Memory exhaustion and paging The VM system has a strategy to allocate memory. The order in which localities are searched depends on the memory object. Objects that desire interleaved memory will first search the interleaved locality and then perform a round-robin search among all the local memory localities. Objects that desire cell local memory will first search that local memory and then will search other localities in a closest-to-furthest order.
PAGE 17
HP-UX 11i release names and release identifiers With HP-UX 11i, HP delivers a highly available, secure, and manageable operating system that meets the demands of end-to-end Internet-critical computing. HP-UX 11i supports enterprise, mission-critical, and technical computing environments. HP-UX 11i is available on both PA-RISC systems and Itanium-based systems. Each HP-UX 11i release has an associated release name and release identifier.