Designing a High Performance Network File Server

ManualsBrandsHP ManualsSoftwareHP-UX 11i v2 Networking Software

An Integrity Superdome Case Study

Dave Olker

Systems Networking and Security Lab

dave.olker@hp.com

Executive Summary .............................................................................................................................. 2

Customer Application Requirements ....................................................................................................... 2

Benchmark Methodology ...................................................................................................................... 3

Detailed Server Configuration ............................................................................................................... 4

Install Required Patches..................................................................................................................... 4

Filesystem Considerations ................................................................................................................. 5

Filesystem Initial Creation .............................................................................................................. 5

Filesystem Tuning.......................................................................................................................... 5

NFS and Kernel RPC Tunable Parameters............................................................................................ 7

Networking Considerations............................................................................................................... 8

Enable TCP Segmentation Offload.................................................................................................. 8

Enable Checksum Offload ............................................................................................................. 8

Increase TCP Send and Receive Windows....................................................................................... 9

Hardware Configuration Considerations...........................................................................................10

Optimally Assign Ethernet Card Interrupt Handling Duties ............................................................... 10

Optimize APA Port Placement and Configuration ........................................................................... 12

Ensure Network Switch is Configured to Match APA Configuration .................................................. 12

Bind nfsktcpd Service Threads to the CPUs Processing Interrupts....................................................... 13

Summary .......................................................................................................................................... 14

For More Information ......................................................................................................................... 14

Summary of content (14 pages)

PAGE 1
Designing a High Performance Network File Server An Integrity Superdome Case Study Dave Olker Systems Networking and Security Lab dave.olker@hp.com Executive Summary .............................................................................................................................. 2 Customer Application Requirements ....................................................................................................... 2 Benchmark Methodology .....................................................
PAGE 2
Executive Summary During a recent customer sales engagement, HP was asked to deliver an Integrity Superdome system capable of serving huge amounts of file data to the customer’s production environment. The customer’s application runs simultaneously on thousands of compute nodes. The compute nodes use the Network File System (NFS) protocol running over a TCP/IP network to retrieve application data from a central file server.
PAGE 3
Benchmark Methodology Server System Figure 1 below illustrates the NFS server used during this benchmark effort. This system includes: • • • • • 32 (1.5 GHz) Integrity CPUs Two I/O Cabinets 16 dual-port A7012A Gigabit Ethernet Adapters 256 Gigabytes of physical RAM A single 128 Gigabyte data file Figure 1 – Integrity Superdome System Used During Benchmark Client Systems The client systems used to generate the I/O traffic during this benchmark were eighty (80) rx1620 systems running HP-UX 11i v2.
PAGE 4
Detailed Server Configuration This section of the paper describes how the Integrity Superdome server was configured at the end of this engagement to meet this customer’s application requirements. Install Required Patches Table 1 below details the patches required to implement this solution. Be sure to check the patch database on the IT Resource Center (http://itrc.hp.com) to verify whether these patches are still current or have been replaced.
PAGE 5
Tip: No two customer environments are identical There will always be differences in application mix, server hardware, CPU load, networking infrastructure, etc. It is therefore important to understand as much about the customer environment and application mix when deciding which of the following recommendations to use in any specific customer environment.
PAGE 6
For example, the characteristics of the application used during this customer engagement were: • • • • Read intensive workload Huge file sizes Massively parallel simultaneous access Large data transfers With these application attributes in mind, several vxtunefs(1M) parameters were identified that helped improve application throughput tremendously. These parameter settings are listed in Table 2.
PAGE 7
NFS and Kernel RPC Tunable Parameters Listed in Table 3 are the HP-UX kernel tunable parameter settings used during this engagement. These parameters are only available after installing the patches outlined in the “Install Required Patches” section on page 4.
PAGE 8
Networking Considerations In order to provide over 3 Gigabytes of file data per second to the NFS clients, this server needed to simultaneously drive 32 Gigabit Ethernet interfaces at wire speed. To achieve this feat, several networking performance optimizations were required. Among them are TCP segmentation offload (TSO), checksum offload (CKO), and increasing the TCP send and receive windows on the NFS client and server systems.
PAGE 9
Checksum Offload is supported on “iether”, “igelan” and “gelan” type Gigabit Ethernet interfaces as well as “hpixgbe” type 10 Gigabit Ethernet adapters. CKO may be enabled or disabled for both the transmit side or receive side of each Ethernet card. Configuring CKO can either be done interactively using the lanadmin(1M) command with the “-X send_cko_on” or “-X recv_cko_on” options or automatically during system boot by configuring the appropriate configuration file for your NIC type in the /etc/rc.config.
PAGE 10
TRANSPORT_NAME[1]=tcp NDD_NAME[1]=tcp_xmit_hiwater_def NDD_VALUE[1]=262144 In addition to HP-UX clients, this engagement involved using many Linux NFS clients. It was therefore necessary to tune the TCP send and receive windows on those systems. Linux clients use the sysctl(8) utility to tune kernel parameters. The specific parameters that control the TCP send and receive buffer sizes are net.ipv4.tcp_rmem and net.ipv4.tcp_wmem.
PAGE 11
Figure 2 – intctl(1M) Output Showing Original Ethernet Interrupt CPU Assignments The intctl command is able to accept an ASCII file with the desired CPU mappings via the –r option. For this example system, an input file called “intctl.mappings” was created containing the optimal interrupt mappings. The contents of the input file are displayed below. # cat intctl.
PAGE 12
Optimize APA Port Placement and Configuration This customer’s application required the NFS server to deliver over 3 Gigabytes of data per second to the client systems, and these clients were distributed across eight networks. As the server used 32 Gigabit Ethernet interfaces to deliver this data, these interfaces were configured into eight APA trunks each containing four NICs.
PAGE 13
It is extremely important to ensure the configuration of the APA link aggregates match the configuration of the link partner (network switch or server). In particular, the server and switch should specify the same trunking protocol and use the same load balancing algorithm. Proper configuration is critical to ensure that the inbound traffic to the NFS server is evenly balanced across all the ports in the APA link aggregates.
PAGE 14
Summary When designing a high performance file server, it is important to ensure that all components of the system are properly configured and tuned for the specific applications being served; including: • • • • Kernel Parameters Filesystems Network Configuration Hardware Configuration All of the recent improvements described in this white paper may not be applicable to implement on every HP-UX server.