Designing a High Performance Network File Server

NFS and Kernel RPC Tunable Parameters

Listed in Table 3 are the HP-UX kernel tunable parameter settings used during this engagement. These

parameters are only available after installing the patches outlined in the “Install Required Patches”

section on page 4.

Table 3 – HP-UX kctune Parameters Tuned during the Engagement

kctune Parameter Description Default Tuned

rpc_svc_maxthreads

Size of per-connection service thread pool 10 1

rpc_svc_nfs_cpu_bind_method

Allows nfsktcpd threads to bind to CPUs 0 1

nfs_srv_read_copy_avoid

Copy avoidance in server-side READ path 0 1

nfs_clnt_avoid_write_copy

Copy avoidance in client-side WRITE path 0 1

nfs_exportfs_rwlock

Enable read/write spinlocks 0 1

The rpc_svc_maxthreads tunable controls the number of nfsktcpd threads the server launches to handle

requests from each NFS client. The default value works very well in most environments – where the

server is able to allocate up to 10 nfsktcpd threads to service requests from each NFS client.

However, in this engagement the customer used thousands of NFS clients simultaneously accessing

the server. Given the number of clients, there were thousands of threads for the server to schedule,

but by limiting each per-client pool to a single thread the server is better able to deal with the thread

scheduling overhead. Again, in most NFS environments the default thread pool size should be used.

One of the major redesign efforts to come out of this engagement is the ability to force the nfsktcpd

service threads to remain bound to a specific CPU for the life of the thread, thereby improving CPU

and cell locality and reducing memory latencies inherent with threads migrating to different CPUs in

different cells. The rpc_svc_nfs_cpu_bind_method tunable controls this behavior. When left at the

default value (0) the nfsktcpd threads are able to migrate to any available CPU. When the feature is

enabled (1) each nfsktcpd thread remains bound to the same CPU for the duration of the thread’s life.

Another key performance enhancement resulting from this engagement involved redesigning the NFS

client and server to avoid an unnecessary step of copying NFS memory buffers while processing a

read or write request. When the tunable nfs_srv_read_copy_avoid is enabled (1) the server passes

memory buffers directly to the Kernel RPC subsystem and avoids the intermediate step of copying the

buffers into an XDR structure. Similarly, when the nfs_clnt_avoid_write_copy tunable is enabled (1)

the client passes memory buffers directly to the Kernel RPC subsystem and avoids copying these

buffers into an XDR structure. By avoiding these unnecessary data copies the overhead associated

with servicing an NFS read or write request is significantly reduced and I/O throughput is increased.

Finally, the nfs_exportfs_rwlock tunable enables a performance enhancement where an exclusive

spinlock protecting the exportfs data structures is replaced by a hashed reader/writer spinlock. This

eliminates spinlock contention by allowing multiple threads to simultaneously read the NFS server’s

exportfs structures. The result is reduced CPU consumption and higher NFS server throughput.

For a detailed description of all ONC/NFS kernel tunable parameters refer to the “Managing NFS

and KRPC Kernel Configurations in HP-UX 11i v2” whitepaper available at http://docs.hp.com

As of the writing of this document the current 11i v2 Kernel RPC patch is PHNE_35117. Be sure to check the ITRC for superseding patches.

As of the writing of this document the current 11i v2 ONC/NFS patch is PHNE_35960. Be sure to check the ITRC for superseding patches.