Designing a High Performance Network File Server

Networking Considerations

In order to provide over 3 Gigabytes of file data per second to the NFS clients, this server needed to

simultaneously drive 32 Gigabit Ethernet interfaces at wire speed. To achieve this feat, several

networking performance optimizations were required. Among them are TCP segmentation offload

(TSO), checksum offload (CKO), and increasing the TCP send and receive windows on the NFS client

and server systems.

Enable TCP Segmentation Offload

TCP segmentation offload (TSO) is a technology for reducing CPU overhead of TCP/IP on fast

networks. The reduction in CPU utilization is achieved primarily by allowing the host to transmit large

frames of data (i.e. frames larger than the links Maximum Transmission Unit or MTU) to the network

interface card (NIC) which are subsequently carved up into smaller MTU-sized frames by the NIC

before transmission on the wire.

For example, typical networks use an MTU size of 1500 bytes, which means a 32KB NFS request

must be divided into more than 20 separate segments of 1500 bytes before it is sent over the

network. TSO allows the host to pass the entire 32KB of data down to the NIC in a single chunk and

allows the card to divide the data into segments of 1500 bytes, add the TCP and IP protocol headers,

and send the packets over the network without interrupting the CPU. This significantly reduces the

amount of CPU overhead associated with transmitting large amounts of data.

On HP-UX 11i v2, TSO is supported on “iether” and “igelan” type Gigabit Ethernet adapters as well

as “hpixgbe” type 10 Gigabit Ethernet adapters. TSO is enabled by configuring a “virtual” MTU on

the Ethernet interface. This VMTU size determines the largest chunk of data the NIC card is able to

receive in a single request. Configuring a VMTU can either be done interactively using the

lanadmin(1M) command with the “-X vmtu” option or automatically during system boot by

configuring the appropriate configuration file for your NIC type in the /etc/rc.config.d

directory.

For example, to configure TSO on an “iether” type interface add a line similar to the following to the

/etc/rc.config.d/hpietherconf file:

HP_IETHER_VMTU[1]=32160

Similarly, to enable TSO on an “igelan” interface, add an appropriate entry for the specific interface

to the /etc/rc.config.d/hpigelanconf file:

HP_IGELAN_VMTU[1]=32160

Finally, to enable TSO on an “hpixgbe “ interface, add an appropriate entry for the specific interface

to the /etc/rc.config.d/hpixgbeconf file:

HP_IXGBE_VMTU[1]=32160

Enable Checksum Offload

Checksum Offload (CKO) is a performance feature that reduces a system CPU’s burden by offloading

computation of the TCP checksum to the network card. The transport layer always performs checksum

validation using a computed checksum. When CKO is enabled (ON) the transport layer will use an

offloaded checksum computation from the network card rather than perform the computation in the

host CPU. When CKO is disabled (OFF) the transport layer will still validate checksums, however it

will perform the computation in the host CPU.