White Papers

Dell HPC NFS Storage Solution High Availability (NSS-HA) Configurations with Dell PowerEdge 12th

Generation Servers

PCI-E slots usage in the Dell PowerEdge R620: option 1 vs. option 2

Figure 3.

SAS

Slot 3

SAS

Slot 2

Slot 1

CPU

x16

Option 1

SAS

Slot 3

SAS

Slot 2

Slot 1

CPU

x16

Option 2

R620

In NSS3-HA, both SAS HBA cards and the InfiniBand HCA card only work with PCI-E x8; thus, it seems

that there should be no difference between the two options, as the PCI-E bandwidth provided in Dell

PowerEdge R620 can satisfy the bandwidth requirement for both SAS cards and InfiniBand card.

However, as the I/O controller is integrated with the processor in the Intel Sandy Bridge

microarchitecture

(4)

; it implies that a processor will get privileged access to some I/O slots due to the

shorter physical distance, which is very similar to the well-known NUMA (Non-Uniform Memory Access)

effect. For the other processor, the access to I/O slots requires an extra hop using the QPI link

between the sockets. Given the non-uniform IO access to the different PCI slots, it is hard to decide if

the two options will have similar performance as the two options present different communication

paths between processors and I/O devices.

Based on more than twenty performance benchmarking tests in the Dell HPC lab, option 1 and option 2

have very similar write performance, but option 1 can deliver up to 4GB/sec sequential read

performance, while the maximum read performance of option 2 is only close to 3.3GB/sec. Given the

higher read performance with option 1, option 1 is recommended in the current NSS-HA solution.

NOTE: Given the affinities between processes and data, some level of performance variations may be

introduced by both options, as a processor may not always fetch data from nearby I/O slots.

For NSS-HA with 10GbE, option 1 is the recommended configuration. In this case, the two SAS cards

should be installed as described in Option 1 above. No InfiniBand card is needed, and an onboard 10GbE

Network Daughter Card (NDC) provides the network to the client.

3.3. Comprehensive comparisons between NSS4-HA and NSS3-HA

releases

The previous sections introduced new features in the current NSS-HA release, which are mainly

introduced by the Dell PowerEdge R620 server. This section provides a comprehensive examination