White Papers

Dell HPC Lustre Storage solution with Intel Omni-Path
number of threads above 64 were simulated by increasing the number of threads per client across all
clients. For instance, for 128 threads, each of the 64 clients ran two threads.
The test environment for the solution has a single MDS pair and a single OSS pair with a total of 960TB
of raw disk space. The OSS pair contains two PowerEdge R730s, each with 256GB of memory, four
12Gbps SAS controllers and a single Intel Omni-Path HFI adapter. Consult the Dell HPC Lustre Storage
Configuration Guide for details of cabling and expansion card locations. The MDS has identical
configurations with 256GB of memory, an Intel Omni-Path HFI adapter and dual 12Gbps SAS controllers.
The Omni-Path fabric is comprised of a large port count Omni-Path fabric core switch for the client
cluster, which the Lustre servers were also directly connected.
Table 2 shows the details about the characteristics for the different software and hardware
components.
Table 2: Dell HPC Lustre Storage solution configuration
Configuration Size
960TB RAW
Lustre Server Version
2.7.15.3
Intel EE for Lustre Version
V3.0
OSS Nodes
2 x PowerEdge R730 Servers
OSS Memory
256GB DDR4 2133MT/s
OSS Processors
2 x Intel Xeon™ E5-2630V4 @ 2.20GHz 10 cores
OSS Server BIOS
2.0.2
OSS Storage Array
4 x PowerVault MD3460
Drives in OSS Storage Arrays
240 3.5" 4 TB 7.2K RPM NL SAS
OSS SAS Controllers
4 x SAS 12Gbps HBA LSI 9300-8e
MDS Nodes
2 x PowerEdge R730 Servers
MDS Memory
256GB DDR4 2133MT/s
MDS Processors
2 x Intel Xeon™ E5-2630V4 @ 2.20GHz 10 cores
MDS Server BIOS
2.0.2
MDS Storage Array
1 x PowerVault MD3420
Drives in MDS Storage Array
24 - 2.5" 300GB NL SAS
MDS SAS Controller
2 x SAS 12Gbps HBA LSI 9300-8e
Data network – Intel Omni-
Path
OSS, MDS Servers
Intel Omni-Path HFI adapter
Compute Nodes
Intel Omni-Path HFI adapter
To prevent inflated results due to caching effects, tests were performed with a cold cache established
with the following technique. Before each test started, a remount of the Lustre File System under test
was executed. A sync was performed and the kernel was instructed to drop caches on all the clients
with the following commands: