White Papers

Dell HPC Lustre Storage solution with Mellanox Infiniband EDR
The test environment for the solution has a single MDS pair and a single OSS pair with a total of 960TB
of raw disk space. The OSS pair contains two PowerEdge R730s, each with 256GB of memory, four
12Gbps SAS controllers and a single Infiniband EDR adapter. Consult the Dell HPC Lustre Storage
Configuration Guide for details of cabling and expansion card locations. The MDS has identical
configurations with 256GB of memory, an Infiniband EDR adapter and dual 12Gbps SAS controllers.
The Infiniband fabric is comprised of a Mellanox switch, the SB7700, for the client cluster, which the
Lustre servers were also directly connected.
Table 2 shows the details about the characteristics for the different software and hardware
components.
Table 2: Dell HPC Lustre Storage solution configuration
Configuration Size
960TB RAW
Lustre Server Version
2.7.15.3
Intel EE for Lustre Version
V3.0
OSS Nodes
2 x PowerEdge R730 Servers
OSS Memory
256GB DDR4 2133MT/s
OSS Processors
2 x Intel Xeon™ E5-2630V4 @ 2.20GHz 10 cores
OSS Server BIOS
2.0.2
OSS Storage Array
4 x PowerVault MD3460
Drives in OSS Storage Arrays
240 3.5" 4 TB 7.2K RPM NL SAS
OSS SAS Controllers
4 x SAS 12Gbps HBA LSI 9300-8e
MDS Nodes
2 x PowerEdge R730 Servers
MDS Memory
256GB DDR4 2133MT/s
MDS Processors
2 x Intel Xeon™ E5-2630V4 @ 2.20GHz 10 cores
MDS Server BIOS
2.0.2
MDS Storage Array
1 x PowerVault MD3420
Drives in MDS Storage Array
24 - 2.5" 300GB NL SAS
MDS SAS Controller
2 x SAS 12Gbps HBA LSI 9300-8e
Data network
Infiniband EDR
OSS, MDS Servers
Mellanox ConnectX-4 Single Port EDR PCIE Adapter LP
(MT4115)
Compute Nodes
Mellanox ConnectX-4 Single Port EDR PCIE Adapter LP
(MT4115)
To prevent inflated results due to caching effects, tests were performed with a cold cache established
with the following technique. Before each test started, a remount of the Lustre File System under test
was executed. A sync was performed and the kernel was instructed to drop caches on all the clients
with the following commands:
 sync