White Papers

Dell Storage for HPC with Intel Enterprise Edition for Lustre sofware
Lustre:
Lustre 2.5.23 + Mellanox OFED Client
OS:
Red Hat Enterprise Linux 6.5 (2.6.32-431.el6.x86_64)
IB SOFTWARE:
Mellanox OFED 2.2-1
Performance analysis was focused on three key performance markers:
 Throughput, data sequentially transferred in GB/s.
 I/O Operations per second (IOPS).
 Metadata Operations per second (OP/s).
The goal is a broad but accurate review of the capabilities of the Dell Storage for HPC with Intel EE for
Lustre. We selected three benchmarks to accomplish our goal: IOzone, IOR and MDtest.
There are two types of file access methods used with the benchmarks. The first file access method is
N-to-N, where every thread of the benchmark (N clients) writes to a different file (N files) on the
storage system. IOzone and IOR can both be configured to use the N-to-N file-access method. For this
study, we use IOzone for N-to-N access method workloads. The second file access method is N-to-1,
where every thread writes to the same file (N clients, 1 file). For this study, we use IOR for N-to-1
access method workloads. IOR can use MPI-IO, HDF5, or POSIX to run N-to-1 file-access tests. For
purpose of our analysis, we used POSIX. N-to-1 testing determines how the file system handles the
overhead introduced with multiple concurrent requests when multiple threads write or read to the
same file. The overhead encountered comes from threads dealing with Lustre’s file locking and
serialized writes. See Appendix A for examples of the commands used to run these benchmarks.
Each set of tests was run on a range of clients to test the scalability of the solution. The number of
simultaneous physical clients involved in each test was varied from a single client to 64 clients. The
number of threads corresponds to the number of physical servers up to 64. Total numbers of threads
above 64 were achieved by increasing the number of threads per client across all clients. For instance,
for 128 threads, each of the 64 clients runs 2 threads.
The test environment for the solution has a single MDS and a single OSS Pair with a total of 960TB of
raw disk space. The OSS pair contains two PowerEdge R630s, each with 256GB of memory, two 12Gbps
SAS controllers and a single Mellanox ConnectX-3 FDR HCA. Consult the Dell Storage for HPC with Intel
EE for Lustre Configuration Guide for details of cabling and expansion card locations. The MDS has
identical configurations with 256GB of memory, a Mellanox ConnectX-3 FDR HCA and dual 12Gbps SAS
controllers.
The InfiniBand fabric is comprised of a 32-port Mellanox M3601Q QDR InfiniBand switch for the client
cluster and a 36-port Mellanox SX6025 FDR InfiniBand switch for the Dell Storage for HPC with Intel EE
for Lustre servers. Three ports from the M3601Q switch were also connected to the SX6025 switch.
Table 2 shows the details about the characteristics for the different software and hardware
components.