Administrator Guide

Solution Architecture
15 Dell EMC Ready Solution for HPC PixStor Storage | Document ID
All the virtual disks on the storage module and HDMD module are exported as volumes that are accessible to
any HBA port from the two R740s connected to them, and each R740 has one HBA port connected to each
ME4 controller from their storage arrays. Such that even if one server is operational and only a single SAS
cable remains connected to each ME4, the solution can still provide access to all data (or metadata) stored in
those arrays.
Figure 6 ME4024 drives assigned to LUN for Configuration with High Demand Meta Data.
Finally, high-speed networks are connected via CX5 adapters to handle information exchange with clients, but
also to evaluate if a node part of a module is operational.
NVMe Tier configuration
Each PE R640 has 8 NVMe devices directly connected to the CPU in Socket 2 (so this is not a balanced
configuration in terms of NUMA domains), and two HCAs Mellanox ConnectX-6 Single Port VPI HDR100
adapters (one per CPU socket). For the configuration characterized, Dell P4610 devices were used, since
they have the same read and write performance for large blocks (3.2 GB/s) and fairly good random IO
performance for small blocks, which are nice features when trying to scale and estimate the number of pairs
needed to meet the requirements of this flash tier. Nevertheless, any NVMe device supported on the
PowerEdge R640 will be supported for the NVMe nodes.
Those NVMe drives are configured as eight RAID 10 devices across a pair of servers, using NVMesh as the
NVMe over Fabric component to allow data redundancy not only at the devices level, but at the server level.
In addition, when any data goes into or out of one of those RAID10 devices, all 16 drives in both servers are
used, increasing the bandwidth of the access to that of all the drives. Therefore, the only restriction for these
NVMe tier servers is that they must be sold and used in pairs.
The R640 tested in this configuration were used with EDR 100 Gb IB connections, but since they already
have CX6 adapters, the NVMe nodes are ready to support HDR100 speeds when used with HDR cables and
switches. Testing HDR100 on these nodes is deferred as part of the HDR100 update for the whole PixStor
solution. Both CX6 interfaces are used to sync data for the RAID 10 (NVMe over fabric) and as the
connectivity for the file system. In addition, they provide hardware redundancy at the adapter, port and cable.
For redundancy at the switch level, dual port CX6 VPI adapters are required, but need to be procured as S&P
components.
To characterize the performance of NVMe nodes, from the system depicted in figure 1, only the high demand
metadata module and the NVMe nodes were used.
Gateway Nodes
Gateway nodes use the same PE R740 as other servers in the solution, which have eight PCIe slots, three
x16 and five x8. The three x16 slots have Mellanox ConnectX-5 Single Port VPI adapters, that can be
configured for either IB EDR 100 Gb or Ethernet connections at any of the speeds supported by those