Concept Guide

Dell EMC BeeGFS Storage Solution Reference Architecture
13 Dell EMC Ready Solutions for HPC BeeGFS High Performance Storage | ID 460
3.7 R740xd, 24x NVMe Drives, Details on CPU Mapping
In the 24xNVMe configuration of the PowerEdge R740xd server, there are two x16 NVMe extender cards
connected to PCIe switches on the backplane which are connected to the drives. Each NVMe drive is
connected to 4x PCIe lanes. This is shown in Figure 8 below:
R740xd, 24x NVMe Details on CPU Mapping
In Non-Uniform Memory Access (NUMA), system memory is divided into zones called nodes, which are
allocated to CPUs or sockets. Access to memory that is local to a CPU is faster than memory connected to
remote CPUs on the system. A threaded application typically performs best when the threads are accessing
memory on the same NUMA node. The performance impact of NUMA misses is significant, generally starting
at a 10% performance hit or higher. To improve performance, the services are configured to use specific
NUMA zones to avoid unnecessary use of UPI cross-socket links thereby reducing latency. Each NUMA zone
handles 12 drives and uses one of the two InfiniBand EDR interfaces on the servers. This NUMA separation
is achieved by manually configuring NUMA balancing by creating custom systemd unit files and by
configuring multihoming. Hence the automatic NUMA balancing is disabled, as shown below:
# cat /proc/sys/kernel/numa_balancing
0