Concept Guide

Benchmarks and test tools
30 Dell EMC Ready Solutions for HPC BeeGFS High Performance Storage | ID 460
A.4 Intel Data Center Tool
The Intel Data Center Tool was used to reformat the Intel P4600 NVMe devices with 512b blocks for
metadata and 4k blocks for storage as shown below:
# On servers with meta on NUMA 0 and storage on NUMA 1
for i in 0 11 {16..23} 1 2 ; do isdct start -f -intelssd $i -nvmeformat
LBAFormat=0 SecureEraseSetting=0 ; done
for i in {3..10} {12..15} ; do isdct start -f -intelssd $i -nvmeformat
LBAFormat=1 SecureEraseSetting=0 ; done
# On servers with all storage
for i in {0..23} ; do isdct start -f -intelssd $i -nvmeformat LBAFormat=1
SecureEraseSetting=0 ; done
However, before formatting the devices, one must confirm which devices are on NUMA zone 0 and which
ones are in NUMA zone 1, using the isdct show -a command. This is because the device IDs do not
match the kernel scan order. Moreover, the LBAFormat is also device specific.
The server needs to be rebooted after changing the low-level NVMe block sizes. The following command can
be used to check the details of the devices.
for i in nvme{0..23} ; do echo -n $i " " ; grep -h .
/sys/class/nvme/$i/{address,*/numa_node,*/*/hw_sector_size} | xargs ; done |
column -t
# Output on an R740XD w/ meta + storage:
nvme0 0000:62:00.0 0 512
nvme1 0000:63:00.0 0 512
nvme2 0000:64:00.0 0 512
nvme3 0000:65:00.0 0 512
nvme4 0000:66:00.0 0 512
nvme5 0000:67:00.0 0 512
nvme6 0000:68:00.0 0 512
nvme7 0000:69:00.0 0 512
nvme8 0000:6a:00.0 0 512
nvme9 0000:6b:00.0 0 512
nvme10 0000:6c:00.0 0 512
nvme11 0000:6d:00.0 0 512
nvme12 0000:b3:00.0 1 4096
nvme13 0000:b4:00.0 1 4096
nvme14 0000:b5:00.0 1 4096
nvme15 0000:b6:00.0 1 4096
nvme16 0000:b7:00.0 1 4096
nvme17 0000:b8:00.0 1 4096
nvme18 0000:b9:00.0 1 4096