White Papers
3 CheXNet – Inference with Nvidia T4 on Dell EMC PowerEdge R7425
Table of contents
Revisions............................................................................................................................................................................. 2
Acknowledgements ............................................................................................................................................................. 2
Executive summary ............................................................................................................................................................. 4
1 Background & Definitions ............................................................................................................................................. 5
1.1 Dell EMC PowerEdge R7425 ............................................................................................................................. 7
2 Test Methodology ......................................................................................................................................................... 8
2.1 Test Design ......................................................................................................................................................... 8
2.2 Test Setup ........................................................................................................................................................ 10
3 Development Methodology ......................................................................................................................................... 11
3.1 Build a CheXNet Model with TensorFlow Framework ...................................................................................... 11
3.2 Train the model for Inference with Estimator .................................................................................................... 16
3.3 Save the Trained Model with TensorFlow Serving for Inference ..................................................................... 17
3.4 Freeze the Saved Model (optional) .................................................................................................................. 17
4 Inference with TensorRT™ ........................................................................................................................................ 19
4.1 TensorRT™ using TensorFlow-TensorRT (TF-TRT) Integrated ...................................................................... 19
4.1.1 TF-TRT Workflow with a Frozen Graph............................................................................................................ 19
4.2 TensorRT™ using TensorRT C++ API ............................................................................................................. 24
5 Results ........................................................................................................................................................................ 30
5.1 CheXNet Inference - Native TensorFlow FP32fp32 with CPU Only ................................................................ 30
5.2 CheXNet Inference - Native TensorFlow fp32 with GPU ................................................................................. 31
5.3 CheXNet Inference –TF-TRT 5.0 Integration in INT8int8 Precision Mode ....................................................... 32
5.4 Benchmarking CheXNet Model Inference with Official ResnetV2_50.............................................................. 34
5.5 CheXNet Inference - Native TensorFlow FP32fp32 with GPU versus TF-TRT 5.0 INT8 ............................... 35
5.6 CheXNet Inference - TF-TRT 5.0 Integration vs Native TRT5 C++ API .......................................................... 39
5.7 CheXNet Inference – Throughput with TensorRT™ at ~7ms Latency Target ................................................. 41
6 Conclusion and Future Work ...................................................................................................................................... 44
A Troubleshooting .......................................................................................................................................................... 45
B References ................................................................................................................................................................. 47
C Appendix - PowerEdge R7425 – GPU Features ........................................................................................................ 49