Get a Deep Learning Model Performance Boost with Intel® Platforms

Increase Performance for Deep Learning Inference

The Intel® Distribution of OpenVINO™ toolkit helps accelerate deep learning inference across a variety of Intel® processors and accelerators. Rather than a one-size-fits-all solution, Intel offers a powerful portfolio of scalable hardware and software solutions, powered by the Intel® Distribution of OpenVINO™ toolkit, to meet the various performance, power, and price requirements of any use case. The benchmarks below demonstrate high performance gains on several public neural networks for a streamlined, quick deployment on Intel® CPU, VPU and FPGA platforms. Use this data to help you decide which hardware is best for your applications and solutions, or to plan your AI workload on the Intel computing already included in your solutions.

Measuring inference performance involves many variables and is extremely use-case and application dependent. We use the below four parameters for measurements, which are key elements to consider for a successful deep learning inference application:

  1. Throughput - Measures the number of inferences delivered within a latency threshold. (for example, number of frames per second). When deploying a system with deep learning inference, select the throughput that delivers the best trade-off between latency and power for the price and performance that meets your requirements.
  2. Value - While throughput is important, what is more critical in edge AI deployments is the performance efficiency or performance-per-cost. Application performance in throughput per dollar of system cost is the best measure of value.
  3. Efficiency - System power is a key consideration from the edge to the data center. When selecting deep learning solutions, power efficiency (throughput/watt) is a critical factor to consider. Intel designs provide excellent power efficiency for running deep learning workloads.
  4. Total Benefit (Most applicable for Intel® VPU Platforms) - Combining the factors of value and efficiency can be a good way to compare which hardware yields the best performance per watt and per dollar for your particular use case.

Intel® Xeon® E-2124G

throughput_xeon_e212g.png
value_xeon_e212g.png
eff_xeon_e212g.png

Intel® Xeon® Silver 4216R

throughput_xeon_silver.png
value_xeon_silver.png
eff_xeon_silver.png

Intel® Xeon® Gold 5218T

throughput_xeon_gold.png
value_xeon_gold.png
eff_xeon_gold.png

Intel® Xeon® Platinum 8270

throughput_xeon_platinum.png
value_xeon_platinum.png
eff_xeon_platinum.png

Intel® Core™ i3-8100

throughput_i3.png
value_i3.png
eff_i3.png

Intel® Core™ i5-8500

throughput_i5.png
value_i5.png
eff_i5.png

Intel® Core™ i7-8700T

throughput_i7.png
value_i7.png
eff_i7.png

Intel® Neural Compute Stick 2

throughput_ncs2.png
value_ncs2.png
eff_ncs2.png
benefit_ncs2.png

Intel® Vision Accelerator Design with Intel® Movidius™ VPUs (Uzel* UI-AR8)

throughput_hddlr.png
value_hddlr.png
eff_hddlr.png

Intel® Vision Accelerator Design with Intel® Arria® 10 FPGA

throughput_ivad_fpga.png
value_ivad_fpga.png
eff_ivad_fpga.png

Platform Configurations

Intel® Distribution of OpenVINO™ toolkit 2020.1 performance benchmark numbers are based on production build 2020.1.023.

Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Learn more at intel.com, or from the OEM or retailer. Performance results are based on testing as of January 29, 2020 and may not reflect all publicly available security updates. See configuration disclosure for details. No product can be absolutely secure.

Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more complete information, see Performance Benchmark Test Disclosure.

Your costs and results may vary.

© Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.

Testing by Intel done on: see test date for each HW platform below.

CPU Inference Engines

Intel® Xeon® E-2124G Intel® Xeon® Silver 4216R Intel® Xeon® Gold 5218T Intel® Xeon® Platinum 8270
Motherboard ASUS* WS C246 PRO Intel® Server Board S2600STB Intel® Server Board S2600STB Intel® Server Board S2600STB
CPU Intel® Xeon® E-2124G CPU @ 3.40GHz Intel® Xeon® Silver 4216R CPU @ 2.20GHz Intel® Xeon® Gold 5218T CPU @ 2.10GHz Intel® Xeon® Platinum 8270 CPU @ 2.70GHz
Hyper Threading N/A ON ON ON
Turbo Setting N/A N/A N/A ON
Memory 2 x 8 GB DDR4 2133 MHz 12 x 32 GB DDR4 2400MHz 12 x 32 GB DDR4 2666MH 12 x 32 GB DDR4 2933MHz
Operating System Ubuntu* 18.04 LTS Ubuntu* 18.04 LTS Ubuntu* 18.04 LTS Ubuntu* 18.04 LTS
Kernel Version 5.3.0-24-generic 5.3.0-24-generic 5.3.0-24-generic 5.3.0-24-generic
BIOS Vendor American Megatrends Inc.* Intel Corporation Intel Corporation Intel Corporation
BIOS Version 0904 SE5C620.86B.02.01.
0009.092820190230
SE5C620.86B.02.01.
0009.092820190230
SE5C620.86B.02.01.
0009.092820190230
BIOS Release April 12, 2019 September 28, 2019 September 28, 2019 September 28, 2019
BIOS Settings Select optimized default settings,
save & exit
Select optimized default settings,
change power policy
to "performance",
save & exit
Select optimized default settings,
change power policy to "performance",
save & exit
Select optimized default settings,
change power policy to "performance",
save & exit
Batch size 1 1 1 1
Precision INT8 INT8 INT8 INT8
Number of concurrent inference requests 4 32 32 52
Test Date January 29, 2020 January 29, 2020 January 29, 2020 January 29, 2020
Power dissipation, TDP in Watt 71 100 105 205
CPU Price on January 29, 2020, USD
Prices may vary
213 1,002 1,349 7,405

CPU Inference Engines (continue)

Intel® Core™ i3-8100 Intel® Core™ i5-8500 Intel® Core™ i7-8700T
Motherboard GIGABYTE* Z390 UD ASUS* PRIME Z370-A GIGABYTE* Z370M DS3H-CF
CPU Intel® Core™ i3-8100 CPU @ 3.60GHz Intel® Core™ i5-8500 CPU @ 3.00GHz Intel® Core™ i7-8700T CPU @ 2.40GHz
Hyper Threading N/A N/A ON
Turbo Setting N/A N/A ON
Memory 2 x 8 GB DDR4 2133MHz 2 x 8 GB DDR4 2133MHz 2 x 8 GB DDR4 2666MHz
Operating System Ubuntu* 18.04 LTS Ubuntu* 18.04 LTS Ubuntu* 18.04 LTS
Kernel Version 5.3.0-24-generic 5.3.0-24-generic 5.3.0-24-generic
BIOS Vendor American Megatrends Inc.* American Megatrends Inc.* American Megatrends Inc.*
BIOS Version F8 2401 F11
BIOS Release May 24, 2019 July 12, 2019 March 13, 2019
BIOS Settings Select optimized default settings,
set OS type to "other",
save & exit
Select optimized default settings,
save & exit
Select optimized default settings,
set OS type to "other",
save & exit
Batch size 1 1 1
Precision INT8 INT8 INT8
Number of concurrent inference requests 4 3 4
Test Date January 29, 2020 January 29, 2020 January 29, 2020
Power dissipation, TDP in Watt 65 65 35
CPU Price on January 29, 2020, USD
Prices may vary
117 192 303

Accelerator Inference Engines

Intel® Neural Compute Stick 2 Intel® Vision Accelerator Design
with Intel® Movidius™ VPUs (Uzel* UI-AR8)
Intel® Vision Accelerator Design
with Intel® Arria® 10 FPGA - IEI/SAF3*
VPU 1 X Intel® Movidius™ Myriad™ X MA2485 8 X Intel® Movidius™ Myriad™ X MA2485 1 X Intel® Arria® 10 FPGA
Connection USB 2.0/3.0 PCIe X4 PCIe X8
Batch size 1 1 1
Precision FP16 FP16 FP11
Number of concurrent inference requests 4 32 5
Power dissipation, TDP in Watt 2.5 30 60
CPU Price on January 29, 2020, USD
Prices may vary
69 768 1,450
Host Computer Intel® Core™ i7 Intel® Core™ i5 Intel® Core™ i5
Motherboard ASUS* Z370-A II Uzelinfo* / US-E1300 IEI/SAF3*
CPU Intel® Core™ i7-8700 CPU @ 3.20GHz Intel® Core™ i5-6600 CPU @ 3.30GHz Intel® Core™ i5-7500T CPU @ 2.70GHz
Hyper Threading ON N/A N/A
Turbo Setting ON ON ON
Memory 4 x 16 GB DDR4 2666MHz 2 x 16 GB DDR4 2400MHz 2 x 16 GB DDR4 2666MHz
Operating System Ubuntu* 18.04 LTS Ubuntu* 18.04 LTS Ubuntu* 16.04 LTS
Kernel Version 5.0.0-23-generic 5.0.0-23-generic 4.13.0-45-generic
BIOS Vendor American Megatrends Inc.* American Megatrends Inc.* American Megatrends Inc.*
BIOS Version 411 5.12 V2RMAR15
BIOS Release September 21, 2018 September 21, 2018 December 03, 2019
Test Date January 29, 2020 January 29, 2020 January 29, 2020

Please follow this link for more detailed configuration descriptions: Configuration Details

For more complete information about performance and benchmark results, visit: www.intel.com/benchmarks and Optimization Notice.