The Inference Engine can infer models in different formats with various input and output formats. This section provides supported and optimal configurations per device.
The Inference Engine provides unique capabilities to infer deep learning models on the following device types with corresponding plugins:
| Plugin | Device types |
|---|---|
| GPU plugin | Intel® Processor Graphics, including Intel® HD Graphics and Intel® Iris® Graphics |
| CPU plugin | Intel® Xeon® with Intel® AVX2 and AVX512, Intel® Core™ Processors with Intel® AVX2, Intel® Atom® Processors with Intel® SSE |
| FPGA plugin (available in the Intel® Distribution of OpenVINO™ toolkit) | Intel® Arria® 10 GX FPGA Development Kit, Intel® Programmable Acceleration Card with Intel® Arria® 10 GX FPGA, Intel® Vision Accelerator Design with an Intel® Arria® 10 FPGA |
| VPU plugins (available in the Intel® Distribution of OpenVINO™ toolkit) | Intel® Movidius™ Neural Compute Stick powered by the Intel® Movidius™ Myriad™ 2, Intel® Neural Compute Stick 2 powered by the Intel® Movidius™ Myriad™ X, Intel® Vision Accelerator Design with Intel® Movidius™ VPUs |
| GNA plugin (available in the Intel® Distribution of OpenVINO™ toolkit) | Intel® Speech Enabling Developer Kit, Amazon Alexa* Premium Far-Field Developer Kit, Intel® Pentium® Silver processor J5005, Intel® Celeron® processor J4005, Intel® Core™ i3-8121U processor |
| Heterogeneous plugin | Heterogeneous plugin enables computing for inference on one network on several Intel® devices. |
The Inference Engine can inference models in different formats with various input and output formats. This chapter provides supported and optimal configurations for each plugin.
| Acronym/Term | Description |
|---|---|
| DL | Deep Learning |
| FP32 format | Single-precision floating-point format |
| FP16 format | Half-precision floating-point format |
| I16 format | 2-byte signed integer format |
| I8 format | 1-byte signed integer format |
| U16 format | 2-byte unsigned integer format |
| U8 format | 1-byte unsigned integer format |
NHWC, NCHW - Image data layout. Refers to the representation of batches of images. NCDHW - Images sequence data layout.
CHW, NC, C - Tensor memory layout. For example, the CHW value at index (c,h,w) is physically located at index (c*H+h)*W+w, for others by analogy
| Plugin | FP32 | FP16 |
|---|---|---|
| CPU plugin | Supported and preferred | Not supported |
| GPU plugin | Supported | Supported and preferred |
| FPGA plugin | Supported | Supported |
| VPU plugins | Not supported | Supported |
| GNA plugin | Supported | Not supported |
| Plugin | FP32 | FP16 | U8 | U16 | I8 | I16 |
|---|---|---|---|---|---|---|
| CPU plugin | Supported | Not supported | Supported | Supported | Not supported | Supported |
| GPU plugin | Supported | Supported* | Supported* | Supported* | Not supported | Supported* |
| FPGA plugin | Supported | Supported* | Supported | Supported | Not supported | Supported |
| VPU plugins | Supported | Supported | Supported | Not supported | Not supported | Not supported |
| GNA plugin | Supported | Not supported | Not supported | Not supported | Supported | Supported |
* - Supported via SetBlob only, GetBlob returns FP32
| Plugin | FP32 | FP16 |
|---|---|---|
| CPU plugin | Supported | Not supported |
| GPU plugin | Supported | Supported |
| FPGA plugin | Supported | Supported |
| VPU plugins | Supported | Supported |
| GNA plugin | Supported | Not supported |
| Plugin | NCDHW | NCHW | NHWC | NC |
|---|---|---|---|---|
| CPU plugin | Supported | Supported | Supported | Supported |
| GPU plugin | Not supported | Supported | Supported | Supported |
| FPGA plugin | Not supported | Supported | Supported | Not supported |
| VPU plugins | Not supported | Supported | Supported | Supported |
| GNA plugin | Not supported | Not supported | Not supported | Supported |
| Number of dimensions | 5 | 4 | 3 | 2 | 1 |
|---|---|---|---|---|---|
| Layout | NCDHW | NCHW | CHW | NC | C |
For setting relevant configuration, refer to the Integrate with Customer Application New Request API topic (step 3 "Configure input and output").
The following layers are supported by the plugins and by Shape Inference feature:
| Layers | GPU | CPU | VPU | GNA | FPGA | ShapeInfer |
|---|---|---|---|---|---|---|
| Activation-Clamp | Supported | Supported*** | Supported | Supported | Supported | Supported |
| Activation-ELU | Supported | Supported*** | Supported | Not Supported | Supported | Supported |
| Activation-Leaky ReLU | Supported | Supported*** | Supported | Supported | Supported | Supported |
| Activation-PReLU | Supported | Supported*** | Supported | Not Supported | Supported | Supported |
| Activation-ReLU | Supported | Supported*** | Supported | Supported | Supported | Supported |
| Activation-ReLU6 | Supported | Supported*** | Supported | Not Supported | Not Supported | Supported |
| Activation-Sigmoid/Logistic | Supported | Supported*** | Supported | Supported | Not Supported | Supported |
| Activation-TanH | Supported | Supported*** | Supported | Supported | Not Supported | Supported |
| ArgMax | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
| BatchNormalization | Supported | Supported | Supported | Not Supported | Supported* | Supported |
| Concat | Supported | Supported*** | Supported | Supported | Supported | Supported |
| Const | Supported | Supported | Supported | Not Supported | Not Supported | Not Supported |
| Convolution-Dilated | Supported | Supported | Supported | Not Supported | Not Supported | Supported |
| Convolution-Dilated 3D | Not Supported | Supported | Not Supported | Not Supported | Not Supported | Not Supported |
| Convolution-Grouped | Supported | Supported | Supported | Not Supported | Supported | Supported |
| Convolution-Grouped 3D | Not Supported | Supported | Not Supported | Not Supported | Not Supported | Not Supported |
| Convolution-Ordinary | Supported | Supported | Supported | Supported* | Supported | Supported |
| Convolution-Ordinary 3D | Not Supported | Supported | Not Supported | Not Supported | Not Supported | Not Supported |
| Crop | Supported | Supported | Supported | Supported | Not Supported | Supported |
| CTCGreedyDecoder | Supported** | Supported** | Supported* | Not Supported | Not Supported | Supported |
| Deconvolution | Supported | Supported | Supported | Not Supported | Supported* | Supported |
| Deconvolution 3D | Not Supported | Supported | Not Supported | Not Supported | Not Supported | Not Supported |
| DetectionOutput | Supported | Supported** | Supported* | Not Supported | Not Supported | Supported |
| Eltwise-Max | Supported | Supported*** | Supported | Not Supported | Not Supported | Supported |
| Eltwise-Mul | Supported | Supported*** | Supported | Supported | Not Supported | Supported |
| Eltwise-Sum | Supported | Supported*** | Supported | Supported | Supported | Supported |
| Flatten | Supported | Supported | Supported | Not Supported | Not Supported | Supported |
| FullyConnected (Inner Product) | Supported | Supported*** | Supported | Supported | Supported | Supported |
| Gather | Not Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
| Gemm | Not Supported | Supported | Not Supported | Not Supported | Not Supported | Supported |
| GRN | Supported** | Supported** | Supported | Not Supported | Not Supported | Supported |
| Interp | Supported** | Supported** | Supported | Not Supported | Not Supported | Supported* |
| LRN (Norm) | Supported | Supported | Supported* | Not Supported | Supported | Supported |
| LSTMCell | Not Supported | Supported | Not Supported | Not Supported | Not Supported | Not Supported |
| Memory | Not Supported | Supported | Not Supported | Supported | Not Supported | Supported |
| MVN | Supported | Supported** | Supported* | Not Supported | Not Supported | Supported |
| Normalize | Supported | Supported** | Supported* | Not Supported | Not Supported | Supported |
| Pad | Supported | Supported** | Supported* | Not Supported | Not Supported | Supported |
| Permute | Supported | Supported | Supported | Supported | Not Supported | Supported |
| Pooling(AVG,MAX) | Supported | Supported | Supported | Supported | Supported | Supported |
| Pooling(AVG,MAX) 3D | Not Supported | Supported | Not Supported | Not Supported | Not Supported | Not Supported |
| Power | Supported | Supported** | Supported | Supported | Supported* | Supported |
| PriorBox | Supported | Supported** | Supported | Not Supported | Not Supported | Supported |
| PriorBoxClustered | Supported** | Supported** | Supported | Not Supported | Not Supported | Supported |
| Proposal | Supported | Supported** | Supported | Not Supported | Not Supported | Supported |
| PSROIPooling | Supported | Supported** | Supported | Not Supported | Not Supported | Supported |
| RegionYolo | Supported | Supported** | Supported | Not Supported | Not Supported | Supported |
| ReorgYolo | Supported | Supported** | Supported | Not Supported | Not Supported | Supported |
| Resample | Supported | Supported** | Supported | Not Supported | Not Supported | Supported |
| Reshape | Supported | Supported*** | Supported | Supported | Not Supported | Supported* |
| RNN | Not Supported | Supported | Not Supported | Not Supported | Not Supported | Not Supported |
| ROIPooling | Supported* | Supported | Supported | Supported | Not Supported | Supported |
| ScaleShift | Supported | Supported*** | Supported* | Supported | Supported | Supported |
| SimplerNMS | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
| Slice | Supported | Supported*** | Supported | Supported | Supported* | Supported |
| SoftMax | Supported | Supported*** | Supported | Not Supported | Not Supported | Supported |
| SpatialTransformer | Not Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
| Split | Supported | Supported*** | Supported | Supported | Supported* | Supported |
| TensorIterator | Not Supported | Supported | Not Supported | Not Supported | Not Supported | Not Supported |
| Tile | Supported** | Supported*** | Supported | Not Supported | Not Supported | Supported |
| Unpooling | Supported | Not Supported | Not Supported | Not Supported | Not Supported | Not Supported |
| Upsampling | Supported | Not Supported | Not Supported | Not Supported | Not Supported | Not Supported |
*- support is limited to the specific parameters. Refer to "Known Layers Limitation" section for the device from the list of supported.
**- support is implemented via custom kernels mechanism.
***- supports NCDHW layout.