The Inference Engine can infer models in different formats with various input and output formats. This section provides supported and optimal configurations per device.
The Inference Engine provides unique capabilities to infer deep learning models on the following device types with corresponding plugins:
Plugin | Device types |
---|---|
GPU plugin | Intel® Processor Graphics, including Intel® HD Graphics and Intel® Iris® Graphics |
CPU plugin | Intel® Xeon® with Intel® AVX2 and AVX512, Intel® Core™ Processors with Intel® AVX2, Intel® Atom® Processors with Intel® SSE |
FPGA plugin (available in the Intel® Distribution of OpenVINO™ toolkit) | Intel® Vision Accelerator Design with an Intel® Arria 10 FPGA (Speed Grade 1), Intel® Vision Accelerator Design with an Intel® Arria 10 FPGA (Speed Grade 2), Intel® Programmable Acceleration Card with Intel® Arria® 10 GX FPGA |
VPU plugins (available in the Intel® Distribution of OpenVINO™ toolkit) | Intel® Movidius™ Neural Compute Stick powered by the Intel® Movidius™ Myriad™ 2, Intel® Neural Compute Stick 2 powered by the Intel® Movidius™ Myriad™ X, Intel® Vision Accelerator Design with Intel® Movidius™ VPUs |
GNA plugin (available in the Intel® Distribution of OpenVINO™ toolkit) | Intel® Speech Enabling Developer Kit, Amazon Alexa* Premium Far-Field Developer Kit, Intel® Pentium® Silver processor J5005, Intel® Celeron® processor J4005, Intel® Core™ i3-8121U processor |
Multi-Device plugin | Multi-Device plugin enables simultaneous inference of the same network on several Intel® devices in parallel |
Heterogeneous plugin | Heterogeneous plugin enables automatic inference splitting between several Intel® devices (for example if a device doesn't support certain layers). |
The Inference Engine can inference models in different formats with various input and output formats. This chapter provides supported and optimal configurations for each plugin.
Acronym/Term | Description |
---|---|
DL | Deep Learning |
FP32 format | Single-precision floating-point format |
FP16 format | Half-precision floating-point format |
I16 format | 2-byte signed integer format |
I8 format | 1-byte signed integer format |
U16 format | 2-byte unsigned integer format |
U8 format | 1-byte unsigned integer format |
NHWC, NCHW - Image data layout. Refers to the representation of batches of images. NCDHW - Images sequence data layout.
CHW, NC, C - Tensor memory layout. For example, the CHW value at index (c,h,w) is physically located at index (c*H+h)*W+w, for others by analogy
Plugin | FP32 | FP16 | I8 |
---|---|---|---|
CPU plugin | Supported and preferred | Supported | Supported |
GPU plugin | Supported | Supported and preferred | Supported* |
FPGA plugin | Supported | Supported | Not supported |
VPU plugins | Not supported | Supported | Not supported |
GNA plugin | Supported | Supported | Not supported |
* - currently, only limited set of topologies might benefit from enabling I8 model on GPU
For Multi-Device and Heterogeneous execution the supported models formats depends on the actual underlying devices. Generally, FP16 is preferable as it is most ubiquitous and performant.
Plugin | FP32 | FP16 | U8 | U16 | I8 | I16 |
---|---|---|---|---|---|---|
CPU plugin | Supported | Not supported | Supported | Supported | Not supported | Supported |
GPU plugin | Supported | Supported* | Supported* | Supported* | Not supported | Supported* |
FPGA plugin | Supported | Supported* | Supported | Supported | Not supported | Supported |
VPU plugins | Supported | Supported | Supported | Not supported | Not supported | Not supported |
GNA plugin | Supported | Not supported | Supported | Not supported | Supported | Supported |
* - Supported via SetBlob
only, GetBlob
returns FP32
For Multi-Device and Heterogeneous execution the supported input precision depends on the actual underlying devices. Generally, U8 is preferable as it is most ubiquitous.
Plugin | FP32 | FP16 |
---|---|---|
CPU plugin | Supported | Not supported |
GPU plugin | Supported | Supported |
FPGA plugin | Supported | Supported |
VPU plugins | Supported | Supported |
GNA plugin | Supported | Not supported |
For Multi-Device and Heterogeneous execution the supported output precision depends on the actual underlying devices. Generally, FP32 is preferable as it is most ubiquitous.
Plugin | NCDHW | NCHW | NHWC | NC |
---|---|---|---|---|
CPU plugin | Supported | Supported | Supported | Supported |
GPU plugin | Supported | Supported | Supported | Supported |
FPGA plugin | Not supported | Supported | Supported | Not supported |
VPU plugins | Not supported | Supported | Supported | Supported |
GNA plugin | Not supported | Not supported | Not supported | Supported |
Number of dimensions | 5 | 4 | 3 | 2 | 1 |
---|---|---|---|---|---|
Layout | NCDHW | NCHW | CHW | NC | C |
For setting relevant configuration, refer to the Integrate with Customer Application New Request API topic (step 3 "Configure input and output").
The following layers are supported by the plugins and by Shape Inference feature:
Layers | GPU | CPU | VPU | GNA | FPGA | ShapeInfer |
---|---|---|---|---|---|---|
Abs | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
Acos | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
Acosh | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
Activation-Clamp | Supported | Supported*** | Supported | Supported | Supported | Supported |
Activation-ELU | Supported | Supported*** | Supported | Not Supported | Supported | Supported |
Activation-Exp | Supported | Supported*** | Not Supported | Not Supported | Not Supported | Supported |
Activation-Leaky ReLU | Supported | Supported*** | Supported | Supported | Supported | Supported |
Activation-Not | Supported | Supported*** | Not Supported | Not Supported | Not Supported | Supported |
Activation-PReLU | Supported | Supported*** | Supported | Not Supported | Supported | Supported |
Activation-ReLU | Supported | Supported*** | Supported | Supported | Supported | Supported |
Activation-ReLU6 | Supported | Supported*** | Supported | Not Supported | Not Supported | Supported |
Activation-Sigmoid/Logistic | Supported | Supported*** | Supported | Supported | Not Supported | Supported |
Activation-TanH | Supported | Supported*** | Supported | Supported | Not Supported | Supported |
ArgMax | Supported | Supported** | Supported | Not Supported | Not Supported | Supported |
Asin | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
Asinh | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
Atan | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
Atanh | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
BatchNormalization | Supported | Supported | Supported | Not Supported | Supported* | Supported |
BinaryConvolution | Supported | Supported | Not Supported | Not Supported | Not Supported | Supported |
Broadcast | Supported | Supported** | Supported | Not Supported | Not Supported | Supported |
Ceil | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
Concat | Supported | Supported*** | Supported | Supported | Supported | Supported |
Const | Supported | Supported | Supported | Supported | Not Supported | Not Supported |
Convolution-Dilated | Supported | Supported | Supported | Not Supported | Supported | Supported |
Convolution-Dilated 3D | Supported | Supported | Not Supported | Not Supported | Not Supported | Not Supported |
Convolution-Grouped | Supported | Supported | Supported | Not Supported | Supported | Supported |
Convolution-Grouped 3D | Supported | Supported | Not Supported | Not Supported | Not Supported | Not Supported |
Convolution-Ordinary | Supported | Supported | Supported | Supported* | Supported | Supported |
Convolution-Ordinary 3D | Supported | Supported | Not Supported | Not Supported | Not Supported | Not Supported |
Cos | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
Cosh | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
Crop | Supported | Supported | Supported | Supported | Not Supported | Supported |
CTCGreedyDecoder | Supported** | Supported** | Supported* | Not Supported | Not Supported | Supported |
Deconvolution | Supported | Supported | Supported | Not Supported | Supported* | Supported |
Deconvolution 3D | Supported | Supported | Not Supported | Not Supported | Not Supported | Not Supported |
DeformableConvolution | Supported | Supported | Not Supported | Not Supported | Not Supported | Supported |
DepthToSpace | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
DetectionOutput | Supported | Supported** | Supported* | Not Supported | Not Supported | Supported |
Eltwise-And | Supported | Supported*** | Not Supported | Not Supported | Not Supported | Supported |
Eltwise-Add | Supported | Supported*** | Not Supported | Not Supported | Supported | Supported |
Eltwise-Div | Supported | Supported*** | Supported | Not Supported | Not Supported | Supported |
Eltwise-Equal | Supported | Supported*** | Supported | Not Supported | Not Supported | Supported |
Eltwise-FloorMod | Supported | Supported*** | Supported | Not Supported | Not Supported | Supported |
Eltwise-Greater | Supported | Supported*** | Supported | Not Supported | Not Supported | Supported |
Eltwise-GreaterEqual | Supported | Supported*** | Supported | Not Supported | Not Supported | Supported |
Eltwise-Less | Supported | Supported*** | Supported | Not Supported | Not Supported | Supported |
Eltwise-LessEqual | Supported | Supported*** | Supported | Not Supported | Not Supported | Supported |
Eltwise-LogicalAnd | Supported | Supported*** | Supported | Not Supported | Not Supported | Supported |
Eltwise-LogicalOr | Supported | Supported*** | Supported | Not Supported | Not Supported | Supported |
Eltwise-LogicalXor | Supported | Supported*** | Supported | Not Supported | Not Supported | Supported |
Eltwise-Max | Supported | Supported*** | Supported | Not Supported | Not Supported | Supported |
Eltwise-Min | Supported | Supported*** | Supported | Not Supported | Not Supported | Supported |
Eltwise-Mul | Supported | Supported*** | Supported | Supported | Not Supported | Supported |
Eltwise-NotEqual | Supported | Supported*** | Supported | Not Supported | Not Supported | Supported |
Eltwise-Pow | Supported | Supported*** | Supported | Not Supported | Not Supported | Supported |
Eltwise-Prod | Supported | Supported*** | Supported | Supported | Not Supported | Supported |
Eltwise-SquaredDiff | Supported | Supported*** | Supported | Not Supported | Not Supported | Supported |
Eltwise-Sub | Supported | Supported*** | Supported | Supported | Supported | Supported |
Eltwise-Sum | Supported | Supported*** | Supported | Supported | Supported | Supported |
Erf | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
Exp | Supported | Supported | Not Supported | Not Supported | Not Supported | Supported |
FakeQuantize | Not Supported | Supported | Not Supported | Not Supported | Not Supported | Supported |
Fill | Not Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
Flatten | Supported | Supported | Supported | Not Supported | Not Supported | Supported |
Floor | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
FullyConnected (Inner Product) | Supported | Supported*** | Supported | Supported | Supported | Supported |
Gather | Supported | Supported** | Supported | Not Supported | Not Supported | Supported |
GatherTree | Not Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
Gemm | Supported | Supported | Supported | Not Supported | Not Supported | Supported |
GRN | Supported** | Supported** | Supported | Not Supported | Not Supported | Supported |
HardSigmoid | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
Interp | Supported** | Supported** | Supported | Not Supported | Not Supported | Supported* |
Log | Supported | Supported** | Supported | Not Supported | Not Supported | Supported |
LRN (Norm) | Supported | Supported | Supported | Not Supported | Supported | Supported |
LSTMCell | Supported | Supported | Supported | Supported | Not Supported | Not Supported |
GRUCell | Supported | Supported | Not Supported | Not Supported | Not Supported | Not Supported |
RNNCell | Supported | Supported | Not Supported | Not Supported | Not Supported | Not Supported |
LSTMSequence | Supported | Supported | Supported | Not Supported | Not Supported | Not Supported |
GRUSequence | Supported | Supported | Not Supported | Not Supported | Not Supported | Not Supported |
RNNSequence | Supported | Supported | Not Supported | Not Supported | Not Supported | Not Supported |
LogSoftmax | Supported | Supported** | Not Supported | Not Supported | Not Supported | Not Supported |
Memory | Not Supported | Supported | Not Supported | Supported | Not Supported | Supported |
MVN | Supported | Supported** | Supported* | Not Supported | Not Supported | Supported |
Neg | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
NonMaxSuppression | Not Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
Normalize | Supported | Supported** | Supported* | Not Supported | Not Supported | Supported |
OneHot | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
Pad | Supported | Supported** | Supported* | Not Supported | Not Supported | Supported |
Permute | Supported | Supported | Supported | Supported* | Not Supported | Supported |
Pooling(AVG,MAX) | Supported | Supported | Supported | Supported | Supported | Supported |
Pooling(AVG,MAX) 3D | Supported | Supported | Not Supported | Not Supported | Not Supported | Not Supported |
Power | Supported | Supported** | Supported | Supported | Supported* | Supported |
PowerFile | Not Supported | Supported** | Not Supported | Not Supported | Not Supported | Not Supported |
PriorBox | Supported | Supported** | Supported | Not Supported | Not Supported | Supported |
PriorBoxClustered | Supported** | Supported** | Supported | Not Supported | Not Supported | Supported |
Proposal | Supported | Supported** | Supported | Not Supported | Not Supported | Supported |
PSROIPooling | Supported | Supported** | Supported | Not Supported | Not Supported | Supported |
Range | Not Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
Reciprocal | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
ReduceAnd | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
ReduceL1 | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
ReduceL2 | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
ReduceLogSum | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
ReduceLogSumExp | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
ReduceMax | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
ReduceMean | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
ReduceMin | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
ReduceOr | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
ReduceProd | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
ReduceSum | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
ReduceSumSquare | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
RegionYolo | Supported | Supported** | Supported | Not Supported | Not Supported | Supported |
ReorgYolo | Supported | Supported** | Supported | Not Supported | Not Supported | Supported |
Resample | Supported | Supported** | Supported | Not Supported | Supported* | Supported |
Reshape | Supported | Supported*** | Supported | Supported | Not Supported | Supported* |
ReverseSequence | Supported | Supported** | Supported | Not Supported | Not Supported | Supported |
RNN | Not Supported | Supported | Supported | Not Supported | Not Supported | Not Supported |
ROIPooling | Supported* | Supported | Supported | Not Supported | Not Supported | Supported |
ScaleShift | Supported | Supported*** | Supported* | Supported | Supported | Supported |
ScatterUpdate | Not Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
Selu | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
ShuffleChannels | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
Sign | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
Sin | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
Sinh | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
SimplerNMS | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
Slice | Supported | Supported*** | Supported | Supported | Supported* | Supported |
SoftMax | Supported | Supported*** | Supported | Not Supported | Not Supported | Supported |
Softplus | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
Softsign | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
SpaceToDepth | Not Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
SpatialTransformer | Not Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
Split | Supported | Supported*** | Supported | Supported | Supported* | Supported |
Squeeze | Supported | Supported** | Supported | Not Supported | Not Supported | Supported |
StridedSlice | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
Tan | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
TensorIterator | Not Supported | Supported | Supported | Not Supported | Not Supported | Not Supported |
Tile | Supported** | Supported*** | Supported | Not Supported | Not Supported | Supported |
TopK | Supported | Supported** | Not Supported | Not Supported | Not Supported | Supported |
Unpooling | Supported | Not Supported | Not Supported | Not Supported | Not Supported | Not Supported |
Unsqueeze | Supported | Supported** | Supported | Not Supported | Not Supported | Supported |
Upsampling | Supported | Not Supported | Not Supported | Not Supported | Not Supported | Not Supported |
*- support is limited to the specific parameters. Refer to "Known Layers Limitation" section for the device from the list of supported.
**- support is implemented via custom kernels mechanism.
***- supports NCDHW layout.