Inference Engine takes two kinds of model description as an input: Intermediate Representation (IR) and nGraph::Function objects. Both should have fixed input shapes to be successfully loaded to the Inference Engine. To feed input data of a shape that is different from the model input shape, resize the model first.
Model resizing on the stage of IR generation or [nGraph::Function creation](TODO: link to nGraph Function creation overview) is the recommended approach. OpenVINO™ provides the following experimental methods for runtime model reshaping:
[ EXPERIMENTAL ] Setting a new input shape with the
InferenceEngine::CNNNetwork::reshape method updates input shapes and propagates them down to the outputs of the model through all intermediate layers.
Shape propagation for
InferenceEngine::CNNNetwork objects created from
nGraph::Function or IR of the version 10 works through the
nGraph shape inference mechanism.
InferenceEngine::CNNNetwork objects created from lower IR versions are considered deprecated and may be reshaped incorrectly or give unexpected results.
To keep the v10 IR resizable by the
InferenceEngine::CNNNetwork::reshape method, convert the model with the additional Model Optimizer key
[ EXPERIMENTAL ] Setting a new batch dimension value with the
The meaning of a model batch may vary depending on choices you made during the model designing. The
InferenceEngine::CNNNetwork::setBatchSize method deduces index of batch dimension relying only on the input rank. This method does not work for models with a non-zero index batch placement or models with inputs without a batch dimension.
Batch-setting algorithm does not involve shape inference mechanism. Batch of input and output shapes for all layers is set to a new batch value without layer validation. It may cause both positive and negative side effects.
Due to the limitations described above, the current method is recommended for simple image processing models only.
Practically, some models are not ready to be resized. In this case, a new input shape cannot be set with the Model Optimizer or the
Operation semantics may impose restrictions on input shapes of the operation. Shape collision during shape propagation may be a sign that a new shape does not satisfy the restrictions. Changing the model input shape may result in intermediate operations shape collision.
Examples of such operations:
Reshapeoperation with a hard-coded output shape value
MatMuloperation with the
Constsecond input cannot be resized by spatial dimensions due to operation semantics
Model structure and logic should not change significantly after resizing.
Poolingoperation with the fixed kernel size [H, W]. During spatial reshape, having the input of the shape [N, C, H1, W1], Pooling with the fixed kernel size [H, W] returns the output of the shape [N, C, H2, W2], where H2 and W2 are commonly not equal to
1. It breaks the classification model structure. For example, publicly available Inception family models from TensorFlow* have this issue.
pipeline.configfile. For details, refer to the Tensorflow Object Detection API models resizing techniques.
The primary method of the feature is
InferenceEngine::CNNNetwork::reshape. It gets new input shapes and propagates it from input to output for all intermediates layers of the given network. The method takes
InferenceEngine::ICNNNetwork::InputShapes - a map of pairs: name of input data and its dimension.
The algorithm for resizing network is the following:
1) Collect the map of input names and shapes from Intermediate Representation (IR) using helper method
2) Set new input shapes
3) Call reshape
Here is a code example:
Shape Inference feature is used in Smart classroom sample.
Inference Engine provides a special mechanism that allows to add the support of shape inference for custom operations. To enable shape inference for custom operations, create the library with custom nGraph operations and load it to the Inference Engine.
Each nGraph operation must implement two methods:
nGraph provides an operation sets (opsets) mechanism for operation versioning. Different opsets distinguish between different versions of one operation.
IMPORTANT: In your library, implement the
InferenceEngine::IExtension::getOpSets()method that returns opsets with custom operations.
When specifying opset names, follow the rules below:
versionattribute specifying the opset.
opset1is the name of default operations set. Operations from the default opset cannot be redefined.
Use a custom opset to create a new operation or extend functionality of an existing operation from another opset.
As already mentioned above, the new approach to shape inference suggests a creation of a custom nGraph operation that contains a special method for shape inference. However, the old approach with the
InferenceEngine::IShapeInferExtension method still works for already existing custom layers. Custom Shape Inference functions are registered by calling
InferenceEngine::ICNNNetwork::AddExtension with the implemented
InferenceEngine::IShapeInferExtension method, which is a holder of custom implementations. The holder requires to implement two key methods:
InferenceEngine::IShapeInferExtension::getShapeInferImpl- Returns custom shape inference implementation for the given type.
InferenceEngine::IShapeInferExtension::getShapeInferTypes- Provides all custom types.
Custom shape inference implementation is represented by the
It is not possible to overwrite built-in shape inference functions. Custom type must be different from the supported ones. Extensibility mechanism of the Shape Inference feature is demonstrated in the Hello Shape Infer SSD sample.