Inference Engine takes two kinds of model description as an input: Intermediate Representation (IR) and nGraph::Function objects. Both should have fixed input shapes to be successfully loaded to the Inference Engine. To feed input data of a shape that is different from the model input shape, resize the model first.
Setting a new input shape with the
InferenceEngine::CNNNetwork::reshape method updates input shapes and propagates them down to the outputs of the model through all intermediate layers.
Shape propagation for
InferenceEngine::CNNNetwork objects created from
nGraph::Function or IR of the version 10 works through the
nGraph shape inference mechanism.
InferenceEngine::CNNNetwork objects created from lower IR versions are considered deprecated and may be reshaped incorrectly or give unexpected results.
To keep the v10 IR resizable by the
InferenceEngine::CNNNetwork::reshape method, convert the model with the additional Model Optimizer key
Setting a new batch dimension value with the
The meaning of a model batch may vary depending on choices you made during the model designing. The
InferenceEngine::CNNNetwork::setBatchSize method deduces index of batch dimension relying only on the input rank. This method does not work for models with a non-zero index batch placement or models with inputs without a batch dimension.
Batch-setting algorithm does not involve shape inference mechanism. Batch of input and output shapes for all layers is set to a new batch value without layer validation. It may cause both positive and negative side effects.
Due to the limitations described above, the current method is recommended for simple image processing models only.
Practically, some models are not ready to be resized. In this case, a new input shape cannot be set with the Model Optimizer or the
Operation semantics may impose restrictions on input shapes of the operation. Shape collision during shape propagation may be a sign that a new shape does not satisfy the restrictions. Changing the model input shape may result in intermediate operations shape collision.
Examples of such operations:
Reshapeoperation with a hard-coded output shape value
MatMuloperation with the
Constsecond input cannot be resized by spatial dimensions due to operation semantics
Model structure and logic should not change significantly after resizing.
Poolingoperation with the fixed kernel size [H, W]. During spatial reshape, having the input of the shape [N, C, H1, W1], Pooling with the fixed kernel size [H, W] returns the output of the shape [N, C, H2, W2], where H2 and W2 are commonly not equal to
1. It breaks the classification model structure. For example, publicly available Inception family models from TensorFlow* have this issue.
pipeline.configfile. For details, refer to the Tensorflow Object Detection API models resizing techniques.
The primary method of the feature is
InferenceEngine::CNNNetwork::reshape. It gets new input shapes and propagates it from input to output for all intermediates layers of the given network. The method takes
InferenceEngine::ICNNNetwork::InputShapes - a map of pairs: name of input data and its dimension.
The algorithm for resizing network is the following:
1) Collect the map of input names and shapes from Intermediate Representation (IR) using helper method
2) Set new input shapes
3) Call reshape
Here is a code example:
Shape Inference feature is used in Smart classroom sample.
Inference Engine provides a special mechanism that allows to add the support of shape inference for custom operations. This mechanism is described in the Extensibility documentation.