Model Optimizer cannot generate an Intermediate Representation from unsupported TensorFlow* operations, as is the case with some custom layers. However, you can still successfully create an Intermediate Representation if you offload the unsupported operations to TensorFlow for computation. To support this scenario, you must build Inference Engine custom layer with the TensorFlow C++ runtime. The layer that offloads computations to TensorFlow has type TFCustomSubgraphCall
in the Inference Engine Intermediate Representation and contains a TensorFlow graph protobuf string describing nodes being offloaded.
Limitations:
- You can only offload operations to TensorFlow from a Linux* OS computer.
- The custom layer supports inference only on a CPU, not on Intel® Integrated Graphics or on Intel® FPGA.
- The Inference Engine uses NCHW layout for tensors, but TensorFlow uses usually NHWC. Model Optimizer performs conversion between these layouts to correctly infer the model. Model Optimizer adds transpose operations to convert sub-graph 4D input tensors from NCHW layout to NHWC and vice versa for the output nodes. These operations are embedded in the protobuf string that describes the TensorFlow sub-graph in the Intermediate Representation
.xml
file. Sometimes this approach fails. For example, the offload convolution to TensorFlow fails, because the convolution layout weights in TensorFlow do not correspond to the layout weights in the Inference Engine. However, offloading convolution nodes plus nodes with weights succeeds, because the node with weights is a part of offloaded sub-graph, so there are no transposes for the weights tensor. The successful nodes are usually of type Const
.
Model Optimizer CLI Options
Two command-line options are available to offload part of the inference to TensorFlow.
NOTE: Use the command-line options on the line with the command:
python3 mo.py --input_model model-file.pb
- Use node name patterns to offload a sub-graph of operations, using the command-line option:
-- tensorflow_subgraph_patterns
This option uses a comma-separated list of regular expressions to match node names. This offload has two primary characteristics:
- All nodes that match a specific regular expression are merged into a single Inference Engine node that TensorFlow* executes.
- All patterns are applied independently, which means two nodes that match two different patterns are not merged into one node. For example, the option
--tensorflow_subgraph_patterns "Scope_1/.*,Scope_2.*"
is merged with all nodes whose names start from Scope_1/
to a new node, and all nodes whose names start from Scope_2
are merged to a different node.
- Offload specific types of operations, using the command-line option:
--tensorflow_operation_patterns
This option specifies a comma-separated list of regular expressions to match node types. This offload has a primary characteristic: all nodes that match a specific regular expression are merged into a single Inference Engine node that TensorFlow executes. For example, the following command offloads all operations of type Concat
, ConcatV2
, Add
, and BiasAdd
to Tensorflow: --tensorflow_operation_patterns "Concat.*,.*Add"
-
[ Unavailable ] Offload all unsupported operations automatically. The option
--tensorflow_operation_patterns
is not available anymore. Instead, use the command-line parameter --tensorflow_operation_patterns
and specify types of unsupported operations separated by commas. To determine the operations, run the Model Optimizer with the input model file and input/output node names (if necessary). If the Model Optimizer detects unsupported operations, it prints the following error message with the list of unsupported operations:
[ ERROR ] List of operations that cannot be converted to Inference Engine IR:
[ ERROR ] <type_of_unsupported_operation> (1)
[ ERROR ] <name_of_unsupported_operation>
[ ERROR ] Part of the nodes was not converted to IR. Stopped.
To offload unsupported operations to TensorFlow, run the following command-line option with the type of an unsupported operation specified: --tensorflow_operation_patterns <type_of_unsupported_operation>
You can use all two options by issuing the commands in this order:
python3 mo.py --input_model model-file.pb --tensorflow_subgraph_patterns
python3 mo.py --input_model model-file.pb --tensorflow_operation_patterns
How to Build a Custom Layer to Offload Computations to TensorFlow*
Your system should meet the requirements needed to build the TensorFlow from the sources described in the following page. The Tensorflow version 1.2, 1.3.1, 1.4, 1.5, 1.6 and 1.7 are supported.
NOTE: You need to perform this step only once.
- Clone the TensorFlow Git repository.
- Set the environment variable
TF_ROOT_DIR
to point to the cloned directory: export TF_ROOT_DIR=<TENSORFLOW_DIR>
- Set the OpenVINO toolkit environment variables running the
setupvars.sh
script: source <INSTALL_DIR>/bin/setupvars.sh
- Build an Inference Engine layer with TensorFlow runtime. This might take about 20 minutes:
./tf_call_ie_layer/build.sh.
- A shared library is generated:
$<TF_ROOT_DIR>/bazel-bin/tensorflow/cc/inference_engine_layer/libtensorflow_call_layer.so
This library is the Inference Engine custom layer, which is used to offload inference to TensorFlow.
How to Run a Model with Operations Offloaded to TensorFlow*
- Compile
extensibility_sample
- Run
extensibility_sample
: ./extensibility_sample -i <path_to_image_file> -m <path_to_IR.xml> -d CPU -l <path_to_libtensorflow_call_layer.so>