Model Optimizer cannot generate an Intermediate Representation from unsupported TensorFlow* operations, as is the case with some custom layers. However, you can still successfully create an Intermediate Representation if you offload the unsupported operations to TensorFlow for computation. To support this scenario, you must build Inference Engine custom layer with the TensorFlow C++ runtime. The layer that offloads computations to TensorFlow has type
TFCustomSubgraphCall in the Inference Engine Intermediate Representation and contains a TensorFlow graph protobuf string describing nodes being offloaded.
- You can only offload operations to TensorFlow from a Linux* OS computer.
- The custom layer supports inference only on a CPU, not on Intel® Integrated Graphics or on Intel® FPGA.
- The Inference Engine uses NCHW layout for tensors, but TensorFlow uses usually NHWC. Model Optimizer performs conversion between these layouts to correctly infer the model. Model Optimizer adds transpose operations to convert sub-graph 4D input tensors from NCHW layout to NHWC and vice versa for the output nodes. These operations are embedded in the protobuf string that describes the TensorFlow sub-graph in the Intermediate Representation
.xml file. Sometimes this approach fails. For example, the offload convolution to TensorFlow fails, because the convolution layout weights in TensorFlow do not correspond to the layout weights in the Inference Engine. However, offloading convolution nodes plus nodes with weights succeeds, because the node with weights is a part of offloaded sub-graph, so there are no transposes for the weights tensor. The successful nodes are usually of type
Model Optimizer CLI options
Three command-line options are available to offload part of the inference to TensorFlow.
NOTE: Use the command-line options on the line with the command:
python3 mo.py --input_model model-file.pb
- Use node name patterns to offload a sub-graph of operations, using the command-line option:
This option uses a comma-separated list of regular expressions to match node names. This offload has two primary characteristics:
- All nodes that match a specific regular expression are merged into a single Inference Engine node that TensorFlow* executes.
- All patterns are applied independently, which means two nodes that match two different patterns are not merged into one node. For example, the option
--tensorflow_subgraph_patterns "Scope_1/.*,Scope_2.*" is merged with all nodes whose names start from
Scope_1/ to a new node, and all nodes whose names start from
Scope_2 are merged to a different node.
- Offload specific types of operations, using the command-line option:
This option specifies a comma-separated list of regular expressions to match node types. This offload has a primary characteristic: all nodes that match a specific regular expression are merged into a single Inference Engine node that TensorFlow executes. For example, the following command offloads all operations of type
BiasAdd to Tensorflow:
- Offload all unsupported operations automatically, using the command-line option:
With this option, the Model Optimizer analyzes a network graph and finds unsupported operations. Model Optimizer finds and offloads the connected sub-graphs of unsupported operations. The unsupported operations are offloaded to TensorFlow.
You can use all three options by issuing the commands in this order:
python3 mo.py --input_model model-file.pb --tensorflow_subgraph_patterns
python3 mo.py --input_model model-file.pb --tensorflow_operation_patterns
python3 mo.py --input_model model-file.pb --offload_unsupported_operations_to_tf
How to Build a Custom Layer to Offload Computations to TensorFlow*
Your system should meet the requirements needed to build the TensorFlow from the sources described in the following page. The Tensorflow version 1.2, 1.3.1, 1.4, 1.5, 1.6 and 1.7 are supported.
NOTE: You need to perform this step only once.
- Clone the TensorFlow Git repository.
- Set the environment variable
TF_ROOT_DIR to point to the cloned directory:
- Set the OpenVINO toolkit environment variables running the
- Build an Inference Engine layer with TensorFlow runtime. This might take about 20 minutes:
- A shared library is generated:
This library is the Inference Engine custom layer, which is used to offload inference to TensorFlow.
How to Run a Model with Operations Offloaded to TensorFlow*
./extensibility_sample -i <path_to_image_file> -m <path_to_IR.xml> -d CPU -l <path_to_libtensorflow_call_layer.so>