Offloading Sub-Graph Inference to TensorFlow*

Model Optimizer cannot generate an Intermediate Representation from unsupported TensorFlow* operations, as is the case with some custom layers. However, you can still successfully create an Intermediate Representation if you offload the unsupported operations to TensorFlow for computation. To support this scenario, you must build Inference Engine custom layer with the TensorFlow C++ runtime. The layer that offloads computations to TensorFlow has type TFCustomSubgraphCall in the Inference Engine Intermediate Representation and contains a TensorFlow graph protobuf string describing nodes being offloaded.


Model Optimizer CLI options

Three command-line options are available to offload part of the inference to TensorFlow.

NOTE: Use the command-line options on the line with the command:

python3 --input_model model-file.pb

You can use all three options by issuing the commands in this order:

python3 --input_model model-file.pb --tensorflow_subgraph_patterns
python3 --input_model model-file.pb --tensorflow_operation_patterns
python3 --input_model model-file.pb --offload_unsupported_operations_to_tf

How to Build a Custom Layer to Offload Computations to TensorFlow*

Your system should meet the requirements needed to build the TensorFlow from the sources described in the following page. The Tensorflow version 1.2, 1.3.1, 1.4, 1.5, 1.6 and 1.7 are supported.

NOTE: You need to perform this step only once.

  1. Clone the TensorFlow Git repository.
  2. Set the environment variable TF_ROOT_DIR to point to the cloned directory:
  3. Set the OpenVINO toolkit environment variables running the script:
    source <INSTALL_DIR>/bin/
  4. Build an Inference Engine layer with TensorFlow runtime. This might take about 20 minutes:
  5. A shared library is generated:
    This library is the Inference Engine custom layer, which is used to offload inference to TensorFlow.

How to Run a Model with Operations Offloaded to TensorFlow*

  1. Compile extensibility_sample
  2. Run extensibility_sample:
    ./extensibility_sample -i <path_to_image_file> -m <path_to_IR.xml> -d CPU -l <>