Integrate the Inference Engine API with Your Application (Legacy)

This section provides a high-level description of the process of integrating the Inference Engine into your application. Refer to the Using Inference Engine Samples section for examples of using the Inference Engine in applications.

Using the Inference Engine API in Your Code

The core libinference_engine.so library implements loading and parsing a model IR, and triggers inference using a specified plugin. The core library has the following API:

Integration process consists of the following steps:

  1. Load a plugin by creating an instance of InferenceEngine::InferenceEnginePluginPtr. You can specify the plugin or let Inference Engine to choose it using InferenceEngine::PluginDispatcher. See the selectPlugin() function in the samples.
    InferenceEngine::PluginDispatcher dispatcher(pluginDirs);
    InferenceEngine::InferenceEnginePluginPtr enginePtr (dispatcher.getSuitablePlugin(TargetDevice::eCPU);
  2. Create an IR reader by creating an instance of InferenceEngine::CNNNetReader and read a model IR:
    auto netBuilder = new InferenceEngine::CNNNetReader();
    netBuilder->ReadNetwork("Model.xml");
    netBuilder->ReadWeights("Model.bin");
  3. Request information about inputs (an image and any other input data required), using the
    InferenceEngine::CNNNetReader::getNetwork() and InferenceEngine::CNNNetwork::getInputsInfo() methods. Set the input number format (precision) using InferenceEngine::InputInfo::setInputPrecision to match the input data format (precision). Allocate input blobs of the appropriate types and feed an image and the input data to the blobs:
    /** Taking information about all topology inputs **/
    InferenceEngine::InputsDataMap inputInfo(netBuilder.getNetwork().getInputsInfo());
    /** Stores all input blobs data **/
    InferenceEngine::BlobMap inputBlobs;
    /** Iterating over all input blobs **/
    for (auto & item : inputInfo) {
    /** Creating input blob **/
    item.second->setInputPrecision(Precision::U8);
    InferenceEngine::TBlob<unsigned char>::Ptr input;
    input = InferenceEngine::make_shared_blob<unsigned char, InferenceEngine::SizeVector>(Precision::U8, item.second->getDims());
    input->allocate();
    inputBlobs[item.first] = input;
    /** Fill input tensor with planes. First b channel, then g and r channels **/
    ...
    }
  4. Request information about outputs, using the InferenceEngine::CNNNetReader::getNetwork() and InferenceEngine::CNNNetwork::getOutputsInfo() methods. Allocate output blobs of the appropriate types:
    InferenceEngine::OutputsDataMap outputInfo(netBuilder.getNetwork().getOutputsInfo());
    InferenceEngine::BlobMap outputBlobs;
    for (auto & item : outputInfo) {
    InferenceEngine::TBlob<float>::Ptr output;
    output = InferenceEngine::make_shared_blob<float, InferenceEngine::SizeVector>(Precision::FP32, item.second->dims);
    output->allocate();
    outputBlobs[item.first] = output;
    }
  5. Load the model to the plugin using InferenceEngine::IInferencePlugin::LoadNetwork():
    InferenceEngine::StatusCode status = enginePtr->LoadNetwork(netBuilder.getNetwork(), &resp);
    if (status != InferenceEngine::OK) {
    throw std::logic_error(resp.msg);
    }
  6. Do inference by calling the InferenceEngine::IInferencePlugin::Infer method:
    enginePtr->Infer(inputBlobs, outputBlobs, &resp);
  7. Go over the output blobs and process the results.
    /** Pointer to the output blob **/
    const TBlob<float>::Ptr fOutput = std::dynamic_pointer_cast<TBlob<float>>(outputBlobs.begin()->second);
    /** fOutput->data()[] - accessing output blob data **/

Building Your Application

For details about building your application, refer to the CMake files for the sample applications. All samples reside in the samples directory in the Inference Engine installation directory.

Running Your Application

Before running compiled binary files, make sure your application can find the Inference Engine libraries. On Linux* operating systems, including Ubuntu* and CentOS*, the LD_LIBRARY_PATH environment variable is usually used to specify directories to be looked for libraries. You can update the LD_LIBRARY_PATH with paths to the directories in the Inference Engine installation directory where the libraries reside.

Add a path the directory containing the core and plugin libraries:

Add paths the directories containing the required third-party libraries:

Alternatively, you can use the following scripts that reside in the Inference Engine directory of the OpenVINO™ toolkit and Intel® Deep Learning Deployment Toolkit installation folders respectively:

To run compiled applications on Microsoft* Windows* OS, make sure that Microsoft* Visual C++ 2015 Redistributable and Intel® C++ Compiler 2017 Redistributable packages are installed and <INSTALL_DIR>/bin/intel64/Release/*.dll files are placed to the application folder or accessible via PATH% environment variable.