Data Structures | Public Types | Public Member Functions | Protected Types | Protected Member Functions | Protected Attributes | Friends
InferenceEngine::AsyncInferRequestThreadSafeDefault Class Reference

Base class with default implementation of asynchronous multi staged inference request. To customize pipeline stages derived class should change the content of AsyncInferRequestThreadSafeDefault::_pipeline member container. It consists of pairs of tasks and executors which will run the task. The class is recommended to be used by plugins as a base class for asynchronous inference request implementation. More...

#include <ie_infer_async_request_thread_safe_default.hpp>

Inheritance diagram for InferenceEngine::AsyncInferRequestThreadSafeDefault:
InferenceEngine::IInferRequestInternal

Public Types

using Ptr = std::shared_ptr< AsyncInferRequestThreadSafeDefault >
 A shared pointer to AsyncInferRequestThreadSafeDefault.
 
- Public Types inherited from InferenceEngine::IInferRequestInternal
using Ptr = std::shared_ptr< IInferRequestInternal >
 A shared pointer to a IInferRequestInternal interface.
 
using Callback = std::function< void(std::exception_ptr)>
 Alias for callback type.
 

Public Member Functions

 AsyncInferRequestThreadSafeDefault (const IInferRequestInternal::Ptr &request, const ITaskExecutor::Ptr &taskExecutor, const ITaskExecutor::Ptr &callbackExecutor)
 Wraps a IInferRequestInternal::Ptr implementation and constructs a AsyncInferRequestThreadSafeDefault::_pipeline where taskExecutor is used to run IInferRequestInternal::Infer asynchronously. More...
 
 ~AsyncInferRequestThreadSafeDefault ()
 Destroys the object, stops AsyncInferRequestThreadSafeDefault::_pipeline and waits for a finish.
 
StatusCode Wait (int64_t millis_timeout) override
 Waits for completion of all pipeline stages If the pipeline raises an exception it will be rethrown here. More...
 
void StartAsync () override
 Start inference of specified input(s) in asynchronous mode. More...
 
void Infer () override
 Infers specified input(s) in synchronous mode. More...
 
std::map< std::string, InferenceEngineProfileInfoGetPerformanceCounts () const override
 Queries performance measures per layer to get feedback of what is the most time consuming layer. Note: not all plugins may provide meaningful data. More...
 
void SetBlob (const std::string &name, const Blob::Ptr &data) override
 Set input/output data to infer. More...
 
void SetBlob (const std::string &name, const Blob::Ptr &data, const PreProcessInfo &info) override
 Sets pre-process for input data. More...
 
Blob::Ptr GetBlob (const std::string &name) override
 Get input/output data to infer. More...
 
const PreProcessInfoGetPreProcess (const std::string &name) const override
 Gets pre-process for input data. More...
 
void SetBatch (int batch) override
 Sets new batch size when dynamic batching is enabled in executable network that created this request. More...
 
void SetCallback (Callback callback) override
 Set callback function which will be called on success or failure of asynchronous request. More...
 
std::vector< std::shared_ptr< InferenceEngine::IVariableStateInternal > > QueryState () override
 Queries memory states. More...
 
void ThrowIfCanceled () const
 
void Cancel () override
 Cancel current inference request execution.
 
- Public Member Functions inherited from InferenceEngine::IInferRequestInternal
 IInferRequestInternal (const InputsDataMap &networkInputs, const OutputsDataMap &networkOutputs)
 Constructs a new instance. More...
 
virtual void InferImpl ()
 The minimal infer function to be implemented by plugins. It infers specified input(s) in synchronous mode. More...
 
virtual void StartAsyncImpl ()
 The minimal asynchronous inference function to be implemented by plugins. It starts inference of specified input(s) in asynchronous mode. More...
 
void checkBlob (const Blob::Ptr &blob, const std::string &name, bool isInput, const SizeVector &refDims={}) const
 Check that blob is valid. Throws an exception if it's not. More...
 
virtual void checkBlobs ()
 Check that all of the blobs is valid. Throws an exception if it's not.
 
void setPointerToExecutableNetworkInternal (const std::shared_ptr< IExecutableNetworkInternal > &exeNetwork)
 Sets the pointer to executable network internal. More...
 
The method will be removed void * GetUserData () noexcept
 Gets the pointer to userData. More...
 
The method will be removed void SetUserData (void *userData) noexcept
 Sets the pointer to userData. More...
 

Protected Types

using Stage = std::pair< ITaskExecutor::Ptr, Task >
 Each pipeline stage is a Task that is executed by specified ITaskExecutor implementation.
 
using Pipeline = std::vector< Stage >
 Pipeline is vector of stages.
 

Protected Member Functions

void CheckState () const
 Throws exception if inference request is busy or canceled.
 
void RunFirstStage (const Pipeline::iterator itBeginStage, const Pipeline::iterator itEndStage, const ITaskExecutor::Ptr callbackExecutor={})
 Creates and run the first stage task. If destructor was not called add a new std::future to the AsyncInferRequestThreadSafeDefault::_futures list that would be used to wait AsyncInferRequestThreadSafeDefault::_pipeline finish. More...
 
void StopAndWait ()
 Forbids pipeline start and wait for all started pipelines. More...
 
virtual void StartAsync_ThreadUnsafe ()
 Starts an asynchronous pipeline thread unsafe. More...
 
virtual void Infer_ThreadUnsafe ()
 Performs inference of pipeline in syncronous mode. More...
 
void InferUsingAsync ()
 Implements Infer() using StartAsync() and Wait()
 
- Protected Member Functions inherited from InferenceEngine::IInferRequestInternal
 ~IInferRequestInternal ()
 Destroys the object.
 
void execDataPreprocessing (InferenceEngine::BlobMap &preprocessedBlobs, bool serial=false)
 Checks and executes input data pre-processing if needed. More...
 
bool findInputAndOutputBlobByName (const std::string &name, InputInfo::Ptr &foundInput, DataPtr &foundOutput) const
 Helper function to find input or output blob by name. More...
 
bool preProcessingRequired (const InputInfo::Ptr &info, const Blob::Ptr &userBlob, const Blob::Ptr &deviceBlob=nullptr)
 Checks whether pre-processing step is required for a given input. More...
 
void addInputPreProcessingFor (const std::string &name, Blob::Ptr const &from, const Blob::Ptr &to)
 

Protected Attributes

ITaskExecutor::Ptr _requestExecutor
 Used to run inference CPU tasks.
 
ITaskExecutor::Ptr _callbackExecutor
 Used to run post inference callback in asynchronous pipline.
 
ITaskExecutor::Ptr _syncCallbackExecutor
 Used to run post inference callback in synchronous pipline.
 
Pipeline _pipeline
 Pipeline variable that should be filled by inherited class.
 
Pipeline _syncPipeline
 Synchronous pipeline variable that should be filled by inherited class.
 
- Protected Attributes inherited from InferenceEngine::IInferRequestInternal
InferenceEngine::InputsDataMap _networkInputs
 Holds information about network inputs info.
 
InferenceEngine::OutputsDataMap _networkOutputs
 Holds information about network outputs data.
 
InferenceEngine::BlobMap _inputs
 A map of user passed blobs for network inputs.
 
InferenceEngine::BlobMap _deviceInputs
 A map of actual network inputs, in plugin specific format.
 
InferenceEngine::BlobMap _outputs
 A map of user passed blobs for network outputs.
 
std::map< std::string, PreProcessDataPtr > _preProcData
 A map of pre-process data per input.
 
int m_curBatch = -1
 Current batch value used in dynamic batching.
 
std::shared_ptr< IExecutableNetworkInternal_exeNetwork
 A shared pointer to IInferRequestInternal. More...
 
Callback _callback
 A callback.
 

Friends

struct DisableCallbackGuard
 

Detailed Description

Base class with default implementation of asynchronous multi staged inference request. To customize pipeline stages derived class should change the content of AsyncInferRequestThreadSafeDefault::_pipeline member container. It consists of pairs of tasks and executors which will run the task. The class is recommended to be used by plugins as a base class for asynchronous inference request implementation.

Note
To synchronize derived context with stages derived class should call AsyncInferRequestThreadSafeDefault::StopAndWait() function in destructor.
Example
Here is an example of asynchronous inference request implementation for some accelerator device. It uses 5 different executors to run different stages of a synchronous inference request.
// Inherits from AsyncInferRequestThreadSafeDefault
class AcceleratorAsyncInferRequest : public AsyncInferRequestThreadSafeDefault {
// Store the pointer to the synchronous request and five executors
AcceleratorAsyncInferRequest(const AcceleratorSyncRequest::Ptr& syncRequest,
const ITaskExecutor::Ptr& preprocessExecutor,
const ITaskExecutor::Ptr& writeToDeviceExecutor,
const ITaskExecutor::Ptr& runOnDeviceExecutor,
const ITaskExecutor::Ptr& readFromDeviceExecutor,
const ITaskExecutor::Ptr& postProcessExecutor) :
AsyncInferRequestThreadSafeDefault(syncRequest, nullptr, nullptr),
_accSyncRequest{syncRequest},
_preprocessExecutor{preprocessExecutor},
_writeToDeviceExecutor{writeToDeviceExecutor},
_runOnDeviceExecutor{runOnDeviceExecutor},
_readFromDeviceExecutor{readFromDeviceExecutor},
_postProcessExecutor{postProcessExecutor}
{
// Five pipeline stages of synchronous infer request are run by different executors
{ _preprocessExecutor , [this] {
_accSyncRequest->Preprocess();
}},
{ _writeToDeviceExecutor , [this] {
_accSyncRequest->WriteToDevice();
}},
{ _runOnDeviceExecutor , [this] {
_accSyncRequest->RunOnDevice();
}},
{ _readFromDeviceExecutor , [this] {
_accSyncRequest->ReadFromDevice();
}},
{ _postProcessExecutor , [this] {
_accSyncRequest->PostProcess();
}},
};
}
// As all stages use _accSyncRequest member we should wait for all stages tasks before the destructor destroy this member.
~AcceleratorAsyncInferRequest() {
}
AcceleratorSyncRequest::Ptr _accSyncRequest;
ITaskExecutor::Ptr _preprocessExecutor, _writeToDeviceExecutor, _runOnDeviceExecutor, _readFromDeviceExecutor, _postProcessExecutor;
};
AsyncInferRequestThreadSafeDefault(const IInferRequestInternal::Ptr &request, const ITaskExecutor::Ptr &taskExecutor, const ITaskExecutor::Ptr &callbackExecutor)
Wraps a IInferRequestInternal::Ptr implementation and constructs a AsyncInferRequestThreadSafeDefault...
Definition: ie_infer_async_request_thread_safe_default.hpp:140
Pipeline _pipeline
Pipeline variable that should be filled by inherited class.
Definition: ie_infer_async_request_thread_safe_default.hpp:324
void StopAndWait()
Forbids pipeline start and wait for all started pipelines.
Definition: ie_infer_async_request_thread_safe_default.hpp:299
std::shared_ptr< ITaskExecutor > Ptr
Definition: ie_itask_executor.hpp:51

Constructor & Destructor Documentation

◆ AsyncInferRequestThreadSafeDefault()

InferenceEngine::AsyncInferRequestThreadSafeDefault::AsyncInferRequestThreadSafeDefault ( const IInferRequestInternal::Ptr request,
const ITaskExecutor::Ptr taskExecutor,
const ITaskExecutor::Ptr callbackExecutor 
)
inline

Wraps a IInferRequestInternal::Ptr implementation and constructs a AsyncInferRequestThreadSafeDefault::_pipeline where taskExecutor is used to run IInferRequestInternal::Infer asynchronously.

Parameters
[in]requestThe synchronous request
[in]taskExecutorThe task executor
[in]callbackExecutorThe callback executor

Member Function Documentation

◆ GetBlob()

Blob::Ptr InferenceEngine::AsyncInferRequestThreadSafeDefault::GetBlob ( const std::string &  name)
inlineoverridevirtual

Get input/output data to infer.

Note
Memory allocation doesn't happen
Parameters
name- a name of input or output blob.
data- a reference to input or output blob. The type of Blob must correspond to the network input precision and size.

Implements InferenceEngine::IInferRequestInternal.

◆ GetPerformanceCounts()

std::map<std::string, InferenceEngineProfileInfo> InferenceEngine::AsyncInferRequestThreadSafeDefault::GetPerformanceCounts ( ) const
inlineoverridevirtual

Queries performance measures per layer to get feedback of what is the most time consuming layer. Note: not all plugins may provide meaningful data.

Returns
- a map of layer names to profiling information for that layer.

Implements InferenceEngine::IInferRequestInternal.

◆ GetPreProcess()

const PreProcessInfo& InferenceEngine::AsyncInferRequestThreadSafeDefault::GetPreProcess ( const std::string &  name) const
inlineoverridevirtual

Gets pre-process for input data.

Parameters
nameName of input blob.
infopointer to a pointer to PreProcessInfo structure

Implements InferenceEngine::IInferRequestInternal.

◆ Infer()

void InferenceEngine::AsyncInferRequestThreadSafeDefault::Infer ( )
inlineoverridevirtual

Infers specified input(s) in synchronous mode.

Note
blocks all method of InferRequest while request is ongoing (running or waiting in queue)

Implements InferenceEngine::IInferRequestInternal.

◆ Infer_ThreadUnsafe()

virtual void InferenceEngine::AsyncInferRequestThreadSafeDefault::Infer_ThreadUnsafe ( )
inlineprotectedvirtual

Performs inference of pipeline in syncronous mode.

Note
Used by Infer which ensures thread-safety and calls this method after.

◆ QueryState()

std::vector<std::shared_ptr<InferenceEngine::IVariableStateInternal> > InferenceEngine::AsyncInferRequestThreadSafeDefault::QueryState ( )
inlineoverridevirtual

Queries memory states.

Returns
Returns memory states

Implements InferenceEngine::IInferRequestInternal.

◆ RunFirstStage()

void InferenceEngine::AsyncInferRequestThreadSafeDefault::RunFirstStage ( const Pipeline::iterator  itBeginStage,
const Pipeline::iterator  itEndStage,
const ITaskExecutor::Ptr  callbackExecutor = {} 
)
inlineprotected

Creates and run the first stage task. If destructor was not called add a new std::future to the AsyncInferRequestThreadSafeDefault::_futures list that would be used to wait AsyncInferRequestThreadSafeDefault::_pipeline finish.

Parameters
[in]itBeginStageIterator to begin of pipeline
[in]itEndStageEnd pipeline iterator
[in]callbackExecutorFinal or error stage executor

◆ SetBatch()

void InferenceEngine::AsyncInferRequestThreadSafeDefault::SetBatch ( int  batch)
inlineoverridevirtual

Sets new batch size when dynamic batching is enabled in executable network that created this request.

Parameters
batch- new batch size to be used by all the following inference calls for this request.

Implements InferenceEngine::IInferRequestInternal.

◆ SetBlob() [1/2]

void InferenceEngine::AsyncInferRequestThreadSafeDefault::SetBlob ( const std::string &  name,
const Blob::Ptr data 
)
inlineoverridevirtual

Set input/output data to infer.

Note
Memory allocation doesn't happen
Parameters
name- a name of input or output blob.
data- a reference to input or output blob. The type of Blob must correspond to the network input precision and size.

Implements InferenceEngine::IInferRequestInternal.

◆ SetBlob() [2/2]

void InferenceEngine::AsyncInferRequestThreadSafeDefault::SetBlob ( const std::string &  name,
const Blob::Ptr data,
const PreProcessInfo info 
)
inlineoverridevirtual

Sets pre-process for input data.

Parameters
nameName of input blob.
data- a reference to input or output blob. The type of Blob must correspond to the network input precision and size.
infoPreprocess info for blob.

Implements InferenceEngine::IInferRequestInternal.

◆ SetCallback()

void InferenceEngine::AsyncInferRequestThreadSafeDefault::SetCallback ( Callback  callback)
inlineoverridevirtual

Set callback function which will be called on success or failure of asynchronous request.

Parameters
callback- function to be called with the following description:

Implements InferenceEngine::IInferRequestInternal.

◆ StartAsync()

void InferenceEngine::AsyncInferRequestThreadSafeDefault::StartAsync ( )
inlineoverridevirtual

Start inference of specified input(s) in asynchronous mode.

Note
The method returns immediately. Inference starts also immediately.

Implements InferenceEngine::IInferRequestInternal.

◆ StartAsync_ThreadUnsafe()

virtual void InferenceEngine::AsyncInferRequestThreadSafeDefault::StartAsync_ThreadUnsafe ( )
inlineprotectedvirtual

Starts an asynchronous pipeline thread unsafe.

Note
Used by StartAsync which ensures thread-safety and calls this method after.

◆ StopAndWait()

void InferenceEngine::AsyncInferRequestThreadSafeDefault::StopAndWait ( )
inlineprotected

Forbids pipeline start and wait for all started pipelines.

Note
Should be called in derived class destructor to wait for completion of usage of derived context captured by pipeline tasks

◆ Wait()

StatusCode InferenceEngine::AsyncInferRequestThreadSafeDefault::Wait ( int64_t  millis_timeout)
inlineoverridevirtual

Waits for completion of all pipeline stages If the pipeline raises an exception it will be rethrown here.

Parameters
millis_timeoutA timeout is ms to wait or special enum value of InferRequest::WaitMode
Returns
A status code

Implements InferenceEngine::IInferRequestInternal.


The documentation for this class was generated from the following file: