Public Member Functions
ie_api.InferRequest Class Reference

This class provides an interface to infer requests of ExecutableNetwork and serves to handle infer requests execution and to set and get output data. More...

Public Member Functions

def __init__ (self)
 There is no explicit class constructor. More...

 
def set_completion_callback (self, py_callback, py_data=None)
 Description: Sets a callback function that is called on success or failure of an asynchronous request. More...

 
def infer (self, inputs=None)
 Starts synchronous inference of the infer request and fill outputs array. More...

 
def async_infer (self, inputs=None)
 Starts asynchronous inference of the infer request and fill outputs array. More...

 
def wait (self, timeout=None)
 Waits for the result to become available. More...

 
def get_perf_counts (self)
 Queries performance measures per layer to get feedback of what is the most time consuming layer. More...

 
 
def set_batch (self, size)
 Sets new batch size for certain infer request when dynamic batching is enabled in executable network that created this request. More...

 

Class Attributes

inputs
outputs
latency
Current infer request inference time in milliseconds.

Detailed Description

This class provides an interface to infer requests of ExecutableNetwork and serves to handle infer requests execution and to set and get output data.

Constructor & Destructor Documentation

§ __init__()

def ie_api.InferRequest.__init__ (   self)

There is no explicit class constructor.

To make a valid InferRequest instance, use load_network() method of the IECore class with specified number of requests to get ExecutableNetwork instance which stores infer requests.

Member Function Documentation

§ async_infer()

def ie_api.InferRequest.async_infer (   self,
  inputs = None 
)

Starts asynchronous inference of the infer request and fill outputs array.

Parameters
inputsA dictionary that maps input layer names to numpy.ndarray objects of proper shape with input data for the layer
Returns
: None

Usage example:

exec_net = plugin.load(network=net, num_requests=2)
exec_net.requests[0].async_infer({input_blob: image})
request_status = exec_net.requests[0].wait()
res = exec_net.requests[0].outputs['prob']

§ get_perf_counts()

def ie_api.InferRequest.get_perf_counts (   self)

Queries performance measures per layer to get feedback of what is the most time consuming layer.

NOTE: Performance counters data and format depends on the plugin

Returns
Dictionary containing per-layer execution information.

Usage example:

exec_net = plugin.load(network=net, num_requests=2)
exec_net.requests[0].infer({input_blob: image})
exec_net.requests[0].get_perf_counts()
{'Conv2D': {'exec_type': 'jit_avx2_1x1',
'real_time': 154,
'cpu_time': 154,
'status': 'EXECUTED',
'layer_type': 'Convolution'},
'Relu6': {'exec_type': 'undef',
'real_time': 0,
'cpu_time': 0,
'status': 'NOT_RUN',
'layer_type': 'Clamp'}
...
}

§ infer()

def ie_api.InferRequest.infer (   self,
  inputs = None 
)

Starts synchronous inference of the infer request and fill outputs array.

Parameters
inputsA dictionary that maps input layer names to numpy.ndarray objects of proper shape with input data for the layer
Returns
None

Usage example:

exec_net = plugin.load(network=net, num_requests=2)
exec_net.requests[0].infer({input_blob: image})
res = exec_net.requests[0].outputs['prob']
np.flip(np.sort(np.squeeze(res)),0)
array([4.85416055e-01, 1.70385033e-01, 1.21873841e-01, 1.18894853e-01,
5.45198545e-02, 2.44456064e-02, 5.41366823e-03, 3.42589128e-03,
2.26027006e-03, 2.12283316e-03 ...])

§ set_batch()

def ie_api.InferRequest.set_batch (   self,
  size 
)

Sets new batch size for certain infer request when dynamic batching is enabled in executable network that created this request.

NOTE: Support of dynamic batch size depends on the target plugin.

Parameters
sizeNew batch size to be used by all the following inference calls for this request
Returns
None

Usage example:

net = IENetwork(model=path_to_xml_file, weights=path_to_bin_file)
# Set max batch size
net.batch = 10
plugin.set_config({"DYN_BATCH_ENABLED": "YES"})
exec_net = plugin.load(network=net)
# Set batch size for certain network.
# NOTE: Input data shape will not be changed, but will be used partially in inference which increases performance
exec_net.requests[0].set_batch(2)

§ set_completion_callback()

def ie_api.InferRequest.set_completion_callback (   self,
  py_callback,
  py_data = None 
)

Description: Sets a callback function that is called on success or failure of an asynchronous request.

Parameters
py_callback- Any defined or lambda function
py_data- Data that is passed to the callback function
Returns
None

Usage example:

callback = lambda status, py_data: print("Request with id {} finished with status {}".format(py_data, status))
net = IENetwork("./model.xml", "./model.bin")
ie = IECore()
exec_net = ie.load_network(net, "CPU", num_requests=4)
for id, req in enumerate(exec_net.requests):
req.set_completion_callback(py_callback=callback, py_data=id)
for req in exec_net.requests:
req.async_infer({"data": img})

§ wait()

def ie_api.InferRequest.wait (   self,
  timeout = None 
)

Waits for the result to become available.

Blocks until specified timeout elapses or the result becomes available, whichever comes first. NOTE: There are special values of the timeout parameter:

  • 0 - Immediately returns the inference status. It does not block or interrupt execution. To find statuses meaning, please refer to InferenceEngine::StatusCode in Inference Engine C++ documentation
  • -1 - Waits until inference result becomes available (default value)
Parameters
timeoutTime to wait in milliseconds or special (0, -1) cases described above. If not specified, timeout value is set to -1 by default.
Returns
Request status code.

Usage example: See async_infer() method of the the InferRequest class.