ie_api.InferRequest Class Reference

This class provides an interface to infer requests of ExecutableNetwork and serves to handle infer requests execution and to set and get output data. More...

Public Member Functions
def	__init__ (self)
	There is no explicit class constructor. More...
def	set_completion_callback (self, py_callback, py_data=None)
	Description: Sets a callback function that is called on success or failure of an asynchronous request. More...
def	infer (self, inputs=None)
	Starts synchronous inference of the infer request and fill outputs array. More...
def	async_infer (self, inputs=None)
	Starts asynchronous inference of the infer request and fill outputs array. More...
def	wait (self, timeout=None)
	Waits for the result to become available. More...
def	get_perf_counts (self)
	Queries performance measures per layer to get feedback of what is the most time consuming layer. More...
def	set_batch (self, size)
	Sets new batch size for certain infer request when dynamic batching is enabled in executable network that created this request. More...

Class Attributes

inputs

outputs

latency

Current infer request inference time in milliseconds.

Detailed Description

This class provides an interface to infer requests of ExecutableNetwork and serves to handle infer requests execution and to set and get output data.

Constructor & Destructor Documentation

__init__()

def ie_api.InferRequest.__init__

(

self

)

There is no explicit class constructor.

To make a valid InferRequest instance, use load_network() method of the IECore class with specified number of requests to get ExecutableNetwork instance which stores infer requests.

Member Function Documentation

async_infer()

def ie_api.InferRequest.async_infer	(		self,
			inputs = None
	)

Starts asynchronous inference of the infer request and fill outputs array.

Parameters

inputs

A dictionary that maps input layer names to numpy.ndarray objects of proper shape with input data for the layer

Returns

: None

Usage example:

exec_net = plugin.load(network=net, num_requests=2)
exec_net.requests[0].async_infer({input_blob: image})
request_status = exec_net.requests[0].wait()
res = exec_net.requests[0].outputs['prob']

get_perf_counts()

def ie_api.InferRequest.get_perf_counts

(

self

)

Queries performance measures per layer to get feedback of what is the most time consuming layer.

NOTE: Performance counters data and format depends on the plugin

Returns

Dictionary containing per-layer execution information.

Usage example:

exec_net = plugin.load(network=net, num_requests=2)
exec_net.requests[0].infer({input_blob: image})
exec_net.requests[0].get_perf_counts()
{'Conv2D': {'exec_type': 'jit_avx2_1x1',
            'real_time': 154,
            'cpu_time': 154,
            'status': 'EXECUTED',
            'layer_type': 'Convolution'},
 'Relu6':  {'exec_type': 'undef',
            'real_time': 0,
            'cpu_time': 0,
            'status': 'NOT_RUN',
            'layer_type': 'Clamp'}
 ...
}

infer()

def ie_api.InferRequest.infer	(		self,
			inputs = None
	)

Starts synchronous inference of the infer request and fill outputs array.

Parameters

inputs

A dictionary that maps input layer names to numpy.ndarray objects of proper shape with input data for the layer

Returns

None

Usage example:

exec_net = plugin.load(network=net, num_requests=2)
exec_net.requests[0].infer({input_blob: image})
res = exec_net.requests[0].outputs['prob']
np.flip(np.sort(np.squeeze(res)),0)
array([4.85416055e-01, 1.70385033e-01, 1.21873841e-01, 1.18894853e-01,
       5.45198545e-02, 2.44456064e-02, 5.41366823e-03, 3.42589128e-03,
       2.26027006e-03, 2.12283316e-03 ...])

set_batch()

def ie_api.InferRequest.set_batch	(		self,
			size
	)

Sets new batch size for certain infer request when dynamic batching is enabled in executable network that created this request.

NOTE: Support of dynamic batch size depends on the target plugin.

Parameters

size	New batch size to be used by all the following inference calls for this request

Returns

None

Usage example:

net = IENetwork(model=path_to_xml_file, weights=path_to_bin_file)
# Set max batch size
net.batch = 10
plugin.set_config({"DYN_BATCH_ENABLED": "YES"})
exec_net = plugin.load(network=net)
# Set batch size for certain network.
# NOTE: Input data shape will not be changed, but will be used partially in inference which increases performance
exec_net.requests[0].set_batch(2)

set_completion_callback()

def ie_api.InferRequest.set_completion_callback	(		self,
			py_callback,
			py_data = None
	)

Description: Sets a callback function that is called on success or failure of an asynchronous request.

Parameters

py_callback	- Any defined or lambda function
py_data	- Data that is passed to the callback function

Returns

None

Usage example:

callback = lambda status, py_data: print("Request with id {} finished with status {}".format(py_data, status))
net = IENetwork("./model.xml", "./model.bin")
ie = IECore()
exec_net = ie.load_network(net, "CPU", num_requests=4)
for id, req in enumerate(exec_net.requests):
    req.set_completion_callback(py_callback=callback, py_data=id)

for req in exec_net.requests:
    req.async_infer({"data": img})

wait()

def ie_api.InferRequest.wait	(		self,
			timeout = None
	)

Waits for the result to become available.

Blocks until specified timeout elapses or the result becomes available, whichever comes first. NOTE: There are special values of the timeout parameter:

• 0 - Immediately returns the inference status. It does not block or interrupt execution. To find statuses meaning, please refer to InferenceEngine::StatusCode in Inference Engine C++ documentation
• -1 - Waits until inference result becomes available (default value)

Parameters

timeout

Time to wait in milliseconds or special (0, -1) cases described above. If not specified, timeout value is set to -1 by default.

Returns

Request status code.

Usage example: See async_infer() method of the the InferRequest class.