Working with Open Model Zoo Models¶
This tutorial is also available as a Jupyter notebook that can be cloned directly from GitHub. See the installation guide for instructions to run this tutorial locally on Windows, Linux or macOS. To run without installing anything, click the launch binder button.
This tutorial shows how to download a model from the Open Model Zoo, convert it to OpenVINO’s IR format, show information about the model, and benchmark the model.
OpenVINO and Open Model Zoo Tools¶
The OpenVINO and Open Model Zoo tools are listed in the table below.
Tool |
Command |
Description |
---|---|---|
Model Downloader |
omz_download er |
Download models from Open Model Zoo |
Model Converter |
omz_converte r |
Convert Open Model Zoo models to OpenVINO’s IR format |
Info Dumper |
omz_info_dum per |
Print information about Open Model Zoo models |
Benchmark Tool |
benchmark_ap p |
Benchmark model performance by computing inference time |
Preparation¶
Model Name¶
Set model_name
to the name of the Open Model Zoo model to use in
this notebook. Refer to the list of
public
and
Intel
pre-trained models for a full list of models that can be used. Set the
model_name
to the model you want to use.
# model_name = "resnet-50-pytorch"
model_name = "mobilenet-v2-pytorch"
Imports¶
import json
import sys
from pathlib import Path
from IPython.display import Markdown, display
from openvino.inference_engine import IECore
sys.path.append("../utils")
from notebook_utils import DeviceNotFoundAlert, NotebookAlert
Settings and Configuration¶
Set the file and directory paths. By default, this demo notebook
downloads models from Open Model Zoo to a directory
open_model_zoo_models
in your $HOME
directory. On Windows, the
$HOME directory is usually c:\users\username
, on Linux
/home/username
. If you want to change the folder, change
base_model_dir
in the cell below.
You can change the following settings:
base_model_dir
: Models will be downloaded into theintel
andpublic
folders in this directory.omz_cache_dir
: Cache folder for Open Model Zoo. Specifying a cache directory is not required for Model Downloader and Model Converter, but it speeds up subsequent downloads.precision
: If specified, only models with this precision will be downloaded and converted.
base_model_dir = Path("~/open_model_zoo_models").expanduser()
omz_cache_dir = Path("~/open_model_zoo_cache").expanduser()
precision = "FP16"
# Check if an iGPU is available on this system to use with Benchmark App
ie = IECore()
gpu_available = "GPU" in ie.available_devices
print(
f"base_model_dir: {base_model_dir}, omz_cache_dir: {omz_cache_dir}, gpu_availble: {gpu_available}"
)
base_model_dir: /opt/home/k8sworker/open_model_zoo_models, omz_cache_dir: /opt/home/k8sworker/open_model_zoo_cache, gpu_availble: False
Download Model from Open Model Zoo¶
Specify, display and run the Model Downloader command to download the model
## Uncomment the next line to show omz_downloader's help which explains the command line options
# !omz_downloader --help
download_command = (
f"omz_downloader --name {model_name} --output_dir {base_model_dir} --cache_dir {omz_cache_dir}"
)
display(Markdown(f"Download command: `{download_command}`"))
display(Markdown(f"Downloading {model_name}..."))
! $download_command
Download command:
omz_downloader --name mobilenet-v2-pytorch --output_dir /opt/home/k8sworker/open_model_zoo_models --cache_dir /opt/home/k8sworker/open_model_zoo_cache
Downloading mobilenet-v2-pytorch…
################|| Downloading mobilenet-v2-pytorch ||################
========== Retrieving /opt/home/k8sworker/open_model_zoo_models/public/mobilenet-v2-pytorch/mobilenet_v2-b0353104.pth from the cache
Convert Model to OpenVINO IR format¶
Specify, display and run the Model Converter command to convert the
model to IR format. Model Conversion may take a while. The output of the
Model Converter command will be displayed. Conversion succeeded if the
last lines of the output include
[ SUCCESS ] Generated IR version 10 model.
For downloaded models
that are already in IR format, conversion will be skipped.
## Uncomment the next line to show omz_converter's help which explains the command line options
# !omz_converter --help
convert_command = f"omz_converter --name {model_name} --precisions {precision} --download_dir {base_model_dir} --output_dir {base_model_dir}"
display(Markdown(f"Convert command: `{convert_command}`"))
display(Markdown(f"Converting {model_name}..."))
! $convert_command
Convert command:
omz_converter --name mobilenet-v2-pytorch --precisions FP16 --download_dir /opt/home/k8sworker/open_model_zoo_models --output_dir /opt/home/k8sworker/open_model_zoo_models
Converting mobilenet-v2-pytorch…
========== Converting mobilenet-v2-pytorch to ONNX
Conversion to ONNX command: /opt/home/k8sworker/cibuilds/ov-notebook/OVNotebookOps-80/.workspace/scm/ov-notebook/.venv/bin/python -- /opt/home/k8sworker/cibuilds/ov-notebook/OVNotebookOps-80/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/open_model_zoo/model_tools/internal_scripts/pytorch_to_onnx.py --model-name=mobilenet_v2 --weights=/opt/home/k8sworker/open_model_zoo_models/public/mobilenet-v2-pytorch/mobilenet_v2-b0353104.pth --import-module=torchvision.models --input-shape=1,3,224,224 --output-file=/opt/home/k8sworker/open_model_zoo_models/public/mobilenet-v2-pytorch/mobilenet-v2.onnx --input-names=data --output-names=prob
ONNX check passed successfully.
========== Converting mobilenet-v2-pytorch to IR (FP16)
Conversion command: /opt/home/k8sworker/cibuilds/ov-notebook/OVNotebookOps-80/.workspace/scm/ov-notebook/.venv/bin/python -m mo --framework=onnx --data_type=FP16 --output_dir=/opt/home/k8sworker/open_model_zoo_models/public/mobilenet-v2-pytorch/FP16 --model_name=mobilenet-v2-pytorch --input=data '--mean_values=data[123.675,116.28,103.53]' '--scale_values=data[58.624,57.12,57.375]' --reverse_input_channels --output=prob --input_model=/opt/home/k8sworker/open_model_zoo_models/public/mobilenet-v2-pytorch/mobilenet-v2.onnx
Model Optimizer arguments:
Common parameters:
- Path to the Input Model: /opt/home/k8sworker/open_model_zoo_models/public/mobilenet-v2-pytorch/mobilenet-v2.onnx
- Path for generated IR: /opt/home/k8sworker/open_model_zoo_models/public/mobilenet-v2-pytorch/FP16
- IR output name: mobilenet-v2-pytorch
- Log level: ERROR
- Batch: Not specified, inherited from the model
- Input layers: data
- Output layers: prob
- Input shapes: Not specified, inherited from the model
- Mean values: data[123.675,116.28,103.53]
- Scale values: data[58.624,57.12,57.375]
- Scale factor: Not specified
- Precision of IR: FP16
- Enable fusing: True
- Enable grouped convolutions fusing: True
- Move mean values to preprocess section: None
- Reverse input channels: True
ONNX specific parameters:
- Inference Engine found in: /opt/home/k8sworker/cibuilds/ov-notebook/OVNotebookOps-80/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/openvino
Inference Engine version: 2021.4.2-3976-0943ed67223-refs/pull/539/head
Model Optimizer version: 2021.4.2-3976-0943ed67223-refs/pull/539/head
[ SUCCESS ] Generated IR version 10 model.
[ SUCCESS ] XML file: /opt/home/k8sworker/open_model_zoo_models/public/mobilenet-v2-pytorch/FP16/mobilenet-v2-pytorch.xml
[ SUCCESS ] BIN file: /opt/home/k8sworker/open_model_zoo_models/public/mobilenet-v2-pytorch/FP16/mobilenet-v2-pytorch.bin
[ SUCCESS ] Total execution time: 6.05 seconds.
[ SUCCESS ] Memory consumed: 121 MB.
It's been a while, check for a new version of Intel(R) Distribution of OpenVINO(TM) toolkit here https://software.intel.com/content/www/us/en/develop/tools/openvino-toolkit/download.html?cid=other&source=prod&campid=ww_2022_bu_IOTG_OpenVINO-2022-1&content=upg_all&medium=organic or on the GitHub*
Get Model Information¶
The Info Dumper prints the following information for Open Model Zoo models:
Model name
Description
Framework that was used to train the model
License URL
Precisions supported by the model
Subdirectory: the location of the downloaded model
Task type
This information can be shown by running
omz_info_dumper --name model_name
in a terminal. The information can
also be parsed and used in scripts.
In the next cell, we run Info Dumper and use json to load the information in a dictionary.
model_info_output = %sx omz_info_dumper --name $model_name
model_info = json.loads(model_info_output.get_nlstr())
if len(model_info) > 1:
NotebookAlert(
f"There are multiple IR files for the {model_name} model. The first model in the "
"omz_info_dumper output will be used for benchmarking. Change "
"`selected_model_info` in the cell below to select a different model from the list.",
"warning",
)
model_info
[{'name': 'mobilenet-v2-pytorch', 'composite_model_name': None, 'description': 'MobileNet V2 is image classification model pre-trained on ImageNet dataset. This is a PyTorch* implementation of MobileNetV2 architecture as described in the paper "Inverted Residuals and Linear Bottlenecks: Mobile Networks for Classification, Detection and Segmentation" <https://arxiv.org/abs/1801.04381>.nThe model input is a blob that consists of a single image of "1, 3, 224, 224" in "RGB" order.nThe model output is typical object classifier for the 1000 different classifications matching with those in the ImageNet database.', 'framework': 'pytorch', 'license_url': 'https://raw.githubusercontent.com/pytorch/vision/master/LICENSE', 'precisions': ['FP16', 'FP32'], 'quantization_output_precisions': ['FP16-INT8', 'FP32-INT8'], 'subdirectory': 'public/mobilenet-v2-pytorch', 'task_type': 'classification'}]
Having the model information in a JSON file allows us to extract the path to the model directory, and build the path to the IR file.
selected_model_info = model_info[0]
model_path = (
base_model_dir
/ Path(selected_model_info["subdirectory"])
/ Path(f"{precision}/{selected_model_info['name']}.xml")
)
print(model_path, "exists:", model_path.exists())
/opt/home/k8sworker/open_model_zoo_models/public/mobilenet-v2-pytorch/FP16/mobilenet-v2-pytorch.xml exists: True
Run Benchmark Tool¶
By default, Benchmark Tool runs inference for 60 seconds in asynchronous mode on CPU. It returns inference speed as latency (milliseconds per image) and throughput (frames per second) values.
## Uncomment the next line to show benchmark_app's help which explains the command line options
# !benchmark_app --help
benchmark_command = f"benchmark_app -m {model_path} -t 15"
display(Markdown(f"Benchmark command: `{benchmark_command}`"))
display(Markdown(f"Benchmarking {model_name} on CPU with async inference for 15 seconds..."))
! $benchmark_command
Benchmark command:
benchmark_app -m /opt/home/k8sworker/open_model_zoo_models/public/mobilenet-v2-pytorch/FP16/mobilenet-v2-pytorch.xml -t 15
Benchmarking mobilenet-v2-pytorch on CPU with async inference for 15 seconds…
[Step 1/11] Parsing and validating input arguments
[ WARNING ] -nstreams default value is determined automatically for a device. Although the automatic selection usually provides a reasonable performance, but it still may be non-optimal for some cases, for more information look at README.
[Step 2/11] Loading Inference Engine
[ INFO ] InferenceEngine:
API version............. 2021.4.2-3976-0943ed67223-refs/pull/539/head
[ INFO ] Device info
CPU
MKLDNNPlugin............ version 2.1
Build................... 2021.4.2-3976-0943ed67223-refs/pull/539/head
[Step 3/11] Setting device configuration
[ WARNING ] -nstreams default value is determined automatically for CPU device. Although the automatic selection usually provides a reasonable performance,but it still may be non-optimal for some cases, for more information look at README.
[Step 4/11] Reading network files
[ INFO ] Read network took 18.37 ms
[Step 5/11] Resizing network to match image sizes and given batch
[ INFO ] Network batch size: 1
[Step 6/11] Configuring input of the model
[ INFO ] Network input 'data' precision U8, dimensions (NCHW): 1 3 224 224
[ INFO ] Network output 'prob' precision FP32, dimensions (NC): 1 1000
[Step 7/11] Loading the model to the device
[ INFO ] Load network took 149.52 ms
[Step 8/11] Setting optimal runtime parameters
[Step 9/11] Creating infer requests and filling input blobs with images
[ WARNING ] No input files were given: all inputs will be filled with random values!
[ INFO ] Infer Request 0 filling
[ INFO ] Fill input 'data' with random values (image is expected)
[ INFO ] Infer Request 1 filling
[ INFO ] Fill input 'data' with random values (image is expected)
[ INFO ] Infer Request 2 filling
[ INFO ] Fill input 'data' with random values (image is expected)
[ INFO ] Infer Request 3 filling
[ INFO ] Fill input 'data' with random values (image is expected)
[ INFO ] Infer Request 4 filling
[ INFO ] Fill input 'data' with random values (image is expected)
[ INFO ] Infer Request 5 filling
[ INFO ] Fill input 'data' with random values (image is expected)
[Step 10/11] Measuring performance (Start inference asynchronously, 6 inference requests using 6 streams for CPU, limits: 15000 ms duration)
[ INFO ] First inference took 9.61 ms
[Step 11/11] Dumping statistics report
Count: 20082 iterations
Duration: 15008.62 ms
Latency: 4.49 ms
Throughput: 1338.03 FPS
Benchmark with Different Settings¶
benchmark_app
displays logging information that is not always
necessary. We parse the output with json and show a more compact result
The following cells show some examples of benchmark_app
with
different parameters. Some useful parameters are:
-d
Device to use for inference. For example: CPU, GPU, MULTI. Default: CPU-t
Time in number of seconds to run inference. Default: 60-api
Use asynchronous (async) or synchronous (sync) inference. Default: async-b
Batch size. Default: 1
Run ! benchmark_app --help
to get an overview of all possible
command line parameters.
In the next cell, we define a benchmark_model()
function that calls
benchmark_app
. This makes it easy to try different combinations. In
the cell below that, we display the available devices on the system.
NOTE: In this notebook we run benchmark_app for 15 seconds to give a quick indication of performance. For more accurate performance, we recommended running inference for at least one minute by setting the
t
parameter to 60 or higher, and runningbenchmark_app
in a terminal/command prompt after closing other applications. You can copy the benchmark command and paste it in a command prompt where you have activated theopenvino_env
environment.
def benchmark_model(model_xml, device="CPU", seconds=60, api="async", batch=1):
ie = IECore()
model_path = Path(model_xml)
if ("GPU" in device) and ("GPU" not in ie.available_devices):
DeviceNotFoundAlert("GPU")
else:
benchmark_command = f"benchmark_app -m {model_path} -d {device} -t {seconds} -api {api} -b {batch}"
display(Markdown(f"**Benchmark {model_path.name} with {device} for {seconds} seconds with {api} inference**"))
display(Markdown(f"Benchmark command: `{benchmark_command}`"))
benchmark_output = %sx $benchmark_command
benchmark_result = [line for line in benchmark_output
if not (line.startswith(r"[") or line.startswith(" ") or line == "")]
print("\n".join(benchmark_result))
ie = IECore()
# Show devices available for OpenVINO Inference Engine
for device in ie.available_devices:
device_name = ie.get_metric(device, "FULL_DEVICE_NAME")
print(f"{device}: {device_name}")
CPU: Intel(R) Core(TM) i9-10920X CPU @ 3.50GHz
benchmark_model(model_path, device="CPU", seconds=15, api="async")
Benchmark mobilenet-v2-pytorch.xml with CPU for 15 seconds with async inference
Benchmark command:
benchmark_app -m /opt/home/k8sworker/open_model_zoo_models/public/mobilenet-v2-pytorch/FP16/mobilenet-v2-pytorch.xml -d CPU -t 15 -api async -b 1
Count: 20142 iterations
Duration: 15006.07 ms
Latency: 4.51 ms
Throughput: 1342.26 FPS
benchmark_model(model_path, device="AUTO", seconds=15, api="async")
Benchmark mobilenet-v2-pytorch.xml with AUTO for 15 seconds with async inference
Benchmark command:
benchmark_app -m /opt/home/k8sworker/open_model_zoo_models/public/mobilenet-v2-pytorch/FP16/mobilenet-v2-pytorch.xml -d AUTO -t 15 -api async -b 1
Count: 9850 iterations
Duration: 15002.30 ms
Latency: 1.51 ms
Throughput: 656.57 FPS
benchmark_model(model_path, device="GPU", seconds=15, api="async")
benchmark_model(model_path, device="MULTI:CPU,GPU", seconds=15, api="async")