Configuration file declares validation process. Every model has to have entry in
models list. Each entry has to contain distinct
Also there are composite models which consist of several parts (models) and the accuracy measurement requires building the pipeline from these parts. Thus, the evaluation is performed by sequentially executing a set of models and impossible to evaluate them independently. Each composite model has to have entry in
evaluations list. Each entry should contain distinct
module_config has to consist of
datasets fields. Custom evaluators are used for such models. More information about defining and using your own evaluator or an existing one can be found in Custom Evaluators Guide
Predefined configuration file
accuracy-check.yml for each Open Model Zoo model can be found in the model directory.
<model_name>.yml file, which is located in current
configs folder, is a link to
-c, --configcommand line.
-m, --modelscommand line option.
-d, --definitions. In order, if you want use definitions file in quantization via Post Training Optimization Toolkit, you should use environment variable
DEFINITIONS_FILEfor specifying path to definitions.
data_sourceis specified can be provided via
-s, --sourcecommand line. If you want to evaluate models using well-known datasets, you need to organize folders with validation datasets in a certain way. More detailed information about dataset preparation you can find in Dataset Preparation Guide. In order, if you want use data source in quantization via Post Training Optimization Toolkit, you should use environment variable
DATA_DIRfor specifying path to root of directories with datasets.
dataset_metaare specified can be provided via
-a, --annotationscommand line. Annotation and dataset_meta (if required) will be stored to this directory after annotation conversion step if they do not exist and can be used for the next running to skip annotation conversion. Detailed information about annotation conversion you can find in Annotation Conversion Guide.
--model_attributescommand line option (usually, it is the model directory).
-td, --target_devicescommand line option. Several devices should be separated by spaces (e.g. -td CPU GPU).
-tf, --target_frameworkcommand line option. Otherwise, if the option is not provided evaluation will be launched with all frameworks mentioned in the configuration file.
See how to evaluate model with using predefined configuration file for densenet-121-tf model.
OMZ_ROOT- root of Open Model Zoo project
DATASET_DIR- root directory with dataset
MODEL_DIR- root directory with model
OPENVINO_DIR- root directory with installed the OpenVINO™ toolkit
Similarly, you can run evaluation for model in FP16 precision
Also you can quantize full-precision models in the IR format into low-precision versions via Model Quantizer
Run evaluation for quantized model: ```sh accuracy-check -c OMZ_ROOT/models/public/densenet-121-tf/accuracy-check.yml -s DATASET_DIR -m MODEL_DIR/public/densenet-121-tf/FP16-INT8 -d OMZ_ROOT/tools/accuracy_checker/dataset_definitions.yml -td CPU GPU ```