PROJECT NOT UNDER ACTIVE MANAGEMENT

This project will no longer be maintained by Intel.
Intel has ceased development and contributions including, but not limited to, maintenance, bug fixes, new releases, or updates, to this project.
Intel no longer accepts patches to this project.
If you have an ongoing need to use this project, are interested in independently developing it, or would like to maintain patches for the open source software community, please create your own fork of this project.

MLPerf Intel OpenVino Inference Code

Environment

• Ubuntu 20.04
• cmake 3.23.2
• gcc/g++ 9.4.0
• Python 3.8.10
• MLPerf™ Inference Benchmark Suite v3.0
• OpenVINO Toolkit 2023.0

Rules

MLPerf Inference Rules are here.

Supported benchmarks

Area	Task	Model	Dataset	SingleStream	MultiStream	Server	Offline
Vision	Image classification	Resnet50-v1.5 (Image classification)	ImageNet (224x224)	✔️	✔️	✔️	✔️
Vision	Object detection	Retinanet (Object detection)	OpenImages (800x800)	✔️	✔️	✔️	✔️
Vision	Medical image segmentation	3D UNET (Medical image segmentation)	KITS 2019 (602x512x512)	✔️	❌	❌	✔️
Language	Language processing	BERT-large (Language processing)	SQuAD v1.1 (max_seq_len=384)	✔️	✔️	✔️	✔️

✔️ - supported ❌ - not supported

Execution Modes

• Performance
• Accuracy
  • User first runs the benchmark in Accuracy mode to generate mlperf_log_accuracy.json
  • User then runs a dedicated accuracy tool provided by MLPerf

BKC on CPX, ICX systems

Use the following to optimize performance on CPX/ICX systems. These BKCs are provided in performance.sh mentioned in How to Build and Run.

• Turbo ON

  echo 0 > /sys/devices/system/cpu/intel_pstate/no_turbo
  

• Set CPU governor to performance (Please rerun this command after reboot):

  echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
  

  OR

  cpupower frequency-set -g performance
  

How to Build and Run

1. Navigate to root repository directory. This directory is your BUILD_DIRECTORY.

2. Run the build script:

   ./build.sh
   

   NOTE: sudo privileges are required

3. Modify BUILD_DIRECTORY in setup_env.sh(if necessary) and source:

   source scripts/setup_env.sh
   

4. Run the performance script for CPX/ICX systems:

   ./scripts/performance.sh
   

5. Download models

   ./scripts/download_models.sh [specific model]
   

6. Download datasets

   ./scripts/download_datasets.sh [specific dataset]
   

7. Modify script ./scripts/run.sh to apply desired parameters.

   The following OpenVINO parameters should be adjusted based on selected hardware target:

   • number of streams
   • number of infer requests
   • number of threads
   • inference precision

8. Update MLPerf parameters (user.conf and mlperf.conf) if it is needed.

9. Run:

   ./scripts/run.sh -m <model> -d <device> -s <scenario> -e <mode>
   

   For example (results will be stored into the ${BUILD_DIRECTORY}/results/resnet50/CPU/Performance/SingleStream folder):

   ./scripts/run.sh -m resnet50 -d CPU -s SingleStream -e Performance
   

   To run all combination of models/devices/scenarios/modes:

   ./scripts/run_all.sh
   

   NOTE: This product is not for production use and scripts are provided as example. For reporting MLPerf results dedicated scripts should be provided for each model with suitable parameters.