OpenVINO

OpenVINO

OpenVINO is an open-source toolkit for optimizing and deploying deep learning models.

• Compiles models for your hardware.
• Supports Linux and Windows.
• Supports CPU, GPU, iGPU, and NPU targets.
• Supports Intel dGPUs, iGPUs, and NPUs.
• Supports AMD dGPUs and iGPUs on Windows with FP16 support.
• Supports NVIDIA GPUs.
• Supports any GPU with OpenCL support.
• Supports CPUs with BF16, FP16, or FP32 support.
• Supports multi-device execution using HETERO mode.

In practice, it is an alternative to TensorRT or Olive that can run on a wide range of hardware.

Installation

Preparations

• Install the drivers for your device
• Install Git and Python

Note

Do not mix OpenVINO with your old install. Treat OpenVINO as a separate backend.

Running SD.Next with OpenVINO

Open a terminal in the folder where you want to install SD.Next, then clone the repository:

git clone https://github.com/vladmandic/sdnext

Then enter into the sdnext folder:

cd sdnext

Then start WebUI with this command:

Windows:

.\webui.bat --use-openvino

Linux:

./webui.sh --use-openvino

Note

It will install the necessary libraries at the first run so it will take a while depending on your internet.

Running SD.Next with Docker

See the Docker wiki if you want to build a custom image.

To run a prebuilt Docker image:

export SDNEXT_DOCKER_ROOT_FOLDER=~/sdnext
sudo docker run -it \
  --name sdnext-openvino \
  --device /dev/dri \
  -p 7860:7860 \
  -v $SDNEXT_DOCKER_ROOT_FOLDER/app:/app \
  -v $SDNEXT_DOCKER_ROOT_FOLDER/python:/mnt/python \
  -v $SDNEXT_DOCKER_ROOT_FOLDER/data:/mnt/data \
  -v $SDNEXT_DOCKER_ROOT_FOLDER/models:/mnt/models \
  -v $SDNEXT_DOCKER_ROOT_FOLDER/huggingface:/root/.cache/huggingface \
  disty0/sdnext-openvino:latest

Note

It installs required libraries on first run, so startup can take a while depending on your connection. Resulting docker image will use 1.1 GB disk space (uncompressed) for the docker image and 2.5 GB for the venv.

More Info

Limitations

• Most TensorRT/Olive limitations also apply here.
• Compilation takes a few minutes and using LoRas will trigger recompilation.
• Attention Slicing and HyperTile will not work.

Custom Devices

Use the OpenVINO devices option in Backend Settings to select a device. Selecting multiple devices combines them into a single HETERO device.

If no device is specified, OpenVINO auto-selects the default device.

Model Caching

OpenVINO can save compiled models to a cache folder so they do not need to be compiled again.

• OpenVINO disable model caching in Backend Settings turns caching off.
• Directory for OpenVINO cache in System Paths sets the cache location.