Intel ARC?

[turnkit]

Has anyone got Whisper accelerated on Intel ARC GPU?

looking at ways to possibly build several smaller affordable dedicated Whisper workstations.

Could this best cost effective vs buying one expensive and fast Nvidia 4000 series system?

re:
https://medium.com/intel-analytics-software/running-tensorflow-stable-diffusion-on-intel-arc-gpus-e6ff0d2b7549

https://www.pcgamer.com/intels-new-dollar249-gpu-price-wipes-out-nvidia-at-the-entry-level/

[771991673]

just found the way to run on intel

import whisper
import intel_extension_for_pytorch as ipex
MODEL = 'small'
model = whisper.load_model(MODEL, device = "xpu")

[leuc]

Large model works on ARC A770 16GB with the latest xpu-master branch from intel-extension-for-pytorch

Some patching required, see: intel/intel-extension-for-pytorch#302

[leuc]

Memory usage on Intel ARC A770 measured with lsgpu

baseline usage

>lsgpu -p|grep ^lmem_
lmem_total_bytes                : 17079205888
lmem_avail_bytes                : 16260358144

tiny.en

lmem_avail_bytes                : 14516899840

tiny

lmem_avail_bytes                : 14497824768

medium

lmem_avail_bytes                : 10508996608

large

lmem_avail_bytes                : 4901576704

tiny.en:   2.3863 GB
tiny       2.4028 GB
medium     6.119  GB
large:    11.3413 GB

[natecarlson]

How is the performance?

[leuc]

Just re-tried with v1.13.120+xpu and both model loading time and decoding have a "good" speed now.

with model file in page cache

time whisper --language en --model tiny.en --device xpu tests/jfk.flac
[00:00.000 --> 00:07.940]  And so my fellow Americans ask not what your country can do for you

real	0m7.218s
user	0m7.727s
sys	0m2.569s

time whisper --language en --model base.en --device xpu tests/jfk.flac
[00:00.000 --> 00:11.000]  And so my fellow Americans, ask not what your country can do for you, ask what you can do for your country.

real	0m10.608s
user	0m11.040s
sys	0m2.973s

time whisper --language en --model medium --device xpu tests/jfk.flac
[00:00.000 --> 00:06.880]  And so, my fellow Americans, ask not what your country can do

real	0m15.856s
user	0m19.593s
sys	0m4.448s

[leuc]

PR for Intel GPU Support #1362

[WildBirdTG]

A770 16G is basically the same price as 3060 12G, for the purpose of this topic, I'll use the later one as the large model works well on it with less headache of set up.

[HubKing]

Well, I would have used Nvidia, if I were not using Wayland Linux. I do heard that recently, it got better on Linux, but when I switched to Linux with an RTX card a few years ago, the desktop was almost unusable, and that was why I switched to Arc.

[HubKing]

Does this mean that currently, Whisper does NOT work with Arc A770 as it is? I am using Arch Linux with an A770. I installed a local Python 3.11.9 with the packages below, and I could run ComfyUI and generate images with this Python. However, when I ran Whisper in the same Python, it seemed like it was using CPU.

$whisper sample.mp3 --model tiny                                                                  1 ↵
100%|█████████████████████████████████████| 72.1M/72.1M [00:06<00:00, 12.3MiB/s]
/home/user/.pyenv/versions/3.11.9/lib/python3.11/site-packages/whisper/transcribe.py:115: UserWarning: FP16 is not supported on CPU; using FP32 instead
  warnings.warn("FP16 is not supported on CPU; using FP32 instead")
Detecting language using up to the first 30 seconds. Use `--language` to specify the language
Detected language: English

$ pip list                       
Package                     Version
--------------------------- ----------------
aiohappyeyeballs            2.4.0
aiohttp                     3.10.5
aiosignal                   1.3.1
annotated-types             0.7.0
attrs                       24.2.0
certifi                     2024.8.30
cffi                        1.17.1
charset-normalizer          3.3.2
commandlines                0.4.1
decorator                   4.4.2
dpcpp-cpp-rt                2024.2.1
einops                      0.8.0
ffmpeg-python               0.2.0
filelock                    3.16.1
filetype                    1.2.0
frozenlist                  1.4.1
fsspec                      2024.9.0
future                      1.0.0
hsh                         1.1.0
huggingface-hub             0.25.0
idna                        3.10
imageio                     2.35.1
imageio-ffmpeg              0.5.1
impi-devel                  2021.13.1
impi-rt                     2021.13.1
intel-cmplr-lib-rt          2024.2.1
intel-cmplr-lib-ur          2024.2.1
intel-cmplr-lic-rt          2024.2.1
intel_extension_for_pytorch 2.3.110+xpu
intel-opencl-rt             2024.2.1
intel-openmp                2024.2.1
intel-sycl-rt               2024.2.1
Jinja2                      3.1.4
kornia                      0.7.3
kornia_rs                   0.1.5
lazy_loader                 0.4
llvmlite                    0.43.0
MarkupSafe                  2.1.5
mkl                         2024.2.1
mkl-dpcpp                   2024.2.1
more-itertools              10.5.0
moviepy                     1.0.3
mpmath                      1.3.0
multidict                   6.1.0
networkx                    3.3
numba                       0.60.0
numpy                       2.0.2
oneccl-bind-pt              2.3.100+xpu
oneccl-devel                2021.13.1
onemkl-sycl-blas            2024.2.1
onemkl-sycl-datafitting     2024.2.1
onemkl-sycl-dft             2024.2.1
onemkl-sycl-lapack          2024.2.1
onemkl-sycl-rng             2024.2.1
onemkl-sycl-sparse          2024.2.1
onemkl-sycl-stats           2024.2.1
onemkl-sycl-vm              2024.2.1
packaging                   24.1
Pillow                      9.5.0
pip                         24.2
proglog                     0.1.10
psutil                      6.0.0
pycparser                   2.22
pydantic                    2.9.2
pydantic_core               2.23.4
PyMatting                   1.1.12
PySocks                     1.7.1
PyYAML                      6.0.2
regex                       2024.9.11
requests                    2.32.3
ruamel.yaml                 0.18.6
ruamel.yaml.clib            0.2.8
safetensors                 0.4.5
scikit-image                0.24.0
scipy                       1.14.1
sentencepiece               0.2.0
setuptools                  65.5.0
six                         1.16.0
soundfile                   0.12.1
spandrel                    0.4.0
sympy                       1.13.3
tbb                         2021.13.1
tifffile                    2024.9.20
tokenizers                  0.19.1
torch                       2.3.1+cxx11.abi
torchaudio                  2.3.1+cxx11.abi
torchsde                    0.2.6
torchvision                 0.18.1+cxx11.abi
tqdm                        4.66.5
trampoline                  0.1.2
transformers                4.44.2
typing_extensions           4.12.2
urllib3                     2.2.3
waitress                    3.0.0
yarl                        1.11.1