NameError: name 'exllamav2_ext' is not defined

[paul0728]

Describe the bug

I am using the Sakura-1.5B-Qwen2.5-v1.0-h8-8bpw-exl2 model from the Link, but when trying to load the model with TGW, this error occurs. I have tried using Python 3.10 and Python 3.11, but I still receive the same error message.

Is there an existing issue for this?

• I have searched the existing issues

Reproduction

1. Clone the repository.
2. Change the current directory to text-generation-webui.
3. Download and install Anaconda
4. conda create -n tgw python=3.11
5. conda activate tgw
6. pip install -r requirements.txt
7. Set up CMD_FLAGS.txt with the following content: --auto-launch --api --trust-remote-code.
8. Double click run_server.bat

@echo off
CALL E:\Anaconda3\Scripts\activate.bat E:\Anaconda3
CALL conda activate tgw

setlocal enabledelayedexpansion
set "cmd=python server.py"

for /f "tokens=*" %%a in (CMD_FLAGS.txt) do (
    set "line=%%a"
    if not "!line:~0,1!"=="#" if not "!line:~0,2!"=="//" (
        set "cmd=!cmd! !line!"
    )
)

%cmd%
pause

9. Load model Sakura-1.5B-Qwen2.5-v1.0-h8-8bpw-exl2.

Screenshot

Logs

23:55:11-479484 INFO     Starting Text generation web UI
23:55:11-483489 WARNING  trust_remote_code is enabled. This is dangerous.
23:55:11-501907 INFO     Loading the extension "openai"
23:55:12-071517 INFO     OpenAI-compatible API URL:

                         http://127.0.0.1:5000


Running on local URL:  http://127.0.0.1:7860

23:55:21-847871 INFO     Loading "Sakura-1.5B-Qwen2.5-v1.0-h8-8bpw-exl2"

 ## Warning: The installed version of PyTorch is 2.5.1+cpu and does not support CUDA or ROCm.

23:55:28-986408 ERROR    Failed to load the model.
Traceback (most recent call last):
  File "E:\translator\text-generation-webui\modules\ui_model_menu.py", line 232, in load_model_wrapper
    shared.model, shared.tokenizer = load_model(selected_model, loader)
  File "E:\translator\text-generation-webui\modules\models.py", line 93, in load_model
    output = load_func_map[loader](model_name)
  File "E:\translator\text-generation-webui\modules\models.py", line 313, in ExLlamav2_HF_loader
    from modules.exllamav2_hf import Exllamav2HF
  File "E:\translator\text-generation-webui\modules\exllamav2_hf.py", line 7, in <module>
    from exllamav2 import (
  File "E:\Anaconda3\envs\tgw\lib\site-packages\exllamav2\__init__.py", line 3, in <module>
    from exllamav2.model import ExLlamaV2
  File "E:\Anaconda3\envs\tgw\lib\site-packages\exllamav2\model.py", line 35, in <module>
    from exllamav2.config import ExLlamaV2Config
  File "E:\Anaconda3\envs\tgw\lib\site-packages\exllamav2\config.py", line 5, in <module>
    from exllamav2.stloader import STFile, cleanup_stfiles
  File "E:\Anaconda3\envs\tgw\lib\site-packages\exllamav2\stloader.py", line 5, in <module>
    from exllamav2.ext import none_tensor, exllamav2_ext as ext_c
  File "E:\Anaconda3\envs\tgw\lib\site-packages\exllamav2\ext.py", line 291, in <module>
    ext_c = exllamav2_ext
NameError: name 'exllamav2_ext' is not defined

System Info

Hardware info:
CPU:  AMD Ryzen 5 5600X 6-Core Processor
GPU: 3080ti 12G
RAM: 32G

System info:
OS: win11
Cuda:

[RSAStudioGames]

Is flash attention installed?
Try the "no_flash_attn" flag and see if it that fixes the issue.

[Alkohole]

You completely missed the second point of the installation instructions, you have CUDA 12.4 and pytorch 2.5.1, but it should be CUDA 12.1 and pytorch 2.4.1, so you have at least EXL2 not working correctly.