Radeon AI R9700 Report

[mechovation]

Got your repo running today on my R9700.

Docker built from rocm/dev-ubuntu-24.04 on Ubuntu24.04 server with ROCm 7.2.1

More important than the speed (below), I now have 4 lanes of 256K context, where previously 128K would overflow to the CPU in Ollama for similar size models.

Looking forward to llama.cpp project catching and being able to try Qwen3.5 and GLM-4.7-Flash.

Thanks!!

Llama-Benchy results with turboquant_plus:

model	test	t/s	peak t/s	ttfr (ms)	est_ppt (ms)	e2e_ttft (ms)
qwen3:30b	pp2048	2753.36 ± 45.38		672.79 ± 27.41	672.26 ± 27.41	672.83 ± 27.41
qwen3:30b	tg32	77.89 ± 0.12	80.42 ± 0.12

model	test	t/s (total)	t/s (req)	peak t/s	peak t/s (req)	ttfr (ms)	est_ppt (ms)	e2e_ttft (ms)
qwen3:30b	pp2048 (c4)	2077.65 ± 221.17	968.76 ± 575.17			2425.59 ± 991.81	2425.13 ± 991.81	2425.62 ± 991.81
qwen3:30b	tg32 (c4)	40.37 ± 0.88	24.22 ± 15.10	122.00 ± 0.00	35.12 ± 8.91

[mechovation]

docker-compose.yaml:

services:
  turboquant:
    build:
      context: .
      dockerfile: Dockerfile
    image: turboquant-rocm:latest
    ports:
      - "8080:8080"
    volumes:
      # Ollama model volume mounted read-only — blobs at /ollama/models/blobs/
      - open-webui_ollama:/ollama:ro
    devices:
      - /dev/kfd:/dev/kfd
      - /dev/dri:/dev/dri
    group_add:
      - video
      - render
    # Validated config for RDNA 4 / Q4_K_M models (README AMD results, line 253)
    # Replace MODEL_BLOB with the sha256 hash from: scripts/find-ollama-model.sh
    command: >
      -m /ollama/models/blobs/sha256-58574f2e94b99fb9e4391408b57e5aeaaaec10f6384e9a699fc2cb43a5c8eabf
      --alias qwen3-30b
      -ngl 99
      -fa on
      -ctk q8_0
      -ctv turbo4
      --host 0.0.0.0
      --port 8080
      -c 262144

volumes:
  open-webui_ollama:
    external: true
    ```

[mechovation]

Dockerfile:

# TurboQuant+ llama-server — AMD ROCm (RDNA 4 / gfx1201)
# Tested target: Radeon AI R9700, gfx1201, ROCm 7.2
#
# Build: docker compose build
# Run:   docker compose up

FROM rocm/dev-ubuntu-24.04:latest

# Build dependencies
# ROCm math libs: hipblas-dev and rocblas-dev provide cmake config files
# required by ggml-hip; rocsolver/rocsparse/hipsparse needed for full BLAS stack
RUN apt-get update && apt-get install -y --no-install-recommends \
    git \
    cmake \
    ninja-build \
    hipblas-dev \
    rocblas-dev \
    rocsolver-dev \
    rocsparse-dev \
    hipsparse-dev \
    && rm -rf /var/lib/apt/lists/*

# Clone TurboQuant llama.cpp fork
WORKDIR /opt
RUN git clone --branch feature/turboquant-kv-cache --depth 1 \
    https://github.com/TheTom/llama-cpp-turboquant.git

WORKDIR /opt/llama-cpp-turboquant

# Build with HIP for gfx1201 (RDNA 4)
# GGML_CUDA_FA_ALL_QUANTS enables FlashAttention for all quant types including turbo
RUN cmake -S . -B build -G Ninja \
    -DGPU_TARGETS=gfx1201 \
    -DGGML_HIP=ON \
    -DGGML_CUDA_FA_ALL_QUANTS=ON \
    -DCMAKE_BUILD_TYPE=Release \
    && cmake --build build --config Release -j$(nproc)

EXPOSE 8080

ENTRYPOINT ["/opt/llama-cpp-turboquant/build/bin/llama-server"]
CMD ["--help"]