Fix turbo-KV build flag (FA_ALL_QUANTS) + add CUDA→GHCR build workflow#48
Closed
noonghunna wants to merge 1 commit into
Closed
Fix turbo-KV build flag (FA_ALL_QUANTS) + add CUDA→GHCR build workflow#48noonghunna wants to merge 1 commit into
noonghunna wants to merge 1 commit into
Conversation
The CUDA Dockerfile omits -DGGML_CUDA_FA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON, required at runtime for the turbo/TCQ KV cache types. Also add a nightly + on-push GitHub Actions workflow that builds the server target and pushes a multi-arch (sm_86/89/120) CUDA image to GHCR, so testers can pull instead of compiling from source. Context: issue Anbeeld#39 / club-3090 discussion. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Owner
|
Implemented CI/CD for both build and Docker images, but in a different way, on-push for main and versioned branches. Fixed missing quant args and a number of other issues with builds across various platforms. Currently v0.3.0 seems to have a successful build, please try if Docker works with no issues. Closing this PR as the end goal should be achieved, if there are still some issues I'll do follow-ups. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two small additions to make v0.3.0 pullable as a CUDA Docker image, in support of #39 / the club-3090 discussion where you asked dual+ GPU folks to test v0.3.0. The biggest barrier to community multi-GPU reports right now is that everyone has to compile from source — a published image turns that into
docker pull.1.
.devops/cuda.Dockerfile— enableFA_ALL_QUANTS(one line)The CUDA build currently omits
-DGGML_CUDA_FA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON. These are required for the turbo / TCQ KV cache types (turbo2/3/4, turbo2_tcq, turbo3_tcq) — without them the image builds fine but those cache types fail at runtime. Your README/CLAUDE.md already document these flags as required; this just brings the Dockerfile in line so adocker buildmatches a source build. Worth landing on its own even if you rework the CI part.2.
.github/workflows/build-cuda-docker.yml— CUDA image → your GHCRservertarget and pushes to your repo's GHCR (ghcr.io/<owner>/beellama.cpp) via the built-inGITHUB_TOKEN— no secrets to set up.v0.3.0(path-filtered to build-affecting files), and manual dispatch (arch selectable).:server-cuda-v0.3.0(always latest) plus an immutable:server-cuda-v0.3.0-<sha>(per-commit, for reproducible bug reports).push/ manual dispatch force a rebuild).Three operational notes:
v0.3.0).86;89;120so one image covers 3090/4090/5090 (the cards you're recruiting). That's heavy for a free hosted runner (disk + time); if it doesn't fit, theworkflow_dispatchinput lets you drop an arch, or swapruns-on:to a larger / self-hosted runner (commented inline).Totally your call on whether/how to take this — happy to adjust anything.
In the meantime we're hosting an unofficial multi-arch (sm_86/89/120) snapshot of v0.3.0 just so the testers in the discussion have something to pull this week. It's a point-in-time snapshot pinned to one commit, though — it won't track your ongoing v0.3.0 work. This workflow is the clean way to make a nightly that does auto-track v0.3.0, which makes the most sense under your ownership of the branch + registry.