diff --git a/MTP.md b/MTP.md index 1f9538fd037b..09346df711e1 100644 --- a/MTP.md +++ b/MTP.md @@ -87,7 +87,8 @@ CLI surface (`common/arg.cpp`, `common/common.cpp`): - `--mtp-head ` (preferred) and `--model-draft / -md` (back-compat alias) — feed the same `mparams_dft.path` field. -- `--draft-block-size ` — head proposes `B - 1` tokens per round. +- `--spec-draft-n-max ` — head proposes `N` tokens per round (replaces + the pre-b10018 `--draft-block-size `, `N = B - 1`). - `--gpu-layers-draft / -ngld`, `-ctkd / -ctvd` — placement and KV typing for the **assistant** weights when offloaded. @@ -443,7 +444,7 @@ Helper scripts live under `scripts/`: Edge presets (`MTP_PRESET`): -| Preset | `--draft-block-size` (B) | `--draft-max` | +| Preset | `DRAFT_BLOCK_SIZE` (B, passed as `--spec-draft-n-max B-1`) | `DRAFT_MAX` (legacy cap, no longer passed) | |---|---:|---:| | `throughput` | 2 | 6 | | `lift` | 3 | 8 | diff --git a/NEXTN.md b/NEXTN.md index 4d8af3897f0c..768484beb596 100644 --- a/NEXTN.md +++ b/NEXTN.md @@ -39,14 +39,14 @@ the same file in the HF cache and takes the shared-model branch: llama-server \ -hf AtomicChat/Qwen3.6-35B-A3B-UDT-MTP-GGUF:Q4_K_XL \ -hfd AtomicChat/Qwen3.6-35B-A3B-UDT-MTP-GGUF:Q4_K_XL \ - --spec-type nextn --draft-max 2 --draft-min 1 \ + --spec-type nextn --spec-draft-n-max 2 --spec-draft-n-min 1 \ -c 8192 -ngl 99 -ngld 99 -fa on # 27B dense llama-server \ -hf AtomicChat/Qwen3.6-27B-UDT-MTP-GGUF:Q4_K_XL \ -hfd AtomicChat/Qwen3.6-27B-UDT-MTP-GGUF:Q4_K_XL \ - --spec-type nextn --draft-max 2 --draft-min 1 \ + --spec-type nextn --spec-draft-n-max 2 --spec-draft-n-min 1 \ -c 8192 -ngl 99 -ngld 99 -fa on ``` @@ -76,7 +76,7 @@ that used to OOM the 35B-A3B target on Apple Silicon (38 GB unified memory). See and switches to the shared-model path (no second model load). Pointing at a standalone NEXTN_ONLY GGUF (`general.architecture = qwen35*_mtp`) still works but loads a second `llama_model`. -- `--draft-max` / `--spec-draft-n-max` — max chained draft tokens per round (see `common` / server arg naming). +- `--spec-draft-n-max` — max chained draft tokens per round (pre-b10018 name: `--draft-max`). - Gemma MTP flags (`--mtp-head`, `llama_decode_mtp_*`, `llama_model_load_mtp_from_file`) are **unchanged**. --- @@ -123,14 +123,14 @@ let `llama-server` pull both from Hugging Face into the local cache: llama-server \ -hf AtomicChat/Qwen3.6-35B-A3B-UDT-MTP-GGUF:Q4_K_XL \ -hfd AtomicChat/Qwen3.6-35B-A3B-UDT-MTP-GGUF:Q4_K_XL \ - --spec-type nextn --draft-max 2 --draft-min 1 + --spec-type nextn --spec-draft-n-max 2 --spec-draft-n-min 1 ``` --- ## 7. Performance notes (MacBook Pro M4 Max, 40-core GPU, 48 GB, Metal) -Median TPS over 2 runs, prompt = 50-token instruction, `--draft-max=2 --draft-min=1`, +Median TPS over 2 runs, prompt = 50-token instruction, `--spec-draft-n-max=2 --spec-draft-n-min=1`, NextN draft DM=2 (single async chain), context 8192. Single-slot (`--parallel 1 -np 1 --cont-batching`), full GPU offload (`-ngl 99 -ngld 99 -fa on`), shared-model draft path (no second mmap of combined `_MTP.gguf`), diff --git a/README.md b/README.md index a6b986999dee..d46840a281e9 100644 --- a/README.md +++ b/README.md @@ -117,7 +117,7 @@ llama-server \ -m /path/to/gemma-4-target.gguf \ --mtp-head /path/to/gemma-4-assistant-Q4_K_M.gguf \ --spec-type mtp \ - --draft-block-size 3 \ + --spec-draft-n-max 2 \ -c 16384 \ -ngl 99 -ngld 99 \ -fa on \ @@ -140,8 +140,8 @@ MTP_PRESET=throughput scripts/run-gemma4-e2b-mtp-server.sh ### Bench snapshot (MacBook Pro M4 Max, 40-core GPU, 48 GB, Metal, single slot) Median tps over 3 runs with Q4_K_M assistant heads. Dense scripts default to -`--draft-block-size 3`; E4B uses `MTP_PRESET=throughput` (`B = 2`, -`--draft-max 6`). See +two drafted tokens per round (`--spec-draft-n-max 2`); E4B uses +`MTP_PRESET=throughput` (one drafted token per round). See `.scratch/bench-logs/gemma-matrix-fullrun-20260512-224705.md`. | model | mode | n=128 tps | n=512 tps | accept@128 | accept@512 | @@ -158,8 +158,8 @@ Median tps over 3 runs with Q4_K_M assistant heads. Dense scripts default to ### Knobs -- `--draft-block-size B` — head emits `B - 1` tokens per round (default 4; - bench used 3). +- `--spec-draft-n-max N` — head emits `N` tokens per round (bench used 2; + replaces the pre-b10018 `--draft-block-size B`, `N = B - 1`). - `--mtp-head ` (preferred) / `-md ` (back-compat alias). - `LLAMA_MTP_SKIP_STREAK_THRESHOLD=N` — adaptive skip after `N` consecutive zero-accept batches (off by default). @@ -225,7 +225,7 @@ llama-server \ -hf AtomicChat/Qwen3.6-35B-A3B-UDT-MTP-GGUF:Q4_K_XL \ -hfd AtomicChat/Qwen3.6-35B-A3B-UDT-MTP-GGUF:Q4_K_XL \ --spec-type nextn \ - --draft-max 2 --draft-min 1 \ + --spec-draft-n-max 2 --spec-draft-n-min 1 \ -c 8192 \ -ngl 99 -ngld 99 \ -ctk turbo3 -ctv turbo3 -fa on \ @@ -238,7 +238,7 @@ Or with a local file (e.g. the artifact stored under `.scratch/`): llama-server \ -m /path/to/Qwen3.6-35B-A3B-UD-Q4_K_XL_MTP.gguf \ -md /path/to/Qwen3.6-35B-A3B-UD-Q4_K_XL_MTP.gguf \ - --spec-type nextn --draft-max 2 --draft-min 1 \ + --spec-type nextn --spec-draft-n-max 2 --spec-draft-n-min 1 \ -c 8192 -ngl 99 -ngld 99 -ctk turbo3 -ctv turbo3 -fa on ``` @@ -256,7 +256,7 @@ If you ship the NextN head as a separate **NEXTN_ONLY** GGUF ### Bench snapshot (MacBook Pro M4 Max, 40-core GPU, 48 GB, Metal, single slot) -Median tps over 3 runs, `--draft-max 2 --draft-min 1`, single-slot, shared +Median tps over 3 runs, `--spec-draft-n-max 2 --spec-draft-n-min 1`, single-slot, shared target/draft model. See `.scratch/bench-logs/qwen-matrix-fullrun-20260512-222625.md`. @@ -277,7 +277,7 @@ target/draft model. See - `--model-draft` / `-md` — pass the **same** path as `--model` for the shared-model path; pass a NEXTN_ONLY GGUF to use the legacy double-load fallback. -- `--draft-max` / `--draft-min` — chained-draft bounds per round +- `--spec-draft-n-max` / `--spec-draft-n-min` — chained-draft bounds per round (current default for the helper scripts: `2 / 1`). - `llama_set_nextn` (C API) — pairs target and draft contexts so that `llama_context_nextn_seq_rm` trims **both** KV caches in one call. @@ -750,7 +750,7 @@ To learn more about model quantization, [read this documentation](tools/quantize -m /path/to/gemma-4-target.gguf \ --mtp-head /path/to/gemma-4-assistant-Q4_K_M.gguf \ --spec-type mtp \ - --draft-block-size 3 \ + --spec-draft-n-max 2 \ -c 16384 \ -ngl 99 -ngld 99 \ -fa on \ @@ -808,7 +808,7 @@ To learn more about model quantization, [read this documentation](tools/quantize -hf unsloth/Qwen3.6-35B-A3B-MTP-GGUF:UD-Q4_K_XL \ -hfd unsloth/Qwen3.6-35B-A3B-MTP-GGUF:UD-Q4_K_XL \ --spec-type nextn \ - --draft-max 2 --draft-min 1 \ + --spec-draft-n-max 2 --spec-draft-n-min 1 \ -c 8192 \ -ngl 99 -ngld 99 \ -ctk turbo3 -ctv turbo3 -fa on \ @@ -821,7 +821,7 @@ To learn more about model quantization, [read this documentation](tools/quantize llama-server \ -m /path/to/Qwen3.6-35B-A3B-UD-Q4_K_XL_MTP.gguf \ -md /path/to/Qwen3.6-35B-A3B-UD-Q4_K_XL_MTP.gguf \ - --spec-type nextn --draft-max 2 --draft-min 1 \ + --spec-type nextn --spec-draft-n-max 2 --spec-draft-n-min 1 \ -c 8192 -ngl 99 -ngld 99 -ctk turbo3 -ctv turbo3 -fa on ``` diff --git a/common/arg.cpp b/common/arg.cpp index 4e6ad090ee76..c856a1e1b78e 100644 --- a/common/arg.cpp +++ b/common/arg.cpp @@ -3877,8 +3877,8 @@ common_params_context common_params_parser_init(common_params & params, llama_ex } ).set_spec().set_examples({LLAMA_EXAMPLE_SPECULATIVE, LLAMA_EXAMPLE_SERVER, LLAMA_EXAMPLE_CLI}).set_env("LLAMA_ARG_N_GPU_LAYERS_DRAFT")); add_opt(common_arg( - {"--spec-draft-model", "-md", "--model-draft"}, "FNAME", - "draft model for speculative decoding (default: unused)", + {"--spec-draft-model", "-md", "--model-draft", "--mtp-head"}, "FNAME", + "draft model for speculative decoding, also accepts an MTP/NextN head GGUF (default: unused)", [](common_params & params, const std::string & value) { params.speculative.draft.mparams.path = value; params.speculative.draft.mparams.hf_file = value; // will be used if --spec-draft-hf is set diff --git a/common/speculative.cpp b/common/speculative.cpp index 580728a2001e..c698e39e3231 100644 --- a/common/speculative.cpp +++ b/common/speculative.cpp @@ -38,7 +38,12 @@ const std::map common_speculative_type_fro {"ngram-map-k", COMMON_SPECULATIVE_TYPE_NGRAM_MAP_K}, {"ngram-map-k4v", COMMON_SPECULATIVE_TYPE_NGRAM_MAP_K4V}, {"ngram-mod", COMMON_SPECULATIVE_TYPE_NGRAM_MOD}, - {"ngram-cache", COMMON_SPECULATIVE_TYPE_NGRAM_CACHE} + {"ngram-cache", COMMON_SPECULATIVE_TYPE_NGRAM_CACHE}, + // fork aliases: Gemma 4 MTP ("mtp", MTP.md) and Qwen NextN ("nextn", + // NEXTN.md) both route to the draft-mtp implementation since the b10018 + // upstream sync. Kept so commands from the fork docs keep working. + {"mtp", COMMON_SPECULATIVE_TYPE_DRAFT_MTP}, + {"nextn", COMMON_SPECULATIVE_TYPE_DRAFT_MTP} }; static std::string common_speculative_get_devices_str(const std::vector & devices) { diff --git a/docs/speculative.md b/docs/speculative.md index cbd31ba9fe7c..c15352d37150 100644 --- a/docs/speculative.md +++ b/docs/speculative.md @@ -31,7 +31,7 @@ A draft model is the most used approach in speculative decoding. For **Gemma 4** targets with the **gemma4_assistant** (MTP head) GGUF, use `--spec-type mtp`. The assistant is **not** a second `llama_context`: weights are loaded into the target model via `llama_model_load_mtp_from_file` (done automatically when using the server/CLI init path). Cross-attention in the MTP graph reads **K/V from the target KV cache** (shared full/sliding layers). - Prefer **`--mtp-head /path/to/assistant.gguf`** for clarity; **`--model-draft` (`-md`)** is accepted as a backward-compatible alias (same path field). -- The draft block size \(B\) is **`--draft-block-size`** (the head proposes `B - 1` tokens per round; default 4). +- The per-round draft length is **`--spec-draft-n-max N`** (the head proposes `N` tokens per round; replaces the pre-b10018 `--draft-block-size B`, `N = B - 1`). - **`--gpu-layers-draft` / `-ngld`** and **`-ctkd` / `-ctvd`** still apply to how the **assistant tensors** are placed and typed when the assistant GGUF is loaded; the target uses `-ngl` and `-ctk`/`-ctv`. Example (paths illustrative). **TurboQuant** KV on the target: `-ctk`/`-ctv`. Assistant-side cache types follow the draft flags if you use them for offload/quant selection. @@ -41,7 +41,7 @@ llama-server \ -m /path/to/gemma-4-target.gguf \ --mtp-head /path/to/gemma-4-assistant.gguf \ --spec-type mtp \ - --draft-block-size 4 \ + --spec-draft-n-max 3 \ -c 16384 \ -ngl 99 -ngld 99 \ -ctk turbo3 -ctv turbo3 \ @@ -89,7 +89,7 @@ Why use the async pair? - **Graph-cache isolation**: MTP graph reuse is no longer invalidated by target decode resets. On Gemma 4 + Q4_K_XL, this alone delivered **~+8% throughput** - in single-slot benchmarks (95.3 → 102.8 tps at `--draft-block-size 3`), with + in single-slot benchmarks (95.3 → 102.8 tps at `--spec-draft-n-max 2`), with identical accept rate. - **Pipeline depth-2 (pure overlap, `llama-server`)**: after target sample/accept and `llama_memory_seq_rm`, the server calls @@ -166,7 +166,7 @@ llama-server \ -m /path/to/qwen3.6-MTP.gguf \ -md /path/to/qwen3.6-MTP.gguf \ --spec-type nextn \ - --draft-max 2 --draft-min 1 \ + --spec-draft-n-max 2 --spec-draft-n-min 1 \ -c 8192 -ngl 99 -ngld 99 -fa on \ --host 127.0.0.1 --port 8080 ``` diff --git a/gguf-py/gguf/tensor_mapping.py b/gguf-py/gguf/tensor_mapping.py index 764dc0dc9069..2dda6183a37f 100644 --- a/gguf-py/gguf/tensor_mapping.py +++ b/gguf-py/gguf/tensor_mapping.py @@ -2415,18 +2415,6 @@ class TensorNameMap: "model.layers.{bid}.shared_head.norm", ), - MODEL_TENSOR.MTP_PRE_PROJECTION: ( - "pre_projection", - ), - MODEL_TENSOR.MTP_POST_PROJECTION: ( - "post_projection", - ), - MODEL_TENSOR.MTP_CENTROIDS: ( - "masked_embedding.centroids", - ), - MODEL_TENSOR.MTP_TOKEN_ORDERING: ( - "masked_embedding.token_ordering", - ), } # architecture-specific block mappings diff --git a/scripts/run-gemma4-31b-mtp-server.sh b/scripts/run-gemma4-31b-mtp-server.sh index 0a428f8d4b2a..46e3adb73b9f 100755 --- a/scripts/run-gemma4-31b-mtp-server.sh +++ b/scripts/run-gemma4-31b-mtp-server.sh @@ -86,9 +86,10 @@ if [[ "$SPEC" == "mtp" ]]; then ARGS+=( --mtp-head "$DRAFT" --spec-type mtp - --draft-block-size "${DRAFT_BLOCK_SIZE:-3}" - --draft-max "${DRAFT_MAX:-16}" - --draft-min "${DRAFT_MIN:-0}" + # b10018 sync: --draft-block-size/--draft-max are gone; per-round chain + # length is --spec-draft-n-max (= old block size B - 1). + --spec-draft-n-max "$(( ${DRAFT_BLOCK_SIZE:-3} - 1 ))" + --spec-draft-n-min "${DRAFT_MIN:-0}" ) else echo "info: speculative decoding disabled (SPEC=${SPEC}); running baseline" >&2 diff --git a/scripts/run-gemma4-e2b-mtp-server.sh b/scripts/run-gemma4-e2b-mtp-server.sh index 0b2947d43ebd..344936b2a1e3 100755 --- a/scripts/run-gemma4-e2b-mtp-server.sh +++ b/scripts/run-gemma4-e2b-mtp-server.sh @@ -111,9 +111,10 @@ if [[ "$SPEC" == "mtp" ]]; then ARGS+=( --mtp-head "$DRAFT" --spec-type mtp - --draft-block-size "${DRAFT_BLOCK_SIZE}" - --draft-max "${DRAFT_MAX}" - --draft-min "${DRAFT_MIN:-0}" + # b10018 sync: --draft-block-size/--draft-max are gone; per-round chain + # length is --spec-draft-n-max (= old block size B - 1). + --spec-draft-n-max "$(( ${DRAFT_BLOCK_SIZE} - 1 ))" + --spec-draft-n-min "${DRAFT_MIN:-0}" ) else echo "info: speculative decoding disabled (SPEC=${SPEC}); running baseline" >&2 diff --git a/scripts/run-gemma4-e4b-mtp-server.sh b/scripts/run-gemma4-e4b-mtp-server.sh index d937cc085249..02f56468b54b 100755 --- a/scripts/run-gemma4-e4b-mtp-server.sh +++ b/scripts/run-gemma4-e4b-mtp-server.sh @@ -117,9 +117,10 @@ if [[ "$SPEC" == "mtp" ]]; then ARGS+=( --mtp-head "$DRAFT" --spec-type mtp - --draft-block-size "${DRAFT_BLOCK_SIZE}" - --draft-max "${DRAFT_MAX}" - --draft-min "${DRAFT_MIN:-0}" + # b10018 sync: --draft-block-size/--draft-max are gone; per-round chain + # length is --spec-draft-n-max (= old block size B - 1). + --spec-draft-n-max "$(( ${DRAFT_BLOCK_SIZE} - 1 ))" + --spec-draft-n-min "${DRAFT_MIN:-0}" ) else echo "info: speculative decoding disabled (SPEC=${SPEC}); running baseline" >&2 diff --git a/scripts/run-gemma4-mtp-server.sh b/scripts/run-gemma4-mtp-server.sh index a00b03b8cea1..348f01c3644c 100755 --- a/scripts/run-gemma4-mtp-server.sh +++ b/scripts/run-gemma4-mtp-server.sh @@ -87,9 +87,10 @@ if [[ "$SPEC" == "mtp" ]]; then ARGS+=( --mtp-head "$DRAFT" --spec-type mtp - --draft-block-size "${DRAFT_BLOCK_SIZE:-3}" - --draft-max "${DRAFT_MAX:-16}" - --draft-min "${DRAFT_MIN:-0}" + # b10018 sync: --draft-block-size/--draft-max are gone; per-round chain + # length is --spec-draft-n-max (= old block size B - 1). + --spec-draft-n-max "$(( ${DRAFT_BLOCK_SIZE:-3} - 1 ))" + --spec-draft-n-min "${DRAFT_MIN:-0}" ) else echo "info: speculative decoding disabled (SPEC=${SPEC}); running baseline" >&2 diff --git a/scripts/run-gemma4.sh b/scripts/run-gemma4.sh index b5a4c6ad4b15..cd15dfbd34c8 100755 --- a/scripts/run-gemma4.sh +++ b/scripts/run-gemma4.sh @@ -108,9 +108,10 @@ if [[ "$SPEC" == "mtp" ]]; then -ctvd "$CTVD" --mtp-head "$DRAFT" --spec-type mtp - --draft-block-size "${DRAFT_BLOCK_SIZE:-4}" - --draft-max "${DRAFT_MAX:-16}" - --draft-min "${DRAFT_MIN:-0}" + # b10018 sync: --draft-block-size/--draft-max are gone; per-round chain + # length is --spec-draft-n-max (= old block size B - 1). + --spec-draft-n-max "$(( ${DRAFT_BLOCK_SIZE:-4} - 1 ))" + --spec-draft-n-min "${DRAFT_MIN:-0}" ) fi diff --git a/scripts/run-qwen36-27b-nextn-server.sh b/scripts/run-qwen36-27b-nextn-server.sh index a9c3f362a8bf..d82605a752a4 100755 --- a/scripts/run-qwen36-27b-nextn-server.sh +++ b/scripts/run-qwen36-27b-nextn-server.sh @@ -69,8 +69,8 @@ if [[ "$SPEC" == "nextn" ]]; then ARGS+=( -md "$DRAFT" --spec-type nextn - --draft-max "${DRAFT_MAX:-16}" - --draft-min "${DRAFT_MIN:-0}" + --spec-draft-n-max "${DRAFT_MAX:-2}" + --spec-draft-n-min "${DRAFT_MIN:-1}" ) else echo "info: speculative decoding disabled (SPEC=${SPEC}); running baseline" >&2 diff --git a/src/llama-model.cpp b/src/llama-model.cpp index 587544619331..fe3780ee1d91 100644 --- a/src/llama-model.cpp +++ b/src/llama-model.cpp @@ -330,6 +330,13 @@ llama_model * llama_model_create(llm_arch arch, const llama_model_params & param llama_model * llama_model_create(llama_model_loader & ml, const llama_model_params & params) { llm_arch arch = ml.get_arch(); if (arch == LLM_ARCH_UNKNOWN) { + if (ml.get_arch_name() == "gemma4_assistant") { + // pre-b10018 fork assistant GGUFs use a different arch name, kv + // namespace and tensor layout than the upstream gemma4-assistant + // support; they cannot be loaded by this build. + throw std::runtime_error("this Gemma 4 assistant GGUF predates the b10018 sync and is no longer supported — " + "re-download the regenerated assistant or reconvert it with conversion/gemma.py"); + } throw std::runtime_error("unknown model architecture: '" + ml.get_arch_name() + "'"); }