Skip to content

Offer MTP drafter download after quant selection #2

Description

@eeshansrivastava89

Context

Some GGUF repos include MTP drafter models alongside the main model quants. For example, huihui-ai/Ornith-1.0-9B-abliterated-MTP-GGUF has:

ornith-9b-mtp-kl-BF16.gguf      17.14 GB  (main model)
ornith-9b-mtp-kl-Q8_0.gguf       9.11 GB  (main model)
ornith-9b-mtp-kl-Q4_K_M.gguf     6.41 GB  (main model)
ornith-9b-mtp-kl-IQ3_M.gguf      5.58 GB  (main model)
mtp-ornith-9b-mtp-kl-Q8_0.gguf   2.26 GB  (MTP drafter)

Similarly, unsloth/gemma-4-E2B-it-GGUF has drafters in an MTP/ subdirectory:

MTP/gemma-4-E2B-it-Q8_0-MTP.gguf  97 MB
MTP/gemma-4-E2B-it-F16-MTP.gguf   170 MB
MTP/gemma-4-E2B-it-BF16-MTP.gguf  170 MB

Current behavior

  • The quant picker (in download.mjs) filters out MTP drafter files (done in v0.18.0)
  • Only the selected main model file is downloaded via hf download <repo> <filename>
  • The MTP drafter is NOT downloaded
  • The scanner (scan.mjs) can match drafters to models by filename pattern, but only if the drafter is already on disk

What to build

After the user selects a quant and before the download starts, check if the repo has MTP drafter files. If it does:

  1. Show a prompt: This model supports MTP speculative decoding (2x speedup). Also download the drafter model? (Y/n)
  2. Show the drafter file name and size (e.g. mtp-ornith-9b-mtp-kl-Q8_0.gguf (2.26 GB))
  3. If multiple drafter quants exist, pick the smallest one that fits (or let the user choose)
  4. Download both files: the main model + the drafter
  5. The scanner will automatically match them when the model is set up

Drafter detection patterns

  • Files in an MTP/ subdirectory: MTP/gemma-4-E2B-it-Q8_0-MTP.gguf
  • Files with mtp- prefix: mtp-ornith-9b-mtp-kl-Q8_0.gguf
  • Files with -MTP. suffix: gemma-4-E2B-it-Q8_0-MTP.gguf

The isDrafterFile() function in huggingface.mjs already detects these patterns. A new function could list ONLY drafter files (the inverse filter).

References

  • Drafter matching: matchDrafter() in scan.mjs
  • Drafter detection: isDrafterFile() in huggingface.mjs
  • Quant picker: pickGgufQuant() in download.mjs
  • MTP setup in glass-box flow: profile-setup.mjs (MTP prompt + drafter detection)

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions