Skip to content

fix(gemma4): support dense Gemma-4 GGUF checkpoints - #359

Open
Cyber-Marty wants to merge 1 commit into
FlashML-org:mainfrom
Cyber-Marty:fix/gemma4-dense-gguf
Open

fix(gemma4): support dense Gemma-4 GGUF checkpoints#359
Cyber-Marty wants to merge 1 commit into
FlashML-org:mainfrom
Cyber-Marty:fix/gemma4-dense-gguf

Conversation

@Cyber-Marty

Copy link
Copy Markdown

Summary

parse_gguf_config (python/freetoken/models/gemma4/gguf.py) unconditionally required
gemma4.expert_count and hardcoded moe_enabled=True, so a dense Gemma-4 GGUF
(google/gemma-4-12B-it, nvidia/Gemma-4-31B-IT-NVFP4) could not load at all — the
GGUF path structurally could not represent a non-MoE checkpoint, while docs/models.md
lists dense family members and GGUF is the documented non-safetensors path.

Fix

Mirror what the HF side (gemma4.config.parse_config) already does:

num_experts = int(m.get("gemma4.expert_count", 0) or 0)
num_experts_per_tok = int(m.get("gemma4.expert_used_count", 0) or 0)
moe_intermediate_size = int(m.get("gemma4.expert_feed_forward_length", 0) or 0)
moe_enabled = num_experts > 0
  • dense GGUFs (no expert keys) parse successfully and route to the dense feed-forward
    path with expert_quant="none" / moe_weight_format="none";
  • MoE GGUFs parse exactly as before (all expert fields present → moe_enabled=True,
    native-Q4_0 expert quant);
  • expert_count=0 in metadata behaves the same as an absent key.

Tests

New tests/models/test_gemma4_gguf_config.py (3 cases). The fixture writes a minimal
GGUF file (magic + version 3 + zero kv/tensor counts), so GGUFReader opens it and
_full_rotary_dim takes its metadata-only fallback — the config parser runs against
plain metadata dicts with no model download:

  • MoE keys present → moe path unchanged (moe_enabled=True, q4_0 quant);
  • keys absent → dense path, expert_quant="none", geometry intact;
  • explicit zeros → same as absent.

tests/models full run: 76 passed, 112 skipped; the 18 failures in
tests/models/qwen4_exp/test_weight.py reproduce on unmodified main (Windows
mmap/threading platform issues) and are unrelated to this change.

Fixes #357

parse_gguf_config unconditionally required gemma4.expert_count and
hardcoded moe_enabled=True, so a dense Gemma-4 GGUF (gemma-4-12B-it,
gemma-4-31B-it) could not load at all — the GGUF path structurally
could not represent a non-MoE checkpoint, while docs/models.md lists
dense family members and GGUF is the documented non-safetensors path.

Mirror the HF-side parse_config: default the expert fields to 0 when
absent from the metadata, derive moe_enabled = num_experts > 0, and
gate expert_quant/moe_weight_format on it. MoE GGUFs parse exactly as
before.

Regression tests build a minimal GGUF header (magic + version + zero
counts) so the parser runs against metadata dicts without any model
download: MoE keys present -> moe path unchanged; keys absent or zero
-> dense path with expert_quant none.

Fixes FlashML-org#357
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

gemma4 GGUF: dense Gemma-4 checkpoints cannot load — parse_gguf_config requires gemma4.expert_count and hardcodes moe_enabled=True

1 participant