Help wanted: published Gemma 3 vindex hydration + Gemma 4 extraction sanity check
Hi! I'm experimenting with LarQL and trying to reproduce the README happy path, then build an equivalent vindex for Gemma 4. I may be using the wrong workflow, but I'm seeing two confusing things.
1. Gemma 3 published vindex: inference files exist on HF, but local pull/run misses them
The README suggests:
larql pull hf://chrishayuk/gemma-3-4b-it-vindex
larql run gemma-3-4b-it-vindex "The capital of France is"
The HF repo appears to include inference files such as:
attn_weights_q4k.bin
interleaved_q4k.bin
lm_head_q4.bin
norms.bin
weight_manifest.json
But after larql pull, my local snapshot only has the browse/core files:
down_meta.bin
embeddings.bin
feature_labels.json
gate_vectors.bin
index.json
tokenizer.json
Then larql run ... fails with:
Error: parse error: attention k-quant weights not found
(looked for attn_weights_kquant.bin and legacy attn_weights_q4k.bin)
So my question is: should larql pull hydrate the inference files for a full vindex, or should larql run hf://... lazily call the weight-download path before loading?
2. Locally extracted Gemma 4 vindex loads, but inference looks implausible
I extracted google/gemma-4-12B-it from HF safetensors:
larql extract <hf-snapshot> \
--output gemma-4-12b-it.q4k-inference.vindex \
--level inference \
--quant q4k \
--drop-gate-vectors \
--resume
The vindex verifies, serves, and reports inference loaded. But /v1/infer for:
returns flat/unrelated top tokens like:
" f", "?", "en", "is", " in", " of"
The same model in LM Studio answers "The capital of France is Paris."
Questions
- What is the intended command sequence to reproduce the README
INFER "The capital of France is" -> Paris example from the published HF vindex?
- Should
larql pull download VINDEX_WEIGHT_FILES, or is there another command I should run before inference?
- Is the
server preset intentionally FFN-only/no-attention, meant to pair with a client slice?
- Is dense
google/gemma-4-12B-it expected to work with larql extract --quant q4k today?
- If Gemma 4 should work, what diagnostic path would you recommend for implausible logits?
Thanks! LarQL is a very interesting project; I'm trying to separate a docs/artifact/hydration issue from user error.
Help wanted: published Gemma 3 vindex hydration + Gemma 4 extraction sanity check
Hi! I'm experimenting with LarQL and trying to reproduce the README happy path, then build an equivalent vindex for Gemma 4. I may be using the wrong workflow, but I'm seeing two confusing things.
1. Gemma 3 published vindex: inference files exist on HF, but local pull/run misses them
The README suggests:
larql pull hf://chrishayuk/gemma-3-4b-it-vindex larql run gemma-3-4b-it-vindex "The capital of France is"The HF repo appears to include inference files such as:
But after
larql pull, my local snapshot only has the browse/core files:Then
larql run ...fails with:So my question is: should
larql pullhydrate the inference files for a full vindex, or shouldlarql run hf://...lazily call the weight-download path before loading?2. Locally extracted Gemma 4 vindex loads, but inference looks implausible
I extracted
google/gemma-4-12B-itfrom HF safetensors:The vindex verifies, serves, and reports inference loaded. But
/v1/inferfor:returns flat/unrelated top tokens like:
The same model in LM Studio answers "The capital of France is Paris."
Questions
INFER "The capital of France is" -> Parisexample from the published HF vindex?larql pulldownloadVINDEX_WEIGHT_FILES, or is there another command I should run before inference?serverpreset intentionally FFN-only/no-attention, meant to pair with a client slice?google/gemma-4-12B-itexpected to work withlarql extract --quant q4ktoday?Thanks! LarQL is a very interesting project; I'm trying to separate a docs/artifact/hydration issue from user error.