[Usage]: KVarN KV cache dtype kvarn_k4v2_g128 only works with head_dim=128?

[ManikandanBalasubramanian]

Your current environment

Description: I'm trying to serve GLM-4.7-Flash with KVarN's kvarn_k4v2_g128 KV cache dtype, but it fails with a head_dim validation error. The model has head_dim=576, which is incompatible with the KVarN dtype requirement.

Question: Is kvarn_k4v2_g128 designed to work only with models having head_dim=128? Or am I missing a configuration step?

Command

python -m vllm.entrypoints.openai.api_server \
  --model /home/mani/workspace/hfModels/GLM-4.7-Flash \
  --host 0.0.0.0 \
  --port 8001 \
  --tensor-parallel-size 2 \
  --block-size 128 \
  --kv-cache-dtype kvarn_k4v2_g128

Error

(APIServer pid=406685) pydantic_core._pydantic_core.ValidationError: 1 validation error for VllmConfig
(APIServer pid=406685)   Value error, kvarn_k4v2_g128 requires head_dim=128, but this model has head_dim=576. KVarN currently supports head_dim=128 only; use a different --kv-cache-dtype for this model. [type=value_error, input_value=ArgsKwargs((), {'model_co... 'shutdown_timeout': 0}), input_type=ArgsKwargs]
(APIServer pid=406685)     For further information visit https://errors.pydantic.dev/2.12/v/value_error

Model Info

• Model: GLM-4.7-Flash
• Architecture: Glm4MoeLiteForCausalLM
• head_dim: 576

Additional Question: Is there a workaround, or should KVarN dtypes be documented as head_dim-specific?

[philippebich]

No, you are not missing anything. I am fixing it right now. I will ping as soon as its done. Sorry for this :/

[borisk1]

Same here.

[philippebich]

Same here.

Which model are you trying to use? GLM-4.7-Flash has MLA which is more than just a head_dim fix. But I am running final tests also there

[ElSnacko]

Hi, just wanted to added in here, I am trying to run Qwen3.6-27B.
head_dim is 256.

[philippebich]

Thanks, you are helping a lot to find issues. I am working on them. Try to be back as soon as possible! Thanks a lot for the help :)

[borisk1]

Same here.

Which model are you trying to use? GLM-4.7-Flash has MLA which is more than just a head_dim fix. But I am running final tests also there

Qwen3.6-27B and gemma-4-31B.

[FaisalBiyari]

google/gemma-4-31B-it has text_config.head_dim = 256 and text_config.global_head_dim = 512.

On GPUs with a 64 KiB shared-memory limit, Gemma 4 requires TILE_SIZE_PREFILL = 16 to initialize. But that is not specifically related to KVarN.

[philippebich]

FYI I did not forget about this. I am doing the final tests, by the end of today or by tomorrow it should be fixed.

[philippebich]

I managed to create an MLA adaptation of our method, so KVarN now also works with MLA models.

To the best of our knowledge, KVarN is the first sub-8-bit KV-cache quantization method in vLLM to support MLA-based models. For MLA, we quantize the compressed KV latent to int4.

You can run it with the same --kv-cache-dtype used for dense models. On an MLA model, KVarN automatically routes to the MLA latent path, with no code or environment changes needed:

vllm serve zai-org/GLM-4.7-Flash \
    --kv-cache-dtype kvarn_k4v2_g128 \
    --block-size 128 \
    --tensor-parallel-size 2

On GLM-4.7-Flash with TP=2, we get:

Metric	bf16	KVarN	KVarN / bf16
Burst throughput @32k	401 tok/s	364 tok/s	0.94x
KV-cache capacity	313K tokens	865K tokens	2.77x
AIME25 accuracy	53.3%	53.3%	parity

MLA already has a much smaller latent KV representation, so it is less memory-bound than standard GQA/MHA attention. This makes it harder to match or exceed bf16 throughput on MLA models compared to standard KV-cache models.

That said, the current implementation already gives almost 3x more KV-cache capacity with parity accuracy.

I will keep optimizing the MLA path further, especially on the throughput side.

Next: fix for the remaining models.

Note: this implementation is the best I could do in this small timeframe. For sure we can do better, I will do my best to improve it in the following days.

[ElSnacko]

Hey, great stuff man, I am really excited to see this in VLLM!
I had one question, is there a reason you went for AIME25 accuracy instead of KL?

[ManikandanBalasubramanian]

@philippebich This is really impressive work. Applying KV compression on top of an MLA architecture is something I haven't seen much, since most quantization approaches tend to avoid touching the KV cache in latent space.
Achieving a 2.77× increase in context capacity without accuracy degradation is a remarkable result.
I'm really looking forward to seeing this integrated into vLLM. Great job

[philippebich]

Hey, great stuff man, I am really excited to see this in VLLM! I had one question, is there a reason you went for AIME25 accuracy instead of KL?

Thanks :) Yes, KL divergence is not a bad metric, but with KV-cache quantization it can be easy to look good on KL if you do not monitor how it evolves over time.

Our KVarN paper is designed to address what happens in reasoning, agentic, and long-context scenarios, where quantization errors can accumulate over long-horizon decoding. We also try to do this without any calibration.

So KL is a useful proxy, but what people ultimately want to see is that accuracy on standard reasoning benchmarks remains intact.

Paper: https://huggingface.co/papers/2606.03458

[philippebich]

@philippebich This is really impressive work. Applying KV compression on top of an MLA architecture is something I haven't seen much, since most quantization approaches tend to avoid touching the KV cache in latent space. Achieving a 2.77× increase in context capacity without accuracy degradation is a remarkable result. I'm really looking forward to seeing this integrated into vLLM. Great job

We are currently talking with the vLLM folks and will try to upstream this.

There are still a few things to fix, including support for some of the models people asked about here, but at least that part should be done today.

Thanks for the support, very much appreciated :)