feat(kvcache): add reliable quantized KV cache - #309

Open

plsgivemeachane wants to merge 7 commits into

FlashML-org:mainfrom

plsgivemeachane:feat/reliable-quantized-kv-cache

Commits on Aug 31, 2026

feat(kvcache): sub-byte Q4_0 KV cache quantization

fangyuan-3149
authored and
plsgivemeachane
committed
c086b28
fix(kvcache): wire --kv-cache-dtype through the engine

fangyuan-3149
authored and
plsgivemeachane
committed
99bd90c
fix(tests): correct sub-byte test expectations and dtype mismatches

fangyuan-3149
authored and
plsgivemeachane
committed
09c9d94
fix(kvcache): correct quantized KV attention data path
plsgivemeachane
committed
aad9a16
fix(attention): keep sink normalizer in fp32
plsgivemeachane
committed
716c1a2

Commits on Sep 1, 2026

fix(cli): allow KV dtype with capacity overrides
plsgivemeachane
committed
cccac24

Commits on Sep 3, 2026

Merge branch 'FlashML-org:main' into feat/reliable-quantized-kv-cache
plsgivemeachane
authored
899bf87