perf(bonsai): packed rows read the decode shadow's FFN on the int8 tensor cores, one weight read for every row (1.33x cb-decode @c8 on Ternary-Bonsai-2)

[FranDev132]

Summary

A packed step of 2 to 8 rows read the decode shadow's ternary FFN through dp4a GEMMs whose work grows with every row. The whole leg (rotations, gate, up, SwiGLU, down) took 61 us per layer at 2 rows, 75 at 4 and 109 at 8. Past 8 rows the same legs already run on the int8 tensor-core rows kernel; this sends 2 to 8 rows there too:

• Activation. hn is rotated into the weights' basis and quantized to int8 once per row, one scale per 128 values, the weight block's granularity.
• Gate and up run in one launch that decodes each weight block once for every row. SwiGLU is folded into the FFN-width rotation, and down runs with k split across CTAs, because its 5120 rows are only 40 of them.
• Batch-invariant sums. The rows kernel sums every row in one fixed order, with a split count set by the weight shape alone, so a row's result does not depend on how many rows share the step. A single row keeps the dp4a path, bit-identical to single-row decode.
• Dead work removed. xn's rotation buffer is now allocated only when a ternary projection reads it. With the default shadow (FFN only), decode rotated and quantized xn every layer for nothing.

Per layer, the whole FFN leg on the RTX 5090 (harness, every rotation and SwiGLU included on both sides):

rows	main (dp4a)	this PR
2	61.4 us	49.8 us
4	75.5 us	50.1 us
8	109.1 us	51.6 us

SPARKINFER_BONSAI_CB_SHADOW_TC=0 keeps the dp4a GEMMs.

Proof of speedup

• Tested on RTX 5090 (sm_120)

Target model(s)

• Muse Glimmer
• Qwen3.8-27B (ModelOpt NVFP4 / DSpark)
• Ternary-Bonsai-2-27B (PTQ1_0 GGUF)
• Shared / all models — the change is in code every model uses and should help more than one

Decode tok/s (end-to-end, from bench/scripts/bench.sh — fill if this PR targets decode):

	decode tok/s
before (main)	134.53
after (this PR)	136.35

Prefill pp tok/s (fill if this PR targets prefill; use --ctx 4096, 32768, 65536, or
131072 and copy the prefill pp line — report your best context. Keep the row labels as-is):

	prefill pp tok/s
before prefill (main)	6987.9
after prefill (this PR)	6990.2

Best prefill context is 512. The numbers come from the bot's invocations on one RTX 5090, main dc723a4 against this branch, with runs interleaved main, PR, PR, main. A sample was re-run if another process touched the GPU during it.

qwen3_gguf_bench <gguf> 128 sweep (real prompt ids, REPS=3):

ctx	decode, main	decode, this PR		prefill, main	prefill, this PR
128	134.53	136.35	+1.4%	4851	4841	-0.2%
512	133.60	135.45	+1.4%	6988	6990	+0.0%
4k	130.51	132.26	+1.3%	9384	9387	+0.0%

qwen3_gguf_cb_bench <gguf> c 256 256 512, aggregate tok/s, 0 request errors:

	main	this PR
c2	211.9	227.9	+7.5%
c4	368.2	431.3	+17.1%
c8	569.0	758.7	+33.4%
c16	1170.5	1162.1	-0.7%

Accuracy. qwen3_gguf_score over the bot's eval_corpus.txt, this PR against main (mean of 4 cross pairs): top-1 0.984, KL 0.0051, PPL 3.681 vs 3.672 (x1.003). Main against itself on the same box reads top-1 0.985 / KL 0.0047.

qwen3_gguf_prefill_check (batched prefill against the token loop, 64 continuation positions, mean of 2 runs, top-1 / KL):

prefix	main	this PR
128	0.984 / 0.0015	0.992 / 0.0017
1024	0.945 / 0.0040	0.984 / 0.0031

eval/bonsai_regression.py serve check (a row that outlives its batch against the same row asked alone): matches a baseline on both the folded and the native server (366/366 characters).

# Ternary-Bonsai-2-27B-PTQ1_0.gguf, RTX 5090, main dc723a4 -> this PR
B main ctx=128 SWEEP_JSON {"128":{"decode_tps":134.5268,"prefill_pp":4853.3492}}
B PR ctx=128 SWEEP_JSON {"128":{"decode_tps":136.3341,"prefill_pp":4835.9012}}
B PR ctx=128 SWEEP_JSON {"128":{"decode_tps":136.3627,"prefill_pp":4846.7208}}
B main ctx=128 SWEEP_JSON {"128":{"decode_tps":134.5243,"prefill_pp":4848.4880}}
B main ctx=512 SWEEP_JSON {"512":{"decode_tps":133.5919,"prefill_pp":6970.8280}}
B PR ctx=512 SWEEP_JSON {"512":{"decode_tps":135.4276,"prefill_pp":6997.0509}}
B PR ctx=512 SWEEP_JSON {"512":{"decode_tps":135.4643,"prefill_pp":6983.3002}}
B main ctx=512 SWEEP_JSON {"512":{"decode_tps":133.6148,"prefill_pp":7004.8994}}
B main ctx=4096 SWEEP_JSON {"4096":{"decode_tps":130.5562,"prefill_pp":9398.7993}}
B PR ctx=4096 SWEEP_JSON {"4096":{"decode_tps":132.2901,"prefill_pp":9384.1694}}
B PR ctx=4096 SWEEP_JSON {"4096":{"decode_tps":132.2208,"prefill_pp":9388.9812}}
B main ctx=4096 SWEEP_JSON {"4096":{"decode_tps":130.4646,"prefill_pp":9369.0717}}
C main cc=2 agg=211.9 errs=0 tok=520
C PR cc=2 agg=227.8 errs=0 tok=520
C PR cc=2 agg=227.9 errs=0 tok=520
C main cc=2 agg=212.0 errs=0 tok=520
C main cc=4 agg=367.9 errs=0 tok=1032
C PR cc=4 agg=431.2 errs=0 tok=1032
C PR cc=4 agg=431.4 errs=0 tok=1032
C main cc=4 agg=368.6 errs=0 tok=1032
C main cc=8 agg=568.7 errs=0 tok=2056
C PR cc=8 agg=758.3 errs=0 tok=2056
C PR cc=8 agg=759.1 errs=0 tok=2056
C main cc=8 agg=569.2 errs=0 tok=2056
C main cc=16 agg=1178.2 errs=0 tok=4104
C PR cc=16 agg=1163.0 errs=0 tok=4104
C PR cc=16 agg=1161.2 errs=0 tok=4104
C main cc=16 agg=1162.7 errs=0 tok=4104

[skyrocket2026]

sparkinfer bonsai auto-eval — eval-bonsai:XL

metric	value
label	eval-bonsai:XL
model	Ternary-Bonsai-2-27B PTQ1_0 GGUF, default (folded) loader
scored at	decode + prefill @ 128/512/4k/16k/32k · concurrent decode @ c2/c4/c8/c16/c32 — 15 axes, each also a regression floor; the label is the best
tier from	bonsai-cb-decode@c8 (+33.4%)
accuracy vs main (teacher-forced)	✅ top-1 0.9770 (bar ≥0.93) · KL 0.00652 (bar ≤0.03) · PPL ×0.9995 of main (bar ≤1.02) over 1177 tokens
prefill path vs main	✅ batched prefill within main's spread at prefix 128/1024
bonsai_regression.py	✅ tensors · score · generate · serve
qwen3.6 guard	✅ no regression (decode + prefill @ 32k)
modelopt qwen3.8 guard	✅ no regression (decode + prefill @ 32k)
unsloth qwen3.8 guard	✅ no regression (decode + prefill @ 32k)
muse glimmer guard	✅ no regression (decode + prefill @ 32k)
PPL PR / main	3.6775 / 3.6794
Polaris receipt	collected, not signed (no key configured)
commit	fa6ec8891, measured merged onto main dc723a4 (this round's baseline)

ctx	phase	main	PR	delta
128	decode	132.73	134.56	+1.4%
128	prefill	5164.05	5167.66	+0.1%
512	decode	131.94	133.69	+1.3%
512	prefill	7047.05	7051.93	+0.1%
4k	decode	129.27	130.88	+1.2%
4k	prefill	9170.47	9169.62	-0.0%
16k	decode	123.61	124.95	+1.1%
16k	prefill	8677.13	8671.74	-0.1%
32k	decode	115.07	116.30	+1.1%
32k	prefill	8020.88	8012.82	-0.1%

Concurrent decode — aggregate tok/s with N requests in flight, median of 3 runs

concurrency	main	PR	delta
c2	208.40	226.10	+8.5%
c4	363.60	427.60	+17.6%
c8	563.20	751.20	+33.4%
c16	1148.50	1148.00	-0.0%
c32	1533.80	1527.00	-0.4%

Prefill path — batched prefill against the token loop, 64 teacher-forced positions, mean of 3 runs per side

prefix	main top-1	PR top-1	top-1 bar	main KL	PR KL	KL bar
128	0.990	0.990	≥0.890	0.0043	0.0042	≤0.0543
1024	0.964	0.969	≥0.864	0.0055	0.0049	≤0.0555

ok

_{Measured on the pinned RTX 5090 against a same-box origin/main from the same round. Any axis regressing below 98% of main is a REJECT; otherwise the label is the best measured delta. none only means no Ternary-Bonsai-2-27B speedup was measured, which is expected for a change aimed at another model. The round's best passing speedup is auto-merged as bonsai-merge-first (only at the exact commit scored); a separate comment says so. It does not close PRs.}

[skyrocket2026]

Auto-merged as the round's bonsai-merge-first winner — verified same-box Ternary-Bonsai-2-27B speedup over main, with every accuracy gate and cross-model guard passing.