perf(bonsai): gate/up stay in their ternary blocks, and packed batches past 8 rows read the whole FFN there on the int8 tensor cores (1.43x cb-decode @c16 on Ternary-Bonsai-2) by FranDev132 · Pull Request #1161 · gittensor-ai-lab/sparkinfer

FranDev132 · 2026-09-24T23:46:48Z

Summary

Ternary-Bonsai-2's default load keeps ffn_gate and ffn_up only in their stored PTQ1_0 blocks (0.22 B/weight), not in the Q4_K refit of the un-rotated weights (0.56 B/weight). Gate and up are already ternary, so the decode shadow's FFN copy is now down's alone. That is 3.9 GB less resident.

Packed decode above 8 rows. At these widths the shadow's dp4a GEMMs stop paying, and the step went back to the Q4_K fold for the whole FFN. Now hn is rotated into the weights' basis and quantized to int8 once, one scale per 128 values (the weight block's granularity, so each block's dot product is exact in int32). Gate and up run in one int8 tensor-core launch (mma.m16n8k32 u8 x s8, weights and activation staged by cp.async) that decodes each weight block once for every row. SwiGLU is folded into the FFN-width rotation. Down reads the shadow's ternary copy with k split 4 ways; its 5120 rows are only 40 CTAs, and at 32 rows the split takes it from 71 to 26.5 us.
Up to 8 rows and single-row decode keep the shadow's kernels unchanged, so every decode axis reads exactly as main.
Prefill. The activation is rotated and row-quantized in one pass. The fused quantized-B GEMM gets a PTQ1 arm (int8 = trit x round(s_b / row_scale)) that is byte-identical to the materialize kernel used past its M limit, which also feeds the gate+up+SwiGLU epilogue GEMM.

SPARKINFER_BONSAI_NATIVE=0 restores the all-folded load. SPARKINFER_BONSAI_CB_TERNARY_DOWN=0 keeps the folded down in the packed step.

Proof of speedup

Tested on RTX 5090 (sm_120)

Target model(s)

Muse Glimmer
Qwen3.8-27B (ModelOpt NVFP4 / DSpark)
Ternary-Bonsai-2-27B (PTQ1_0 GGUF)
Shared / all models — the change is in code every model uses and should help more than one

Decode tok/s (end-to-end, from bench/scripts/bench.sh — fill if this PR targets decode):

	decode tok/s
before (main)	134.81
after (this PR)	134.71

Prefill pp tok/s (fill if this PR targets prefill; use --ctx 4096, 32768, 65536, or
131072 and copy the prefill pp line — report your best context. Keep the row labels as-is):

	prefill pp tok/s
before prefill (main)	6591.9
after prefill (this PR)	7001.9

Best prefill context is 512. The numbers come from the bot's invocations on one RTX 5090, main 20a4fbd against this branch, with runs interleaved main, PR, PR, main. A sample was re-run if another process touched the GPU during it.

qwen3_gguf_bench <gguf> 128 sweep (real prompt ids, REPS=3):

ctx	decode, main	decode, this PR		prefill, main	prefill, this PR
128	134.81	134.71	-0.1%	4691	4868	+3.8%
512	133.92	133.83	-0.1%	6592	7002	+6.2%
4k	130.81	130.72	-0.1%	9607	9737	+1.4%
16k	124.97	124.87	-0.1%	9349	9377	+0.3%
32k	116.12	115.99	-0.1%	8678	8675	-0.0%

qwen3_gguf_cb_bench <gguf> c 256 256 512, aggregate tok/s, 0 request errors:

	main	this PR
c2	211.2	211.2	-0.0%
c4	366.7	368.4	+0.5%
c8	561.1	568.5	+1.3%
c16	812.4	1163.1	+43.2%
c32	1100.2	1549.9	+40.9%

Accuracy. qwen3_gguf_score over the bot's eval_corpus.txt, this PR against main (mean of 4 cross pairs): top-1 0.984, KL 0.0051, PPL 3.677 vs 3.672 (x1.001). Main against itself on the same box reads top-1 0.984 / KL 0.0044.

qwen3_gguf_prefill_check (batched prefill against the token loop, 64 continuation positions, mean of 2 runs, top-1 / KL):

prefix	main	this PR
128	1.000 / 0.0017	1.000 / 0.0018
1024	0.930 / 0.0049	0.961 / 0.0032

eval/bonsai_regression.py serve check (a row that outlives its batch against the same row asked alone): matches on both the default and SPARKINFER_BONSAI_NATIVE=all servers.

# Ternary-Bonsai-2-27B-PTQ1_0.gguf, RTX 5090, main 20a4fbd -> this PR
B main ctx=128 SWEEP_JSON {"128":{"decode_tps":134.8175,"prefill_pp":4698.6830}}
B PR ctx=128 SWEEP_JSON {"128":{"decode_tps":134.6981,"prefill_pp":4885.6308}}
B PR ctx=128 SWEEP_JSON {"128":{"decode_tps":134.7158,"prefill_pp":4849.4336}}
B main ctx=128 SWEEP_JSON {"128":{"decode_tps":134.8034,"prefill_pp":4683.2910}}
B main ctx=512 SWEEP_JSON {"512":{"decode_tps":133.9004,"prefill_pp":6588.4160}}
B PR ctx=512 SWEEP_JSON {"512":{"decode_tps":133.8190,"prefill_pp":7004.0913}}
B PR ctx=512 SWEEP_JSON {"512":{"decode_tps":133.8433,"prefill_pp":6999.7008}}
B main ctx=512 SWEEP_JSON {"512":{"decode_tps":133.9323,"prefill_pp":6595.3646}}
B main ctx=4096 SWEEP_JSON {"4096":{"decode_tps":130.8005,"prefill_pp":9616.8458}}
B PR ctx=4096 SWEEP_JSON {"4096":{"decode_tps":130.7359,"prefill_pp":9734.6070}}
B PR ctx=4096 SWEEP_JSON {"4096":{"decode_tps":130.7081,"prefill_pp":9740.0523}}
B main ctx=4096 SWEEP_JSON {"4096":{"decode_tps":130.8105,"prefill_pp":9596.5546}}
B main ctx=16384 SWEEP_JSON {"16384":{"decode_tps":124.9225,"prefill_pp":9352.9178}}
B PR ctx=16384 SWEEP_JSON {"16384":{"decode_tps":124.8359,"prefill_pp":9370.4960}}
B PR ctx=16384 SWEEP_JSON {"16384":{"decode_tps":124.9061,"prefill_pp":9383.1438}}
B main ctx=16384 SWEEP_JSON {"16384":{"decode_tps":125.0079,"prefill_pp":9345.7490}}
B main ctx=32768 SWEEP_JSON {"32768":{"decode_tps":116.1106,"prefill_pp":8659.7667}}
B PR ctx=32768 SWEEP_JSON {"32768":{"decode_tps":115.9878,"prefill_pp":8646.3204}}
B PR ctx=32768 SWEEP_JSON {"32768":{"decode_tps":115.9977,"prefill_pp":8704.5106}}
B main ctx=32768 SWEEP_JSON {"32768":{"decode_tps":116.1275,"prefill_pp":8695.2529}}
C main cc=2 agg=211.2 errs=0 tok=520
C PR cc=2 agg=210.9 errs=0 tok=520
C PR cc=2 agg=211.4 errs=0 tok=520
C main cc=2 agg=211.3 errs=0 tok=520
C main cc=4 agg=366.9 errs=0 tok=1032
C PR cc=4 agg=368.7 errs=0 tok=1032
C PR cc=4 agg=368.2 errs=0 tok=1032
C main cc=4 agg=366.5 errs=0 tok=1032
C main cc=8 agg=561.0 errs=0 tok=2056
C PR cc=8 agg=568.7 errs=0 tok=2056
C PR cc=8 agg=568.4 errs=0 tok=2056
C main cc=8 agg=561.2 errs=0 tok=2056
C main cc=16 agg=812.0 errs=0 tok=4104
C PR cc=16 agg=1163.8 errs=0 tok=4104
C PR cc=16 agg=1162.4 errs=0 tok=4104
C main cc=16 agg=812.8 errs=0 tok=4104
C main cc=32 agg=1108.1 errs=0 tok=8200
C PR cc=32 agg=1549.4 errs=0 tok=8200
C PR cc=32 agg=1550.4 errs=0 tok=8200
C main cc=32 agg=1092.4 errs=0 tok=8200

skyrocket2026 · 2026-09-25T02:45:02Z

sparkinfer bonsai auto-eval — `eval-bonsai:XL`

metric	value
label	`eval-bonsai:XL`
model	Ternary-Bonsai-2-27B PTQ1_0 GGUF, default (folded) loader
scored at	decode + prefill @ 128/512/4k/16k/32k · concurrent decode @ c2/c4/c8/c16/c32 — 15 axes, each also a regression floor; the label is the best
tier from	`bonsai-decode@128` (+28.3%)
accuracy vs main (teacher-forced)	✅ top-1 0.9566 (bar ≥0.93) · KL 0.01550 (bar ≤0.03) · PPL ×1.0028 of main (bar ≤1.02) over 1177 tokens
prefill path vs main	✅ batched prefill within main's spread at prefix 128/1024
bonsai_regression.py	✅ tensors · score · generate · serve
qwen3.6 guard	✅ no regression (decode + prefill @ 32k)
modelopt qwen3.8 guard	✅ no regression (decode + prefill @ 32k)
unsloth qwen3.8 guard	✅ no regression (decode + prefill @ 32k)
muse glimmer guard	✅ no regression (decode + prefill @ 32k)
PPL PR / main	3.6999 / 3.6895
Polaris receipt	collected, not signed (no key configured)
commit	`67b9e5e24`, measured merged onto `main` `053ef27` (this round's baseline)

ctx	phase	main	PR	delta
128	decode	99.01	127.01	+28.3%
128	prefill	4939.89	5126.23	+3.8%
512	decode	98.38	126.09	+28.2%
512	prefill	6611.15	7007.25	+6.0%
4k	decode	96.84	123.62	+27.7%
4k	prefill	8920.29	9119.79	+2.2%
16k	decode	93.55	118.35	+26.5%
16k	prefill	8924.31	8996.71	+0.8%
32k	decode	88.59	110.51	+24.7%
32k	prefill	8454.70	8496.10	+0.5%

Concurrent decode — aggregate tok/s with N requests in flight, median of 3 runs

concurrency	main	PR	delta
c2	168.90	205.30	+21.6%
c4	301.20	376.90	+25.1%
c8	530.00	674.00	+27.2%
c16	822.20	1013.50	+23.3%
c32	1115.60	1389.10	+24.5%

Prefill path — batched prefill against the token loop, 64 teacher-forced positions, mean of 3 runs per side

prefix	main top-1	PR top-1	top-1 bar	main KL	PR KL	KL bar
128	0.984	0.984	≥0.884	0.0055	0.0039	≤0.0555
1024	0.969	0.964	≥0.869	0.0048	0.0046	≤0.0548

ok

_{Measured on the pinned RTX 5090 against a same-box origin/main from the same round. Any axis regressing below 98% of main is a REJECT; otherwise the label is the best measured delta. none only means no Ternary-Bonsai-2-27B speedup was measured, which is expected for a change aimed at another model. The round's best passing speedup is auto-merged as bonsai-merge-first (only at the exact commit scored); a separate comment says so. It does not close PRs.}

@c16

…s past 8 rows read the whole FFN there on the int8 tensor cores (1.43x cb-decode @c16 on Ternary-Bonsai-2) The default Ternary-Bonsai-2 load keeps ffn_gate/ffn_up in their stored PTQ1_0 blocks (0.22 B/weight) instead of the Q4_K refit of the un-rotated weights (0.56 B/weight); the decode shadow then holds only down's ternary copy. 3.9 GB less resident. - packed decode above 8 rows, where the shadow's dp4a GEMMs stop paying and the step went back to the Q4_K fold: hn rotated and int8-quantized once (one scale per 128, the weight block's granularity), gate and up in one int8 tensor-core launch (mma.m16n8k32 u8 x s8, cp.async-staged weights and activation), SwiGLU folded into the FFN-width rotation, then down from the shadow's ternary copy with k split 4 ways (5120 rows are only 40 CTAs). Up to 8 rows keep the shadow's bit-exact path; single-row decode is unchanged. - prefill: the activation rotated and row-quantized in one pass; the fused quantized-B GEMM gets a PTQ1 arm, byte-identical to the materialize kernel used past its M limit, which also feeds the gate+up+SwiGLU epilogue GEMM. SPARKINFER_BONSAI_NATIVE=0 restores the all-folded load; SPARKINFER_BONSAI_CB_TERNARY_DOWN=0 keeps the folded down in the packed step.

skyrocket2026 · 2026-09-25T10:34:19Z

sparkinfer bonsai auto-eval — `eval-bonsai:XL`

metric	value
label	`eval-bonsai:XL`
model	Ternary-Bonsai-2-27B PTQ1_0 GGUF, default (folded) loader
scored at	decode + prefill @ 128/512/4k/16k/32k · concurrent decode @ c2/c4/c8/c16/c32 — 15 axes, each also a regression floor; the label is the best
tier from	`bonsai-cb-decode@c16` (+43.1%)
accuracy vs main (teacher-forced)	✅ top-1 0.9864 (bar ≥0.93) · KL 0.00487 (bar ≤0.03) · PPL ×0.9970 of main (bar ≤1.02) over 1177 tokens
prefill path vs main	✅ batched prefill within main's spread at prefix 128/1024
bonsai_regression.py	⚠️ not gated — main fails it this round (the PR fails it); a failure already on main cannot reject a PR
qwen3.6 guard	✅ no regression (decode + prefill @ 32k)
modelopt qwen3.8 guard	✅ no regression (decode + prefill @ 32k)
unsloth qwen3.8 guard	✅ no regression (decode + prefill @ 32k)
muse glimmer guard	✅ no regression (decode + prefill @ 32k)
PPL PR / main	3.6713 / 3.6824
Polaris receipt	collected, not signed (no key configured)
commit	`4b270325c`, measured merged onto `main` `20a4fbd` (this round's baseline)

ctx	phase	main	PR	delta
128	decode	132.69	132.72	+0.0%
128	prefill	4932.59	5163.48	+4.7%
512	decode	131.98	131.92	-0.0%
512	prefill	6606.73	7043.06	+6.6%
4k	decode	129.40	129.21	-0.1%
4k	prefill	8997.80	9164.29	+1.9%
16k	decode	123.80	123.54	-0.2%
16k	prefill	8599.87	8675.32	+0.9%
32k	decode	115.18	114.97	-0.2%
32k	prefill	7972.11	8011.42	+0.5%

Concurrent decode — aggregate tok/s with N requests in flight, median of 3 runs

concurrency	main	PR	delta
c2	207.70	208.40	+0.3%
c4	361.00	363.50	+0.7%
c8	555.00	563.10	+1.5%
c16	801.50	1146.80	+43.1%

_{Concurrency c32 not scored this round — no paired measurement (a width that fails to run is dropped, never counted as a regression).}

Prefill path — batched prefill against the token loop, 64 teacher-forced positions, mean of 3 runs per side

prefix	main top-1	PR top-1	top-1 bar	main KL	PR KL	KL bar
128	0.990	0.995	≥0.890	0.0069	0.0041	≤0.0569
1024	0.969	0.948	≥0.869	0.0065	0.0055	≤0.0565

ok

_{Measured on the pinned RTX 5090 against a same-box origin/main from the same round. Any axis regressing below 98% of main is a REJECT; otherwise the label is the best measured delta. none only means no Ternary-Bonsai-2-27B speedup was measured, which is expected for a change aimed at another model. The round's best passing speedup is auto-merged as bonsai-merge-first (only at the exact commit scored); a separate comment says so. It does not close PRs.}

skyrocket2026 · 2026-09-25T10:34:26Z

Auto-merged as the round's bonsai-merge-first winner — verified same-box Ternary-Bonsai-2-27B speedup over main, with every accuracy gate and cross-model guard passing.

skyrocket2026 · 2026-09-25T10:34:39Z

Note on this run's environment (not a PR result). This round ran on a newly provisioned eval box, and two gaps came from the box, not from this PR:

bonsai_regression.py could not run there (No module named 'safetensors'; transformers was also missing). main failed it the same way, so it was not gated for this PR or any other in the round. Both modules are installed now.
Concurrent decode at c32 produced no paired measurement and was dropped; the bot never counts a width that fails to run as a regression. It was not explained in this run.

Neither is a finding against this PR. The tier comes from the measured concurrent-decode gain at c16 (+43.1%), with every accuracy gate and cross-model guard passing.

skyrocket2026 added eval-bonsai:XL sparkinfer Ternary-Bonsai-2-27B vs-main speed tier: XL eval:XL eval verdict parked by a 3-day noise ban — restores to eval:XL labels Sep 25, 2026

skyrocket2026 added the bonsai-needs-rebase conflicts with main, or lost this round's bonsai-merge-first label Sep 25, 2026

FranDev132 force-pushed the perf/bonsai-native-gate-up branch from 67b9e5e to 4b27032 Compare September 25, 2026 08:04

skyrocket2026 added eval-bonsai:XL sparkinfer Ternary-Bonsai-2-27B vs-main speed tier: XL and removed eval-bonsai:XL sparkinfer Ternary-Bonsai-2-27B vs-main speed tier: XL bonsai-needs-rebase conflicts with main, or lost this round's bonsai-merge-first labels Sep 25, 2026

skyrocket2026 added the bonsai-merge-first round winner: biggest verified Ternary-Bonsai-2-27B speedup — auto-merge candidate label Sep 25, 2026

skyrocket2026 merged commit dc723a4 into gittensor-ai-lab:main Sep 25, 2026
5 checks passed

Navigation Menu

perf(bonsai): gate/up stay in their ternary blocks, and packed batches past 8 rows read the whole FFN there on the int8 tensor cores (1.43x cb-decode @c16 on Ternary-Bonsai-2) - #1161

perf(bonsai): gate/up stay in their ternary blocks, and packed batches past 8 rows read the whole FFN there on the int8 tensor cores (1.43x cb-decode @c16 on Ternary-Bonsai-2)#1161
skyrocket2026 merged 1 commit into
gittensor-ai-lab:mainfrom
FranDev132:perf/bonsai-native-gate-up

FranDev132 commented Sep 24, 2026 •

edited

Loading

Uh oh!

skyrocket2026 commented Sep 25, 2026

Uh oh!

skyrocket2026 commented Sep 25, 2026

Uh oh!

Uh oh!

skyrocket2026 commented Sep 25, 2026

Uh oh!

skyrocket2026 commented Sep 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

FranDev132 commented Sep 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Proof of speedup

Uh oh!

skyrocket2026 commented Sep 25, 2026

sparkinfer bonsai auto-eval — eval-bonsai:XL

Uh oh!

skyrocket2026 commented Sep 25, 2026

sparkinfer bonsai auto-eval — eval-bonsai:XL

Uh oh!

Uh oh!

skyrocket2026 commented Sep 25, 2026

Uh oh!

skyrocket2026 commented Sep 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

FranDev132 commented Sep 24, 2026 •

edited

Loading

sparkinfer bonsai auto-eval — `eval-bonsai:XL`

sparkinfer bonsai auto-eval — `eval-bonsai:XL`