serialize: does Tell.prefill retain the same autoreleased Metal wrappers as the lane path did? (measure before fixing)

[penta2himajin]

Follow-up from #148 / PR #149, raised by an independent read-only review of that fix. Measurement first — no fix proposed.

The observation

#148's root cause was: the steel-hybrid prefill wraps every per-layer MLX temporary in a noCopy MTLBuffer (arrToBuf → asMTLBuffer); those wrappers are autoreleased, and the pool drains only at thread exit. On the lane path that thread is ContinuousScheduler's decode thread, so a whole prefill's wrappers stayed live at once:

retained Metal ≈ concurrency × promptLen × ~2.03MB/token

Fixed for lanes by draining per prefill chunk (PR #149).

That mechanism is path-agnostic. Tell.prefill (swift/Sources/QwispCore/TellRuntime.swift, the chunk loop) runs the same steel-hybrid forwardRowsHybrid at chunk 1024, on a per-request detached thread (LLMBackend.swift), with no per-chunk pool. Serialize's pool does drain at request end, so there is no cross-request ratchet — but the in-request peak should scale with prompt length by the same law.

Why this is not obviously already known

If the coefficient were the same, a 48K-token serialize prefill would imply ~100GB of simultaneous wrapper retention, which plainly does not happen — long serialize prefills work today. So either:

• the coefficient differs materially on that path (different call shape, fewer/smaller wrapped temporaries), or
• something already bounds it there, or
• nobody has ever watched footprint during a long serialize prefill.

This contradiction is unresolved by reading alone and is exactly why this is a measurement issue, not a fix issue.

The measurement

One long single-stream prefill (e.g. 32–48K tokens) on the default serialize path, sampling footprint <pid> throughout (not ps rss — it undercounts wired GPU pages), plus MLX.Memory active/cache and MTLDevice.currentAllocatedSize if the QWISP_LANE_MEMDBG-style hook from PR #149 is generalised to that path.

Expected discriminator:

• footprint peak scales at ~MB per prompt token during prefill → same bug, quantify before fixing
• footprint stays flat / scales at KB per token → the lane path was special, close this

If it does reproduce

Note both natural fix sites (forwardRowsHybrid, Tell.prefill) are on the frozen forward path (CLAUDE.md prohibition 3), which is why PR #149 placed its drain lane-locally. A shared-point fix needs owner sign-off; a Tell.prefill-local drain mirroring PR #149 would be the lower-risk shape.

Refs

#148 (root cause + full measurement trail) · PR #149 (the lane fix) · TellRuntime.swift Tell.prefill · LLMBackend.swift (per-request thread) · SeedlessFusedVerify.swift arrToBuf / forwardRowsHybrid

[penta2himajin]

Answered, and the answer splits in two. Closing; two follow-ups filed.

Merged in #161. Summary of what was measured, since the outcome is not the one this issue was framed around.

The question this issue asked

"Does Tell.prefill retain the same autoreleased Metal wrappers the lane path did?" — yes, the mechanism is identical. Breakdown at every 4th prefill chunk (new QWISP_PREFILL_MEMDBG, mirroring QWISP_LANE_MEMDBG's contract and interpretation table), 35,178-token prefill:

• active 20,036 → 21,019MB — flat, so not live MLX references
• nonMLXmetal 25,936 → 101,645MB — ratchets, so raw MTLBuffer accumulation

Third row of the table: MLX is not the holder, Memory.cacheLimit cannot reach it, only draining works. It accumulates across Tell.prefill calls — the server prefills in 2048-token windows and nonMLXmetal rises monotonically through all of them, because the pool drains at THREAD exit and the request thread outlives every window.

And the answer that matters

Paired A/B, same binary, QWISP_PREFILL_CHUNK_POOL=0 vs 1:

	pool OFF	pool ON
nonMLXmetal	25,936 → 101,645MB	23,596 → 21,269MB (flat)
footprint peak	35,840MB	35,840MB

The drain eliminates the ratchet and buys zero physical memory. Those wrappers alias existing pages — they inflate currentAllocatedSize past 100GB without costing resident memory. It shipped as a free lifetime change that removes a bookkeeping ratchet, explicitly not as a memory fix.

So this issue's own framing — "if it reproduces, quantify before fixing" — resolved as: it reproduces, and the fix it implied is not the fix for the memory.

Where the 16GB actually goes

Attributed with no code change, by diffing footprint's per-category output between baseline and peak:

Δ MB	Δ regions	category
+15,360	+2,151	IOAccelerator (graphics) — 94%
< 800 combined	—	all MALLOC categories

GPU memory, not CPU — which refutes the prefix-blob hypothesis. Of that, MLX accounts for ~8.1GB (cache +7.7, active +0.4), leaving ~7.3GB reported by neither MLX nor MTLDevice.currentAllocatedSize.

The serialize footprint coefficient is 477 KB/token vs the lane path's 2.03 MB/token — 4.4x smaller, which is exactly why a 48K prefill lands near 41GB instead of the ~100GB the lane law predicts, i.e. why this survived unnoticed on a 64GB box. This issue's three-way contradiction resolves as option 1 (the coefficient differs materially), but for a reason the issue did not anticipate: the wrappers were never the physical cost on this path.

Method note

An earlier attribution pass ran QWISP_PREFILL_MEMDBG=2, which calls clearCache() at every report. That masked the MLX cache growth completely and produced a "cache flat" reading that was then used to attribute the leak. The observer changed the observed. Mode 1 is the setting for attribution; mode 2 answers only "does the pool give it back". Recorded so the next person does not repeat it.

Follow-ups

Both are genuinely different questions, so they are separate issues rather than a re-scope of this one:

• serialize's MLX pool is unbounded. Memory.cacheLimit is set on the lane path only (fix(seedless): lane-mode OOM — drain prefill's autoreleased Metal wrappers per chunk (#148) #149: "the serialize path is untouched"). Measured +7.7GB here. Actionable lever with a shipped precedent.
• ~7.3GB of IOAccelerator growth is unattributed and invisible to both available instruments. Needs one that can see driver-side regions.

Closing.