Observation
Memory.cacheLimit is bound on the lane path only. From #149's own comment: "Lane-scoped (LaneBackend is only built under QWISP_LANES); the serialize path is untouched." Default QWISP_LANE_MLX_CACHE_MB = 2048.
The default serialize path therefore runs with MLX's free-buffer pool unbounded, and #150 measured what that costs: on a 35,178-token prefill the pool grew +7.7GB (cache 4,845 → 12,523MB), roughly half of the 16GB total footprint growth. Confirmed by the observer effect that hid it: with QWISP_PREFILL_MEMDBG=2 (which calls clearCache() every report) the same prefill peaked ~6GB lower.
Why this is not obviously a win
On lanes, bounding the pool measured −22.5GB peak AND ~10% faster in the crash configuration (#149) — the speed came from relieving memory pressure on a 64GB box. Neither number transfers:
- serialize holds one request's working set, not B lanes' — the pool may be recycling buffers that a limit would force it to re-allocate, i.e. the same bound could cost time here rather than save it.
- the +7.7GB was measured on a machine where nothing was under pressure. Whether reclaiming it changes anything depends on the tier, and streaming tiers (where RAM is actually scarce) were not measured at all.
Step 0
Paired A/B on the serialize path, same binary, one long prefill plus a decode run:
QWISP_PREFILL_MLX_CACHE_MB unset (today) vs 2048 vs 512.- Report
footprint peak, MLX.Memory active/cache, and tok/s for both prefill and decode — the speed side is the one that decides it, not the memory side.
- Repeat at a streaming tier (
QWISP_RAW_C low / qwisp simulate), where the memory would actually matter.
Kill criteria, fixed before looking: if bounding costs >2% decode tok/s at resident tier, close — the reclaimed memory is not needed there. If it is speed-neutral AND reclaims >4GB, ship it default-on for the streaming tiers first.
Instruments already exist: QWISP_PREFILL_MEMDBG=1 for the breakdown (mode 2 perturbs — see #150), footprint <pid> (never ps rss).
Refs
#150 (measurement that surfaced this) · #148 / #149 (the lane precedent and its numbers) · TellRuntime.swift Tell.prefill · LaneServe.swift (the lane-scoped MLX.Memory.cacheLimit site)
Observation
Memory.cacheLimitis bound on the lane path only. From #149's own comment: "Lane-scoped (LaneBackend is only built under QWISP_LANES); the serialize path is untouched." DefaultQWISP_LANE_MLX_CACHE_MB= 2048.The default serialize path therefore runs with MLX's free-buffer pool unbounded, and #150 measured what that costs: on a 35,178-token prefill the pool grew +7.7GB (
cache4,845 → 12,523MB), roughly half of the 16GB total footprint growth. Confirmed by the observer effect that hid it: withQWISP_PREFILL_MEMDBG=2(which callsclearCache()every report) the same prefill peaked ~6GB lower.Why this is not obviously a win
On lanes, bounding the pool measured −22.5GB peak AND ~10% faster in the crash configuration (#149) — the speed came from relieving memory pressure on a 64GB box. Neither number transfers:
Step 0
Paired A/B on the serialize path, same binary, one long prefill plus a decode run:
QWISP_PREFILL_MLX_CACHE_MBunset (today) vs 2048 vs 512.footprintpeak,MLX.Memoryactive/cache, and tok/s for both prefill and decode — the speed side is the one that decides it, not the memory side.QWISP_RAW_Clow /qwisp simulate), where the memory would actually matter.Kill criteria, fixed before looking: if bounding costs >2% decode tok/s at resident tier, close — the reclaimed memory is not needed there. If it is speed-neutral AND reclaims >4GB, ship it default-on for the streaming tiers first.
Instruments already exist:
QWISP_PREFILL_MEMDBG=1for the breakdown (mode 2 perturbs — see #150),footprint <pid>(neverps rss).Refs
#150 (measurement that surfaced this) · #148 / #149 (the lane precedent and its numbers) ·
TellRuntime.swiftTell.prefill·LaneServe.swift(the lane-scopedMLX.Memory.cacheLimitsite)