Skip to content

fix(seedless): check command buffer status, poison the backend on GPU failure - #171

Merged
penta2himajin merged 1 commit into
mainfrom
claude/issue169-gpu-failure-firewall
Aug 2, 2026
Merged

fix(seedless): check command buffer status, poison the backend on GPU failure#171
penta2himajin merged 1 commit into
mainfrom
claude/issue169-gpu-failure-firewall

Conversation

@penta2himajin

Copy link
Copy Markdown
Owner

Refs #169. Stops the silent failure; does not fix the OOM itself.

What this changes

A failed Metal command buffer leaves its output buffers untouched. Shared buffers are zero-initialised, so reading them anyway yields an all-zero logit row, which argmax_rows tie-breaks to index 0 — token id 0 (!), forever. Nothing inspected cb.status before this: 94 waitUntilCompleted call sites, zero error checks.

  • commitAndWaitChecked(cb, label) -> String? — nil on success, a fault description otherwise. Non-nil means the caller must not read that command buffer's outputs.
  • GPUFaultState — per-instance, sticky, first-fault-wins. Deliberately not process-global: one failed request must not kill a long-running server. SeedlessFusedForward owns one.
  • Applied to the decode readback funnels — stepArgmax (.resident/.bolt) and chainedStepArgmax — plus an entry guard so a poisoned backend refuses rather than reading stale buffers. The chain advances CPU-side KV length at encode time, so continuing after a failure is unsound, not merely wasteful.
  • Decode path selection is now logged unconditionally. The same logical op has three implementations (raw fused step, GPU chain, MLX-composed fallback) and which one runs is a runtime decision. Without this line, instrumentation lands on a path that never executes — that cost most of the debugging time on bolt: decode command buffer fails with GPU OOM and cb.error is never checked → unbounded token-id-0 (!) output above ~2–6K prompt tokens #169.
  • QWISP_LOGIT_DBG (default off): per-row logit/hidden stats and cb.status.

Evidence

16GB-class config, one prose prompt truncated to a ladder, cb.error read at the failing commit:

config fails at
bolt mixed C=128 ~4.0–4.3K prompt tok
bolt generic C=128 between 6.6K and 14.7K
strict (8GB budget, C=64) passes 41,489 tok

At every failure: MTLCommandBufferErrorDomain Code=8 kIOGPUCommandBufferCallbackErrorOutOfMemory, logits all zero (nan=0), while hBuf and normed held healthy leftover values from prefill — the CB simply never ran.

Not in this PR, deliberately

Gates

RAWTESTS 99/99. The new locked test asserts the checked-commit contract and that the poison is per-instance, sticky, and does not leak between backends. Known limitation: failure is injected via a test flag, so removing only the cb.status comparison while keeping the injection check would not be caught — the static gate in the follow-up should close that.

🤖 Generated with Claude Code

… failure (#169)

A failed Metal command buffer leaves its output buffers untouched. Shared
buffers are zero-initialised, so reading them anyway yields an all-zero logit
row, which argmax_rows tie-breaks to index 0 — token id 0, forever. Nothing in
this codebase inspected cb.status before this commit (94 waitUntilCompleted
call sites, zero error checks), so a GPU OOM surfaced as a plausible-looking
`<word>` followed by an unbounded run of `!`.

Measured on a 16GB-class config: bolt mixed C=128 fails at ~4.0-4.3K prompt
tokens, bolt generic C=128 between 6.6K and 14.7K, strict passes 41K even at an
8GB budget. cb.error read
`kIOGPUCommandBufferCallbackErrorOutOfMemory` at every failing point, while the
hidden state and lm_head input were healthy — the CB simply never ran.

- commitAndWaitChecked(cb, label) -> String?: nil on success, a fault
  description otherwise. A non-nil result means the caller must not read any
  output buffer of that command buffer.
- GPUFaultState: per-instance, sticky, first-fault-wins. Deliberately NOT
  process-global — one failed request must not kill a long-running server.
  SeedlessFusedForward owns one.
- Applied to the decode readback funnels: stepArgmax (.resident/.bolt) and
  chainedStepArgmax, plus an entry guard so a poisoned backend refuses rather
  than reading stale buffers. chain advances CPU-side KV length at encode time,
  so continuing after a failure is unsound, not merely wasteful.
- Decode path selection is now logged unconditionally. The same logical op has
  three implementations (raw fused step, GPU chain, MLX-composed fallback) and
  which one runs is a runtime decision; without this line an instrumentation
  attempt lands on a path that never executes.
- QWISP_LOGIT_DBG (default off): per-row logit/hidden stats and cb.status, the
  instrumentation that found this.

Not covered here, deliberately: the strict path's split command buffers,
stepArgmaxBatch, prefill CBs, a static gate against bare waitUntilCompleted,
and an error channel on LLMBackend.generate's AsyncStream. Today the failure
stops generation and reports on stderr; it does not yet reach the caller as a
typed error. The OOM itself is a separate capacity bug and stays open in #169.

RAWTESTS 99/99 (new locked test asserts the checked-commit contract and that
the poison is per-instance, sticky, and does not leak between backends).

Refs #169

Co-Authored-By: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant