Every one of these was found by testing against reality rather than by reading the code, and every one is now pinned by a test. I'm listing them because a build with no recorded failures usually means nobody looked hard, and because how a bug was closed says more than whether it existed.
Ordered by how much damage each would have done unnoticed.
Severity: would have killed the live demo.
Found: running the financials task against a real page for the first time. It pulled Apple's numbers correctly, then froze on Microsoft's page. Not slow — frozen. Token count stopped moving, no screenshot, no error, no timeout. The spinner kept spinning.
Cause: stockanalysis.com renders continuously-animating charts.
Playwright's page.screenshot() waits for the page to reach visual
stability before capturing, and on that page stability never arrives. There
was no timeout on the call, so the run parked there forever.
Why it mattered more than a crash: this product's entire promise is "no silent failures." A crash is loud and recoverable. A hang looks like work in progress, so nobody intervenes — it is the single worst outcome the design can produce, and it was sitting in the happy path.
Fix — structural, not a patch. Patching just the screenshot call would
have left the same class of bug everywhere else. Instead every awaited
operation now carries a hard ceiling: 45s per browser action, 12s per
screenshot capture, 25s per page read. Captures freeze animations
(animations="disabled"). A timeout is not swallowed — it is handed to the
agent as a fact it must route around ("this page could not be read in time,
try another route"), and surfaced to the human in the feed.
Pinned by: tests/test_resilience.py — serves a page with an infinite
CSS animation and 600 elements, asserts capture completes. It used to hang;
it now returns in ~0.05s.
Severity: would have looked like a total product failure mid-demo.
Found: a production run on Railway. The result panel never appeared — just a black screen. The run itself had completed fine server-side.
Cause: the model returned notes as a plain string where the schema
said list of strings. The frontend called .map() on it, React threw, and
the whole tree unmounted. One malformed field took down the entire
interface.
Fix — three layers, because one is a single point of failure. Payloads
are normalized at the source in agent.py (a string becomes a one-item
list; an unusable table is dropped and the drop is disclosed in the notes
rather than silently swallowed). The render path in Result.tsx guards
independently. A React error boundary wraps the app so any future rendering
bug degrades to a readable message with a reload button instead of a dead
screen.
Worth noting: this only appeared with real model output. No scripted test would have produced it, because I wrote the scripts to match my own schema. Reality doesn't.
Pinned by: tests/test_resilience.py — feeds deliberately malformed
payloads through the finish path and asserts the UI contract survives.
Severity: run-ending, but loud.
Found: the very first end-to-end run with real Claude. Two steps in, the
API started returning 400: tool_use ids were found without tool_result blocks.
Cause: Claude occasionally emitted two tool_use blocks in one turn.
The loop executed the first and answered only that one, leaving the second
unanswered — which violates the Messages API contract, so every subsequent
request failed.
Why the fix isn't "handle both": one action per turn is a design invariant here, not an implementation detail. The human is watching each step and can veto it, so steps must be atomic and individually approvable. Executing two actions per turn would quietly break the approval model.
Fix: enforced at the API level with
tool_choice: {disable_parallel_tool_use: true} rather than merely
requested in the prompt, plus defensive handling that answers any extra
calls with an explicit "not executed, one action per turn" result so the
history stays valid even if the constraint is ever lifted.
Severity: would have taken down a live demo if two people opened the link at once.
Found: deliberately, during adversarial testing. Three simultaneous runs all completed correctly — so I measured what they cost. Resident memory: ~377 MB per concurrent Chromium. The deploy host has 1 GB.
Cause: nothing was limiting concurrency. Two runs fit. A third pushes the container past its memory ceiling, and the OOM killer takes down every session — including the one being demoed.
Fix: capacity is refused rather than discovered by crashing. Runs beyond
GLASSBOX_MAX_CONCURRENT (default 2) get an explicit message — "the demo
server is at capacity, each run drives a real browser" — instead of a dead
server. /api/health reports active_runs and max_concurrent so the
limit is observable rather than a mystery.
Verified: four simultaneous runs against a two-run limit — two succeeded, two were refused politely, nothing crashed.
Severity: cost me a showcase task.
Found: the SEC filings task failed in production while working from a residential IP.
Diagnosis: the agent's own report suggested the User-Agent. That was a
reasonable hypothesis — SEC's fair-access policy does require automated
clients to declare themselves — so I set a compliant declared UA
(Glassbox-Agent/1.0 (contact: ...)). It still failed.
Rather than guess again, I made the agent read back its own identity from
httpbin.org/user-agent. The declared UA was arriving perfectly. SEC was
still refusing. That isolates it: the block is IP-level, targeting
datacenter ranges, and no code change can fix it.
Resolution: rather than fake the demo, I sourced the same figures from a reachable provider and cross-checked all six against the actual 10-K filings — exact match. The declared User-Agent stayed, because it is correct behaviour regardless.
Why it's listed here: the useful part wasn't the fix, it was refusing to accept a plausible diagnosis without isolating the variable. The agent's guess was wrong, and it would have been easy to ship a "fix" that changed nothing.
Severity: cosmetic, but wasteful.
Found: watching a production run. The sandbox goal says "the portal at
/sandbox", and the agent's browser starts on a blank page — so a bare path
has no origin to resolve against. It tried localhost/sandbox, then
localhost:3000, got connection refused, and eventually found port 8080 by
itself.
Silver lining: this was unscripted recovery working exactly as designed, and it's genuinely good to have watched it happen.
Fix: the server now tells the agent its own address when a goal
references /sandbox. Recovery is a safety net, not a substitute for giving
the agent the information it needs.
Found: the operator-correction scenario failed once — but the product was fine (it still returned 14/14 rows). The test rejected "the first click after step 5", and on that run the model paginated by URL instead of clicking, so the rejection never fired.
Fix: the test now rejects the first post-login action whatever tool it uses. A test that depends on which valid path a model chooses is a flaky test, and a flaky test is worse than no test — it trains you to ignore red.
Listed because the same discipline applies to test code as product code, and because "the test was wrong" is a real finding worth recording rather than quietly editing away.
Severity: aborted a task after it had already gathered half its data.
Found: running the new multi-site research brief on production. The agent read the (very large) NVIDIA Wikipedia page and captured every fact correctly, then on navigating to the second source the Chromium renderer ran out of memory on the 1 GB host and the tab crashed with "Page crashed." The agent tried to recover, but every retry also failed and it aborted honestly after six attempts.
Cause, two parts. (1) A crashed renderer takes its whole browser context with it — the page object is dead, so no later navigation or read on it can ever succeed. The agent's normal "try another route" recovery can't help, because the thing it acts through is broken. (2) The crash itself was memory pressure: a very large, image-heavy page on a constrained host.
Fix — prevention plus self-healing.
- Prevention: the browser now launches with image decoding disabled
(
--blink-settings=imagesEnabled=false) plus--disable-gpuand a capped V8 heap. Disabling images is the single biggest memory saving and costs the agent nothing — it reads the page's text, not its pictures. The human still sees layout, text and charts in the screenshot; only photos are absent. - Self-healing: a crash is now detected in the action executor, the browser rebuilds its context (a clean renderer), and the agent is told the page was reset and to re-navigate. A fatal crash becomes a recoverable blip. If the browser process itself died, the whole thing relaunches.
Also fixed alongside: goto() was prepending https:// to anything not
starting with http — which mangled about:blank, data: and file: URLs.
Now those schemes pass through untouched.
Pinned by: tests/test_resilience.py — kills the page mid-run and asserts
the executor detects it, resets, and the run continues. Also verified that
disabling images leaves the page text fully intact (the memory fix must not
hide data) and that screenshots still capture.