Fix Safari OOM on VFS image switching: reclaim Atomics.wait-blocked workers by brandonpayton · Pull Request #863 · Automattic/kandelo

brandonpayton · 2026-07-09T21:49:30Z

Problem

Repeatedly switching between Kandelo VFS images in the same Safari tab grows memory without bound and eventually throws Out of Memory. Chrome is fine.

Root cause

On JavaScriptCore (JSC), Worker.terminate() cannot free a worker parked in Atomics.wait — the state every blocked process/thread worker sits in on its syscall channel. So each teardown of a running machine leaks all of its workers' threads + committed working set. WordPress (~10 blocked daemons) leaked ~900–1100 MiB and ~10 threads per switch, monotonic → OOM. V8 interrupts Atomics.wait on terminate; JSC does not.

The boundary is the engine, not "browser vs Node". JSC backs both Safari and Bun; V8 backs Chrome and Node. So the leak hits the browser host and the Node host when the latter runs under Bun.

Two related WebKit bugs exist but neither resolves this on current builds: #278866 (JS Atomics.wait termination, fixed Aug 2024 — does not cover this) and #250569 (RESOLVED CONFIGURATION CHANGED, disputed). This PR is the runtime-side workaround.

Fix — cooperatively exit blocked workers before terminating

libc/glue/channel_syscall.c: on a queued SIGKILL (never delivered to the guest in normal operation → unambiguous "exit now"), call the kernel_exit import directly. Crucially not musl _exit(), which channels SYS_exit_group then spins for(;;) SYS_exit, re-parking the worker in Atomics.wait. The kernel_exit import is _Noreturn; the following unreachable trap is caught by worker-main as a clean exit → worker returns to its idle JS event loop → terminate() reclaims it.
host/src/kernel-worker.ts killAllBlockedForTeardown(): wakes every process channel parked at CH_STATUS==CH_PENDING (main threads + pthreads) — keyed on channel status, not per-resource wait maps, so it also catches accept()/epoll-blocked daemons — completing each syscall with -EINTR and queueing SIGKILL. Skips processes the kernel already reports Exited (kernel_get_process_exit_status(pid) != -1) so a sibling thread's real exit status isn't clobbered (a pthread exit(0) becoming 137). Returns the set of woken pids so the drain waits only for those.
Both host entries' handleDestroy (browser-kernel-worker-entry.ts and node-kernel-worker-entry.ts): wake, drain until the woken workers exit and are reclaimed, then terminate any stragglers. Applied unconditionally on both hosts — required on JSC (Safari, Bun), a bounded no-op on V8 (Chrome, Node) — rather than sniffing the engine.

Also migrates all browser demos to kernel-owned FS and removes the legacy main-thread VFS SharedArrayBuffer APIs (a secondary, smaller Safari leak where the persistent main thread co-owned each machine's growable VFS buffer). Adds BrowserKernel.initFromImage() / readFileFromVfs(); removes memfs/fsSab options, init(), the fs getter, registerLazyFiles/ensureMaterialized, and the lazy-registration module.

Documenting the workaround for removal

Every participating site is tagged [JSC-TERMINATE-ATOMICS-WAIT-LEAK] (git grep finds them all), and docs/jsc-terminate-atomics-wait-workaround.md gives the engine table, all sites, a step-by-step removal procedure, a known-limitation note, and the validation to re-run. When the JSC bug is fixed: grep the marker, follow the doc.

Testing on both engines

pkgs.bun added to the flake dev shell (JSC runtime, same engine as Safari).
host/test/teardown-reclaim.test.ts (+ fixture examples/block-forever.c): spawns a daemon parked in a blocking syscall, destroys the kernel, and asserts it was woken into a cooperative exit (status 137 = 128 + SIGKILL) and drained promptly — not force-terminated. Runs under both V8 and JSC (bun x vitest). pthread.test.ts (exit(0) preservation with a blocked main thread) also runs on both.
host npm scripts test:teardown / :bun / :engines; the CI vitest suite (scripts/ci-run-test-suite.sh) re-runs the teardown + pthread tests under Bun after the V8 run.

Validation

	threads/iter	RSS/iter
Before	+11 (monotonic)	+900–1100 MiB (monotonic → OOM)
After (Playwright WebKit, WordPress boot/destroy ×10)	flat (slope −0.1)	converges to a bounded ~3.4–4.1 GB plateau

Host vitest: 798 pass (3 pre-existing failures are missing wasm64/dlopen/fork-replay test binaries in the worktree, unrelated).
Cross-engine teardown test: green on Node (V8) and Bun (JSC).
The actual OS-level memory/thread reclamation is measured out-of-process via Playwright WebKit; the in-process tests assert the cooperative wake path ran (the engine-dependent part).

Committed repro harnesses: apps/browser-demos/public/terminate-atomics-test.html, apps/browser-demos/public/wasm-memory-reclaim-test.html.

Notes

No ABI_VERSION bump: no struct/syscall/channel-layout/export changes; uses the existing kernel_exit import and CH_SIG fields.
The fix takes effect once programs are relinked with the new glue (in libc.a); CI rebuilds all binaries/VFS images from source. Old-glue binaries degrade gracefully (still leak on JSC, no crash).
Known limitation: a process that exit_groups via one thread while another is still Atomics.wait-parked is skipped by the wake (to preserve its exit status) and force-terminated instead — so that one stuck worker still leaks on JSC. Narrow/unusual case, documented in the workaround doc.

Manual test

Recommend confirming in real Safari via ./run.sh browser: switch between demo images repeatedly and watch memory stay bounded.

🤖 Generated with Claude Code

…orkers Repeatedly switching Kandelo VFS images in one Safari tab grew memory without bound and eventually threw "Out of Memory". Root cause: on WebKit, `Worker.terminate()` cannot free a worker parked in `Atomics.wait` on its syscall channel — the state every blocked process/thread worker sits in — so each machine teardown leaked all of its workers' threads and committed working set (WordPress ≈ 10 blocked daemons ≈ +900-1100 MiB and +~10 threads per switch, monotonic). V8 interrupts the wait on terminate; JSC does not. Fix — cooperatively drive blocked workers to a clean exit before terminating: - libc/glue/channel_syscall.c: on a queued SIGKILL (never delivered to the guest in normal operation, so unambiguous) call the `kernel_exit` import directly. This must NOT use musl `_exit()`, which issues SYS_exit_group over the channel and then spins `for(;;) SYS_exit`, re-parking the worker in Atomics.wait. The `kernel_exit` import is _Noreturn; the following `unreachable` trap is caught by worker-main as a clean exit, unwinding the wasm so the worker returns to its idle JS event loop and terminate() reclaims its thread + memory. - host/src/kernel-worker.ts: killAllBlockedForTeardown() wakes every process channel parked at CH_STATUS==CH_PENDING (main threads + pthreads) — keyed on channel status, not the per-resource wait maps, so it also catches accept()/epoll-blocked daemons — completing each syscall with -EINTR and queueing SIGKILL. - host/src/browser-kernel-worker-entry.ts (handleDestroy): wake blocked workers, drain until they exit and are reclaimed, then terminate stragglers. Node host is unchanged: V8's terminate() already frees Atomics.wait-blocked workers, and the glue change is dormant unless the kernel queues SIGKILL, which only the browser teardown does. Justified by the WebKit platform boundary. Also migrate all browser demos to kernel-owned FS and remove the legacy main-thread VFS SharedArrayBuffer APIs (a secondary, smaller Safari leak: the persistent main thread co-owned each machine's growable VFS buffer, which WebKit reclaims only under GC pressure that WASM/SAB memory rarely triggers). Demos now assemble a transient build FS, saveImage() it, and initFromImage() so the kernel worker solely owns the live VFS; destroy() drops the main thread's framebuffer/PTY buffers. Adds BrowserKernel.initFromImage() and readFileFromVfs(); removes memfs/fsSab options, init(), the fs getter, registerLazyFiles/ensureMaterialized, and the lazy-registration module. Validated (Playwright WebKit, kernel-owned WordPress boot/destroy x10, freshly relinked nginx+php-fpm+dinit): before +11 threads & +900-1100 MiB per switch (monotonic); after threads flat (slope -0.1) and RSS converges to a bounded ~3.4-4.1 GB plateau. Host vitest: 797 pass (3 pre-existing failures are missing wasm64/dlopen/fork-replay test binaries in the worktree). Committed repro harnesses: public/terminate-atomics-test.html, public/wasm-memory-reclaim-test.html. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

github-actions · 2026-07-10T00:06:21Z

Phase B-1 matrix build status — `pr-863-staging`

ABI v16. 68 built, 0 failed, 68 total.

Package	Arch	Status	Sha
libcurl	wasm32	built	`0352cfac`
libcxx	wasm32	built	`51570a32`
libcxx	wasm64	built	`ff1cf2bc`
libpng	wasm32	built	`612b4024`
libxml2	wasm32	built	`adb341c8`
libxml2	wasm64	built	`0f24dd35`
openssl	wasm32	built	`af4147e0`
openssl	wasm64	built	`5d0705a0`
sqlite	wasm32	built	`bcc5822f`
sqlite	wasm64	built	`1c43a0c4`
zlib	wasm32	built	`94efb1ca`
zlib	wasm64	built	`9e57ce8b`
bc	wasm32	built	`70239e6c`
bzip2	wasm32	built	`335c1ef6`
coreutils	wasm32	built	`6be6bfcf`
curl	wasm32	built	`8a13c8fa`
dash	wasm32	built	`dfe7bbdf`
diffutils	wasm32	built	`d7506509`
dinit	wasm32	built	`ac72da07`
fbdoom	wasm32	built	`a891f021`
file	wasm32	built	`9d7484eb`
findutils	wasm32	built	`4a31d1a0`
gawk	wasm32	built	`9218c3a0`
git	wasm32	built	`38ee841f`
grep	wasm32	built	`af4adf56`
gzip	wasm32	built	`aba1a5ab`
hello	wasm32	built	`79c2823c`
kandelo-sdk	wasm32	built	`a3c912f4`
kernel	wasm32	built	`1fd2a7a9`
less	wasm32	built	`2df7e50b`
lsof	wasm32	built	`16155a21`
m4	wasm32	built	`48f6cfea`
make	wasm32	built	`2e3173bf`
mariadb	wasm32	built	`4287f28d`
mariadb	wasm64	built	`97356c69`
modeset	wasm32	built	`4cadc656`
msmtpd	wasm32	built	`96063975`
nano	wasm32	built	`dda16c4a`
ncurses	wasm32	built	`ff22bffc`
netcat	wasm32	built	`44b69b16`
nginx	wasm32	built	`5dd2c3c0`
php	wasm32	built	`2f12ee7f`
posix-utils-lite	wasm32	built	`b818663f`
sed	wasm32	built	`6f17774d`
spidermonkey	wasm32	built	`6988045e`
tar	wasm32	built	`4f4942b8`
tcl	wasm32	built	`5887c381`
unzip	wasm32	built	`8668e002`
userspace	wasm32	built	`a49149da`
vim	wasm32	built	`9d000b71`
wget	wasm32	built	`6dd06a78`
xz	wasm32	built	`b41441ea`
zip	wasm32	built	`fa768da6`
zstd	wasm32	built	`383fea58`
bash	wasm32	built	`aaa4d9f1`
mariadb-test	wasm32	built	`f87473d0`
mariadb-vfs	wasm32	built	`b63206cd`
mariadb-vfs	wasm64	built	`af14c8cb`
nethack	wasm32	built	`0473fe97`
node	wasm32	built	`6767bf7c`
spidermonkey-node	wasm32	built	`81aca532`
vim-browser-bundle	wasm32	built	`0f294516`
nethack-browser-bundle	wasm32	built	`4a74cf2a`
rootfs	wasm32	built	`235c7085`
shell	wasm32	built	`f7dccdb9`
lamp	wasm32	built	`64104909`
node-vfs	wasm32	built	`7618c9c6`
wordpress	wasm32	built	`76133d30`

_{Auto-generated; replaced on each push. Raw data in the publish-status workflow artifact.}

…+ document removal The engine boundary is JSC vs V8, not browser vs Node: Bun embeds JavaScriptCore, so the Node host entry hits the same `Worker.terminate()`-can't-free-an- Atomics.wait-blocked-worker leak under Bun. The prior "browser-only, Node unchanged" split was wrong. Apply the cooperative wake-drain-terminate teardown in `node-kernel-worker-entry.ts` `handleDestroy` too, unconditionally — required on JSC (Safari, Bun), a bounded no-op on V8 (Chrome, Node) — matching the browser host and the Host Runtime Contract's peer-hosts rule. - kernel-worker.ts `killAllBlockedForTeardown`: skip processes the kernel already reports Exited (`kernel_get_process_exit_status(pid) != -1`) — waking a still- parked sibling of a thread that already `exit_group`'d would clobber the real exit status (a `pthread` exit(0) becoming 137; fixes a pthread.test.ts regression this change introduced). Return the set of woken pids so both handleDestroy drains wait only for those; a not-woken straggler never posts {exit} and is force-terminated instead of waited on. - Reframe all comments from "WebKit"/browser to the JSC engine boundary, correct a stale note (the glue calls the `kernel_exit` import, not `_exit`), and tag every participating site `[JSC-TERMINATE-ATOMICS-WAIT-LEAK]`. - Add docs/jsc-terminate-atomics-wait-workaround.md: the engine table, every tagged site, the removal procedure, a known-limitation note, and validation (incl. Bun). Enables removing the whole workaround when the JSC bug is fixed. Validated: host vitest 797 pass (3 pre-existing binary-not-found), pthread.test.ts green; browser WordPress boot/destroy ×6 still flat threads + bounded RSS plateau. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…un to the flake The workaround exists for JavaScriptCore, which Safari AND Bun share, but nothing ran the teardown path on JSC. Add `pkgs.bun` to the flake dev shell and a cross-engine test so the JSC behaviour is actually exercised, not just reasoned about. - examples/block-forever.c: a fixture that parks in a blocking nanosleep (Atomics.wait on its channel), modeling an idle daemon. It never exits on its own, so a cooperative teardown exit is observable as status 137 (128 + SIGKILL, from the glue's kernel_exit). - host/test/teardown-reclaim.test.ts: spawns that daemon, destroys the kernel, and asserts it was WOKEN into the cooperative exit (137) and the drain resolved promptly — not force-terminated while parked. Runs under both engines via `bun x vitest`. Verified locally: passes on Node (V8) and Bun (JSC); the existing pthread exit(0)-preservation test also passes on both. - host/package.json: `test:teardown` / `test:teardown:bun` / `:engines` scripts. - scripts/ci-run-test-suite.sh: the `vitest` suite now re-runs the teardown + pthread tests under Bun after the V8 run. - flake.nix: add pkgs.bun (JSC runtime) to the dev shell. The actual memory/thread reclamation is still only measured out-of-process via Playwright WebKit (the in-process tests assert the cooperative path ran, which is the engine-dependent part). Full host vitest: 798 pass (3 pre-existing binary-not-found). See docs/jsc-terminate-atomics-wait-workaround.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

github-actions · 2026-07-10T21:45:41Z

prepare-merge: test-gate passed against the synthetic PR merge and binaries-abi-v16. Missing entries were built or promoted from PR staging only when their merged-tree cache key matched; merge-gate=success posted on PR HEAD and squash auto-merge enabled.

brandonpayton and others added 2 commits July 10, 2026 12:40

brandonpayton added the ready-to-ship label Jul 10, 2026

github-actions Bot enabled auto-merge (squash) July 10, 2026 21:45

brandonpayton disabled auto-merge July 10, 2026 23:21

brandonpayton merged commit 31e77f5 into main Jul 10, 2026
183 checks passed

brandonpayton deleted the emdash/safari-out-of-memory-errors-w5ots branch July 10, 2026 23:22

This was referenced Jul 11, 2026

Fix pthread fork continuation buffer corruption #867

Open

Preserve guest stderr on normal browser exits #875

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix Safari OOM on VFS image switching: reclaim Atomics.wait-blocked workers#863

Fix Safari OOM on VFS image switching: reclaim Atomics.wait-blocked workers#863
brandonpayton merged 3 commits into
mainfrom
emdash/safari-out-of-memory-errors-w5ots

brandonpayton commented Jul 9, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 10, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 10, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

brandonpayton commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Root cause

Fix — cooperatively exit blocked workers before terminating

Documenting the workaround for removal

Testing on both engines

Validation

Notes

Manual test

Uh oh!

github-actions Bot commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Phase B-1 matrix build status — pr-863-staging

Uh oh!

github-actions Bot commented Jul 10, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

brandonpayton commented Jul 9, 2026 •

edited

Loading

github-actions Bot commented Jul 10, 2026 •

edited

Loading

Phase B-1 matrix build status — `pr-863-staging`