Skip to content

fix: drain builder before mutating worker runtime - #195

Merged
sunwoo162 merged 1 commit into
mainfrom
fix/bloom-worker-safe-deploy
Sep 3, 2026
Merged

sunwoo162 merged 1 commit into
mainfrom
fix/bloom-worker-safe-deploy

Conversation

@sunwoo162

Copy link
Copy Markdown
Owner

Summary

  • transfer the runtime bridge into a per-run staging directory instead of the live path
  • request Builder drain and wait for the active busy cycle to finish before touching the shared checkout
  • remove stale busy markers only when their recorded PID is no longer alive
  • smoke-test a next bridge candidate and atomically promote it to the live path
  • always remove the drain marker and staging artifacts on exit/signals

Root cause

Production Live E2E run #45 was interrupted by worker deployment: pm2 began deleting bloom-builder-worker roughly 100ms before the active runtime bridge exited with code null. Stage 1 (#193) added worker-side drain/busy markers and is already deployed.

Verification

  • pnpm test:production-runtime (57/57)
  • pnpm run build:bloom-worker
  • git diff --check

Rollout

After merge, verify Deploy to Server and Deploy Bloom Worker both succeed and confirm the worker deploy logs show Builder drain request/completion, runtime bridge smoke, and both worker processes online.

@sunwoo162
sunwoo162 merged commit 6a06e56 into main Sep 3, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant