Repository navigation
Journal: event streams share one watcher per process instead of each polling - #105
Merged
Merged
Conversation
PALBIBEK
force-pushed
the
journal-indexes
branch
from
November 3, 2026 21:41
a6acc19 to
9068cb0
Compare
…polling Every open stream read the journal and the partial reply four times a second and its session's status every two: 500 streams took the API's p95 to 2.5 s. supervisor.journal.watch asks once a round for every session the process's streams follow (last seq, status, partial reply time; one query per 500 sessions) and wakes the streams whose session changed. On Postgres each journal write and partial reply notifies on commit (pg_notify) and the watch listens, so a stream wakes as soon as the write lands; the shared poll then runs every 2 s as a backstop, and a dropped listening connection is noticed there and reopened, polling fast meanwhile. A watched stream reads only when its session changed and ends when its status closes (or the session is gone). Without a watch, iter_journal_sse polls as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ge2HHQ32aPUxZf3EnfAfpt
PALBIBEK
force-pushed
the
journal-watch
branch
from
November 3, 2026 21:46
46975d4 to
04fbe77
Compare
PALBIBEK
added a commit
that referenced
this pull request
Nov 4, 2026
…abase notifications (#114) #105 sent pg_notify with every journal write and partial reply. A transaction that sends one takes a lock on the whole Postgres server as it commits, so every commit on the server waited its turn (16 writers: 348 appends/s with notifications, 394 without; worse as writers grow), and behind a pooler in transaction mode the listening connection would silently hear nothing. Writes no longer notify, and the watch has no listening connection: its shared poll runs every 150 ms (one small query per process, whatever the number of streams), a starting stream is told at once, and a session's end is seen within a round instead of within 2 s. The journal package's docstring now says what each module is. Claude-Session: https://claude.ai/code/session_01Ge2HHQ32aPUxZf3EnfAfpt Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every open stream read the journal and the partial reply four times a second and its
session's status every two: 500 streams took the API's p95 to 2.5 s.
supervisor.journal.watch asks once a round for every session the process's streams
follow (last seq, status, partial reply time; one query per 500 sessions) and wakes
the streams whose session changed. On Postgres each journal write and partial reply
notifies on commit (pg_notify) and the watch listens, so a stream wakes as soon as the
write lands; the shared poll then runs every 2 s as a backstop, and a dropped listening
connection is noticed there and reopened, polling fast meanwhile.
A watched stream reads only when its session changed and ends when its status closes
(or the session is gone). Without a watch, iter_journal_sse polls as before.
Journal plan, PR 7 of 12. Stacked on
journal-indexes; this PR's own change is its last commit.Tests
SUPERVISOR_TEST_PG_URL); ruff and mypy are clean.journal-cleanup): lint and types, tests, backend, frontend, e2e, compose smoke, journeys.Generated by Claude Code