Skip to content

ingest: read a transcript again when it has grown since it was stored - #30

Open
kaihirota wants to merge 2 commits into
prime-radiant-inc:mainfrom
kaihirota:ingest-regrown-transcripts
Open

kaihirota wants to merge 2 commits into
prime-radiant-inc:mainfrom
kaihirota:ingest-regrown-transcripts

Conversation

@kaihirota

Copy link
Copy Markdown

What happens today

ingestSessions skips a file when its path is already in sessions:

const existing = checkStmt.get(file);
if (existing) { skipped++; continue; }

A transcript is appended to for as long as its session stays open, so the same path holds
more every day that session runs. Once a file has been seen it is never read again, and the
session stays at whatever length it happened to be the first time it was scanned.

Nothing downstream can tell. The run reports no errors, and the sessions that keep arriving
afterwards are mostly subagents, which groupSessionsByDateAndProject excludes — so the
journal simply stops gaining entries and every daily run still exits 0.

Measured on one machine: a session was stored at 195 messages from a file that had
reached 16,379 lines five days later. Journal entries stopped four days before anybody
noticed. After the change that session re-read as 1,082 messages, and main sessions
appeared for the two days the table had been missing.

What this changes

A file already seen is read again when its modification time is later than the ingest that
stored it, and the session is replaced.

The replacing path is the one --force already used: the duplicate-id guard and the
delete-then-insert inside the transaction both now key off replacing rather than force.
No new machinery.

Two details worth naming:

  • datetime('now') writes UTC without a zone, so ingested_at is parsed as UTC explicitly.
  • It also writes whole seconds, so the file's mtime is floored to seconds before comparing.
    Comparing milliseconds against seconds reads any file written in the same second as its own
    ingest as newer than it, and re-reads it forever. The existing
    skips already-ingested sessions test catches that.

Also

README.md said auto_sync_interval is seconds. startTimer multiplies it by 60 * 1000,
and the settings screen already says minutes and labels the field with it. Only the README
disagreed, and it made a reader expect a sync a minute rather than an hour.

Tests

One added: a fixture is ingested, a real message appended, its mtime pushed past the ingest,
and the session asserted to be re-read, to have grown, and to have been replaced rather than
duplicated.

bun run check passes: 95 tests, typecheck clean.

kaihirota and others added 2 commits August 24, 2026 12:55
A session's transcript is appended to for as long as that session stays open, so the same
path holds more every day the session runs. The skip test was the path alone, which froze
each session at whatever length it happened to be the first night it was scanned.

On this machine one session was stored at 195 messages from a file that had reached 16,379
lines five days later. Nothing downstream could tell: the daily run reported no errors, the
journal simply stopped gaining entries, because the only sessions arriving after that were
subagents and those are not summarized.

A file is now re-read when its modification time is later than the ingest that stored it,
and its session replaced. The comparison is at whole seconds, because the ingest time is
written by datetime('now') and a file written inside the same second as its own ingest is
not a file that grew afterwards.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XewuV2izBAhDPD2gEsGbKf
The timer multiplies the setting by sixty thousand, so the default of 60 is hourly. The
settings screen already says minutes and labels the field with it; only this file disagreed,
and it made a reader expect a sync a minute.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XewuV2izBAhDPD2gEsGbKf
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant