Skip to content

Receipted scheduler-wake ledger (pending_schedule_wakes) grows unbounded — accepted rows retain full prompt payload forever #1035

Description

@olegbrok

Summary

pending_schedule_wakes retains every successfully-delivered ("receipted-ran-once") fire forever: confirm_pending_schedule_wake[_by_fire] (agent_registry.py:4767-4838) sets accepted_at>0 and keeps the row as the idempotency/dedup ledger. The only DELETE is the discard primitive (agent_registry.py:4852), guarded accepted_at=0 AND parked_at=0, so it never removes accepted rows. There is no retention/prune for accepted rows — contrast the heartbeat table, which is pruned to "keep last 100 per agent" (agent_registry.py:4885). Net: one permanent row per fire, and each row stores the full prompt text.

This is correct for the dedup/idempotency purpose (the retain-as-ledger design is intentional, and the health metric + replay/stale-sweep both filter accepted_at=0 so these rows are invisible to strand accounting) — but the ledger has no upper bound.

Impact (bytes, not rows)

Row count understates it — the prompt column dominates the bytes. Measured on the TOD box (26.08.015):

Projection for ONE agent: #78 ≈ 552 KB/day, #66 ≈ 57 KB/day, ~15 other daily fires ≈ 75 KB/day → ≈ 685 KB/day ≈ 250 MB/year/agent, multiplied by every agent on the box. In a SQLite/WAL DB this bloats the file, slows table scans, and balloons backups over time. Not urgent (slow, monotonic), but genuinely unbounded.

Fix (cheap, preferred)

Null the prompt on accept. A receipted row only needs its identity for dedup/idempotency (schedule_id, fired_at, accepted_at) — never its payload (accepted rows never replay; the replay/sweep paths filter accepted_at=0). Set prompt='' in the confirm_pending_schedule_wake[_by_fire] UPDATE (agent_registry.py:4786-4795 and 4821-4829). Immediate byte relief; keeps the dedup identity indefinitely.

Optional (secondary, row-count)

Add a retention prune of accepted rows older than N days, mirroring the heartbeat prune at agent_registry.py:4885. Safe: old fires are never re-attempted.

Tradeoff

Nulling the prompt loses the payload in list_schedule_wake_ledger forensic views for accepted rows (identity + schedule_name + fired_at + accepted_at remain; the prompt is reconstructable from the live schedule). Acceptable for a dedup ledger.

Provenance

Found + quantified by onesie (TOD fleet) during 26.08.015 post-deploy observation; root-caused to design (retain-as-ledger, no pruner) and verified against origin code by barsik. Related surface: the 26.08.015 wake-delivery incident (#1029/#1031) part-4 stranded-row health metric — confirmed working (filters accepted_at=0); this is a separate ledger-growth item, not a strand.

🤖 Opened by Barsik

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions