doc: design durable subscribe

[antiguru]

Design document for a resumable SUBSCRIBE, plus the user documentation written alongside it.

A SUBSCRIBE cannot be resumed today. AS OF already expresses the resume, but nothing holds the timestamp readable, and the default compaction window is one second, so a client that remembers a position finds it compacted away. Our own console shows the cost: it opens one WebSocket per subscribe and re-runs from scratch on reconnect, holding a stale snapshot behind a resubscribing flag until a new snapshot completes.

The proposal is a durable subscription: a named catalog object holding a read hold on a storage collection, advanced by an ACKNOWLEDGE statement. The hold defines a window of readable time, the acknowledgement moves its lower edge, and a wall-clock deadline bounds how long the window stays open without progress.

Core decisions

• A read hold, not a read policy. ReadPolicy::Multiple has zero construction sites, policy installation only ratchets capabilities upward so a lower frontier installed behind an advanced one is a silent no-op, the installation API is keyed by CompactionWindow and cannot express a per-collection constant, and an ordinary ALTER ... RETAIN HISTORY would discard the contribution irreversibly. Holds report the frontier actually acquired; policies report nothing.
• One hold, not two. The compute controller never relaxes a sink's input hold, so a continuously connected reader would pin history from its attach point forever and the deadline would never notice. The subscription's hold is therefore the attached dataflow's input floor.
• A wall-clock deadline, not a frontier distance. A REFRESH materialized view would otherwise need a value aware of every refresh policy in its ancestry, and a stalled source, paused source, or zero-replica cluster freezes the frontier precisely when releasing the hold matters most.
• A timestamp is a better resume token than a log position. One timestamp names a consistent cut across every collection in the timeline. Log positions of concurrent non-conflicting transactions interleave, so systems built on them need transaction boundaries and synthetic pre-commit watermarks to recover the order a timestamp gives for free.

Surface

CREATE DURABLE SUBSCRIPTION s ON t WITH (ACKNOWLEDGE WITHIN '1m');
SUBSCRIBE USING DURABLE SUBSCRIPTION s WITH (PROGRESS);
ACKNOWLEDGE DURABLE SUBSCRIPTION s UP TO 1723459200000;

ACKNOWLEDGE ... UP TO reuses the one exclusive-upper convention SUBSCRIBE already has, rather than adding a third; AT read as inclusive while meaning exclusive. ACKNOWLEDGE WITHIN mirrors the adjacent RETAIN HISTORY FOR '1h' and names the obligation accurately now that the bound is wall-clock. SNAPSHOT defaults to false here, inverting plain SUBSCRIBE, and a requested snapshot is taken at the acknowledged position so the state received corresponds to a position the client can name. ENVELOPE UPSERT, ENVELOPE DEBEZIUM, and AS OF AT LEAST are rejected.

Recovery from expiry is two attaches, not one option. A plain attach on an expired subscription fails with a distinguishable error, and SUBSCRIBE ... WITH (RESET) resets to the current time and delivers a snapshot there. The reason is that a consumer otherwise cannot tell a fresh snapshot from a continuation: a snapshot arrives as insertions at the resume timestamp, indistinguishable from genuine insertions at that time, and the alternatives are a new column on every SUBSCRIBE or an advisory session notice. With two attaches the client knows because it asked. A live subscription deliberately cannot be fast-forwarded by either spelling of RESET, since that would discard data it still holds history for; a client that lost local state while live reconciles instead, paying a replay bounded by ACKNOWLEDGE WITHIN.

Reviews folded in

The document has been through an adversarial design review and the repository's documentation review, and was then evaluated against what a local-first sync engine would require of it, reading Zero, ElectricSQL, PowerSync, and Replicache documentation.

That last review inverted one conclusion worth calling out. Rejecting the envelopes looked like it excluded the natural keyed consumer, but every engine surveyed keeps its own previous state rather than asking the upstream for old values, and the raw diff stream gives them more than a Postgres feed does: every retraction carries the full old row, which is REPLICA IDENTITY FULL semantics by construction, and WITHIN TIMESTAMP ORDER BY mz_diff orders retractions before additions within a timestamp. It also corrected the idempotence rule, which had said to deduplicate on the timestamp and row. The unit is the timestamp: apply each atomically, record it, skip ones already applied, which works because resume always lands on a timestamp boundary.

Verdict on consumability: a PowerSync-shaped engine could consume this with changes. Zero could not without work outside this design, because its upstream must also host its own mutation bookkeeping tables, with lastMutationID incremented in the same transaction as the user's write and replicated back through the feed, and Materialize supports neither primary keys nor UPDATE inside transactions.

A "Known quirks and interactions" section documents rather than fixes several existing behaviors the feature makes easier to trigger, including that a subscription's hold lowers the as-of chosen for objects created later over the same inputs, which for materialized views is written durably into create_sql; that combining several subscriptions into a consistent cut is safe but unassisted, and unsound across timelines; and that no mapping exists from an upstream source position to the Materialize timestamp at which it became visible, which is a hard dependency for any consumer that needs to correlate its own writes with the feed.

On the user documentation

The pages under doc/user/ document a feature that does not exist. They were written as part of designing it, to expose awkwardness in the SQL surface, and they earned the place. Writing them is what surfaced that progress messages are mandatory rather than optional, that the deadline must be required rather than defaulted, that retention is the minimum over readers so one stuck reader holds history for all of them, that at-least-once delivery is a genuine step back from what the existing manual pattern achieves for consumers with a transactional sink, and that the reset path had no way to tell a consumer what it had received.

They must not merge before the feature exists. Hugo would publish them. They are here for review of the contract, and the intent is to land them with the implementation. Happy to move them under doc/developer/design/ as an appendix if reviewers prefer.

doc/user/content/transform-data/patterns/durable-subscriptions.md is a shipped private-preview page that was titled "Durable subscriptions", documenting the manual RETAIN HISTORY plus client-recorded-timestamp pattern. This renames it to "Resuming subscriptions" to free the name. The manual pattern stays documented, because recording the position yourself is what makes exactly-once possible.

Open questions

Seven are listed in the document. The two most likely to change the shape of the implementation are whether the durable attach can bypass index import cleanly, and the value of the deadline ceiling, which should be in hours rather than the minutes an interactive client needs.

No tracking issue is filed yet; one is needed before this merges.

🤖 Generated with Claude Code