doc: design durable subscribe

doc/user/content/headless/sql-command-privileges/acknowledge.md

@@ -0,0 +1,5 @@
+---
+headless: true
+---
+- Ownership of the durable subscription.
+- `SELECT` privileges on the object it subscribes to.

doc/user/content/headless/sql-command-privileges/alter-durable-subscription.md

@@ -0,0 +1,4 @@
+---
+headless: true
+---
+- Ownership of the durable subscription.

doc/user/content/headless/sql-command-privileges/create-durable-subscription.md

@@ -0,0 +1,6 @@
+---
+headless: true
+---
+- `CREATE` privileges on the containing schema.
+- `SELECT` privileges on the object being subscribed to.
+- `USAGE` privileges on the schema containing that object.

doc/user/content/headless/sql-command-privileges/drop-durable-subscription.md

@@ -0,0 +1,4 @@
+---
+headless: true
+---
+- Ownership of the durable subscription.

doc/user/content/sql/acknowledge.md

@@ -0,0 +1,131 @@
+---
+title: "ACKNOWLEDGE"
+description: "`ACKNOWLEDGE` advances the position of a durable subscription."
+menu:
+  main:
+    parent: 'commands'
+---
+
+{{< private-preview />}}
+
+`ACKNOWLEDGE` tells Materialize how far you have processed a [durable
+subscription](/sql/create-durable-subscription/), which advances the position it
+resumes from and releases the history before it.
+
+## Syntax
+
+```mzsql
+ACKNOWLEDGE DURABLE SUBSCRIPTION <name> UP TO <timestamp>
+;
+```
+
+| Field | Use |
+| --- | --- |
+| `<name>` | The durable subscription to advance. |
+| `<timestamp>` | An [`mz_timestamp`](/sql/types/mz_timestamp/). Asserts that you have durably processed every update at times **strictly before** this value. |
+
+## Details
+
+### What to acknowledge
+
+Acknowledge the `mz_timestamp` of a progress message, which is why
+[`SUBSCRIBE`](/sql/subscribe/#progress) must be run `WITH (PROGRESS)` when you
+intend to acknowledge. A progress message with timestamp `t` means no further
+updates will arrive at times strictly before `t`, which is exactly the claim
+`ACKNOWLEDGE ... UP TO t` makes back to Materialize. The bound is exclusive in
+both directions, so the number you read from the progress message is the number
+you send back unchanged.
+
+Do not acknowledge the timestamp of an ordinary row. Not every timestamp
+produces a progress message, and a row at time `t` does not mean that time `t`
+is complete, so acknowledging it can skip updates you have not seen.
+
+`UP TO` is exclusive here for the same reason it is exclusive on
+[`SUBSCRIBE`](/sql/subscribe/#up-to), which makes the batch pattern symmetric:
+read `UP TO` a timestamp, then acknowledge `UP TO` that same timestamp.
+
+### Order of operations
+
+Commit your data first, then acknowledge. If you acknowledge before your own
+processing is durable, and your application then fails, the acknowledged updates
+are gone and cannot be re-delivered.
+
+### Semantics
+
+`ACKNOWLEDGE` is:
+
+*   **Monotone.** Acknowledging a timestamp at or below the current position is
+    accepted and has no effect, so retrying is safe.
+
+*   **Idempotent.** Sending the same acknowledgement twice is indistinguishable
+    from sending it once.
+
+*   **Not transactional.** The acknowledgement takes effect immediately and is
+    not undone by `ROLLBACK`. This is deliberate: your data really was
+    committed, so rolling back must not un-acknowledge it.
+
+Acknowledging a timestamp beyond the object's write frontier is an error. The
+frontier is the largest timestamp for which the subscription could have sent you
+a progress message, and it is reported for every object in
+[`mz_internal.mz_frontiers`](/reference/system-catalog/mz_internal/#mz_frontiers).
+
+### Where you can run it
+
+`ACKNOWLEDGE` may be run on the same connection as the subscription, interleaved
+between `FETCH` statements, or on a separate connection. Because a subscription
+is named, no coordination between connections is needed, and because
+acknowledgements are monotone, they cannot arrive out of order in any way that
+matters.
+
+Running `ACKNOWLEDGE` while no one is reading the subscription is allowed. This
+matters for the separate-connection case, where the reading connection may drop
+while an acknowledgement is in flight.
+
+### Effect on resuming and on storage
+
+The acknowledged position is where the subscription resumes, and it determines
+how much history Materialize retains. Acknowledging more often releases storage
+sooner and shortens the replay after a failure; acknowledging less often reduces
+round trips. Materialize records the position durably on a short interval rather
+than on every statement, so history is released slightly after you acknowledge.
+
+When you resume without an explicit `AS OF`, Materialize positions the
+subscription so that you receive updates at and after the acknowledged time. With
+`SNAPSHOT true` you receive the state *as of* that time, with updates at that
+time already folded in, followed by later updates. Either way you do not subtract
+anything. The subtraction is only needed if you choose to pass [`AS
+OF`](/sql/create-durable-subscription/#where-reading-starts-and-stops) yourself,
+which is an exclusive bound.
+
+## Examples
+
+Acknowledging from a progress message received on the same connection:
+
+```mzsql
+FETCH ALL c WITH (timeout = '1s');
+```
+
+```nofmt
+ mz_timestamp  | mz_progressed | mz_diff | auction_id | amount
+---------------+---------------+---------+------------+--------
+ 1723459199000 | f             |       1 |          1 |     42
+ 1723459200000 | t             |         |            |
+```
+
+```mzsql
+ACKNOWLEDGE DURABLE SUBSCRIPTION winning_bids_feed UP TO 1723459200000;
+```
+
+## Privileges
+
+The privileges required to execute this statement are:
+
+{{% include-headless "/headless/sql-command-privileges/acknowledge" %}}
+
+## Related pages
+
+*   [`CREATE DURABLE SUBSCRIPTION`](/sql/create-durable-subscription/)
+*   [`ALTER DURABLE SUBSCRIPTION`](/sql/alter-durable-subscription/)
+*   [`DROP DURABLE SUBSCRIPTION`](/sql/drop-durable-subscription/)
+*   [`SUBSCRIBE`](/sql/subscribe/)
+*   [Resuming subscriptions](/transform-data/patterns/durable-subscriptions/)

doc/user/content/sql/alter-durable-subscription.md

@@ -0,0 +1,85 @@
+---
+title: "ALTER DURABLE SUBSCRIPTION"
+description: "`ALTER DURABLE SUBSCRIPTION` changes the acknowledgement deadline of a durable subscription, resets one that has expired, or transfers ownership."
+menu:
+  main:
+    parent: 'commands'
+---
+
+{{< private-preview />}}
+
+`ALTER DURABLE SUBSCRIPTION` changes the acknowledgement deadline of a [durable
+subscription](/sql/create-durable-subscription/), resets one that has expired, or
+transfers its ownership.
+
+## Syntax
+
+```mzsql
+ALTER DURABLE SUBSCRIPTION <name> SET (ACKNOWLEDGE WITHIN <interval>);
+ALTER DURABLE SUBSCRIPTION <name> RESET;
+ALTER DURABLE SUBSCRIPTION <name> OWNER TO <new_owner>;
+```
+
+| Field | Use |
+| --- | --- |
+| `SET (ACKNOWLEDGE WITHIN <interval>)` | Change how long you may go without acknowledging. Takes effect immediately, including for a subscription that is currently behind. |
+| `RESET` | Re-arm an expired subscription at the current time. |
+| `OWNER TO <new_owner>` | Transfer ownership to another role. |
+
+## Details
+
+### Changing the acknowledgement deadline
+
+Increasing the deadline gives a reader more time to recover, and increases the
+history that may be retained on its behalf. Decreasing it can expire a
+subscription immediately, if the reader has already gone longer than the new
+value without acknowledging.
+
+The value must fall between the system-wide minimum and maximum, the same bounds
+[`CREATE DURABLE SUBSCRIPTION`](/sql/create-durable-subscription/#acknowledgement-deadline)
+enforces. A minimum exists because the acknowledged position is recorded durably
+on an interval, so a deadline close to that interval would expire readers that
+are acknowledging correctly.
+
+### Resetting an expired subscription
+
+`RESET` moves an expired subscription to the current time and makes it usable
+again. It does not recover the history that was released when the subscription
+expired, so the next read must request `SNAPSHOT true` to get a usable starting
+state.
+
+Use `RESET` rather than dropping and recreating: it preserves the subscription's
+name, owner, and privileges. Requiring it, instead of silently resuming from
+whatever history happens to remain, is what keeps a gap in the data from passing
+unnoticed.
+
+`RESET` on a subscription that has not expired is an error. Fencing a live reader
+by resetting its position is not something to do by accident.
+
+## Examples
+
+```mzsql
+ALTER DURABLE SUBSCRIPTION winning_bids_feed SET (ACKNOWLEDGE WITHIN '5m');
+```
+
+```mzsql
+ALTER DURABLE SUBSCRIPTION winning_bids_feed RESET;
+```
+
+```mzsql
+ALTER DURABLE SUBSCRIPTION winning_bids_feed OWNER TO analytics_owner;
+```
+
+## Privileges
+
+The privileges required to execute this statement are:
+
+{{% include-headless "/headless/sql-command-privileges/alter-durable-subscription" %}}
+
+## Related pages
+
+*   [`CREATE DURABLE SUBSCRIPTION`](/sql/create-durable-subscription/)
+*   [`ACKNOWLEDGE`](/sql/acknowledge/)
+*   [`DROP DURABLE SUBSCRIPTION`](/sql/drop-durable-subscription/)
+*   [`SUBSCRIBE`](/sql/subscribe/)
+*   [Resuming subscriptions](/transform-data/patterns/durable-subscriptions/)