Engine-level CDC, webhooks, and change streams

[justrach]

Overview

Native change data capture by tailing the WAL — real-time event streams and HTTP webhooks triggered by data changes.

ON INSERT INTO orders → HTTP POST to customer's webhook
ON CHANGE IN users WHERE role = 'admin' → push event to stream

Why this matters

Supabase Realtime had to build a whole Elixir service to tail Postgres WAL externally. We own the WAL, so we can stream changes out natively. Customers get real-time events without polling — table stakes for any cloud database.

Implementation approach

WAL tailing

• WAL consumer that reads new entries as they're written
• Cursor tracking: each subscriber has a position in the WAL
• Catch-up reads for subscribers that fall behind
• Handle WAL compaction/rotation without losing events

Change streams (pull-based)

• Subscribe to changes on a collection or filter
• SSE (Server-Sent Events) endpoint: `GET /db/orders/changes?filter=...`
• Wire protocol: streaming change events on a persistent connection
• Cursor-based: client reconnects from last seen position

Webhooks (push-based)

• Register webhook: URL + filter + event types (insert/update/delete)
• HTTP POST with JSON payload on matching changes
• HMAC-SHA256 signature on webhook payloads (crypto module already has this)
• Retry with exponential backoff on delivery failure
• Webhook management: create, list, delete, pause

Event format

{
  "event": "insert",
  "collection": "orders",
  "tenant": "customer-a",
  "timestamp": "2026-03-29T...",
  "document": { ... },
  "previous": null
}

API surface

• HTTP: `POST /db/:collection/webhooks`, `GET /db/:collection/changes`
• Wire protocol: SUBSCRIBE command
• Admin: webhook management, subscriber lag monitoring

Dependencies

• WAL infrastructure (done)
• HMAC-SHA256 (done, in crypto module)
• HTTP server (done)

Effort: Medium

Priority: P3 in roadmap (table stakes for cloud DB)