Skip to content

Commit 0fa0d3e

Browse files
tylerhartwigclaude
andcommitted
docs: remove RDS Blue/Green approach from PG major-version upgrade guide
The Blue/Green approach could not deliver the zero-downtime property the guide is named for. RDS refuses to create the deployment while a Materialize source holds a slot on blue, so the only way to start one is to drop the source -- which cascades the downstream dataflow -- and the new source must re-snapshot against green regardless. The resulting staleness window spans provisioning plus a full re-snapshot. Replace it with a "Why managed Blue/Green deployments don't work" section that keeps the reproduced constraint and its consequences, so readers who reach for RDS Blue/Green learn where it blocks. Also switch the cutover from hand-waved "repoint consumers" to an atomic ALTER SCHEMA ... SWAP WITH, and carry over the dbt blue/green caveats about sinks needing a dedicated schema and cluster and active SUBSCRIBEs breaking at the swap. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent d73693f commit 0fa0d3e

1 file changed

Lines changed: 74 additions & 97 deletions

File tree

‎doc/user/content/ingest-data/postgres/major-version-upgrade.md‎

Lines changed: 74 additions & 97 deletions
Original file line numberDiff line numberDiff line change
@@ -20,25 +20,26 @@ new instance running the new version, and the replication slot does **not**
2020
carry over automatically. How you handle the slot determines whether reads stay
2121
fresh through the upgrade.
2222

23-
This guide covers two approaches:
24-
25-
| Approach | Reads stay fresh? | Who moves the data? | Use when |
26-
|----------|-------------------|---------------------|----------|
27-
| [Parallel source (self-managed)](#approach-a-parallel-source-self-managed) | **Yes, continuously** | You (native logical replication) | Zero read downtime is a hard requirement. |
28-
| [Amazon RDS Blue/Green](#approach-b-amazon-rds-bluegreen-managed) | No — bounded freshness gap | AWS | A brief staleness window is acceptable and you want AWS to manage the cutover. |
23+
The approach in this guide keeps Materialize continuously fresh. You keep the
24+
existing source running against the old primary the entire time, build the
25+
new-version primary in parallel, and hydrate a second source against it before
26+
cutting consumers over.
2927

3028
{{< note >}}
31-
Both approaches involve a brief **write** freeze on the application at the
32-
moment of cutover, which is inherent to any major-version upgrade. The
33-
difference is whether Materialize keeps serving fresh **reads** throughout.
29+
This procedure involves a brief **write** freeze on the application at the
30+
moment of cutover, which is inherent to any major-version upgrade. Materialize
31+
keeps serving fresh **reads** throughout.
3432
{{< /note >}}
3533

36-
## Approach A: Parallel source (self-managed)
34+
{{< warning >}}
35+
Managed upgrade services that swap the primary out from under Materialize — such
36+
as [Amazon RDS Blue/Green
37+
deployments](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/blue-green-deployments.html)
38+
— cannot be used while a Materialize source is attached. See [Why managed
39+
Blue/Green deployments don't work](#why-managed-bluegreen-deployments-dont-work).
40+
{{< /warning >}}
3741

38-
This is the only approach that keeps Materialize continuously fresh. You keep
39-
the existing source running against the old primary the entire time, build the
40-
new-version primary in parallel, and hydrate a second source against it before
41-
cutting consumers over.
42+
## Upgrade with a parallel source
4243

4344
```text
4445
app writes ──▶ PG (old major) ──native logical replication──▶ PG (new major)
@@ -145,110 +146,86 @@ Once the new source has caught up, perform a coordinated cutover:
145146
1. **Drain both sources** in Materialize so they reflect the frozen state, then
146147
verify the fingerprint matches across the old primary, new primary, existing
147148
source, and new source.
148-
1. **Repoint consumers** from the existing source's objects to the `upgrade`
149-
schema's objects.
150-
1. **Resume application writes**, now pointed at the new primary.
151-
152-
Throughout the cutover, reads against Materialize stay live and fresh — the
153-
existing source serves until the flip, and the new source is already caught up
154-
at the flip.
149+
1. **Swap the schemas.** Rather than repointing each consumer, swap the
150+
production schema with the `upgrade` schema. The swap is atomic, so
151+
consumers keep referencing the same schema-qualified names and move together
152+
at a single instant:
155153
156-
### 5. Decommission
154+
```mzsql
155+
ALTER SCHEMA public SWAP WITH upgrade;
156+
```
157157
158-
Drop the subscription on the new primary, drop the old source in Materialize,
159-
and decommission the old primary.
158+
To roll back, run the same statement again.
160159
161-
## Approach B: Amazon RDS Blue/Green (managed)
160+
1. **Resume application writes**, now pointed at the new primary.
162161
163-
[Amazon RDS Blue/Green deployments](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/blue-green-deployments.html)
164-
let AWS build the upgraded "green" instance, keep it in sync, and swap
165-
endpoints at switchover. This is operationally simpler, but it **cannot keep
166-
Materialize continuously fresh** because of the constraint below.
162+
Throughout the cutover, reads against Materialize stay live and fresh — the
163+
existing source serves until the swap, and the new source is already caught up
164+
at the swap.
167165
168166
{{< warning >}}
169-
**A Blue/Green deployment cannot be created while a Materialize source is
170-
attached to the blue instance.** Materialize's logical replication slot counts
171-
as *external replication*, and RDS refuses to create the deployment
172-
(`CREATING_READ_REPLICA_OF_SOURCE` fails with *"external replication on the blue
173-
primary instance"*). You must [drop the
174-
source](/sql/drop-source/) first, which starts an unavoidable **freshness
175-
gap**. No RDS flag bypasses this.
167+
If you have [sinks](/sql/create-sink/), create them in a **dedicated schema and
168+
cluster** that is excluded from the swap. Sinks must not be recreated as part of
169+
a blue/green cutover; instead, cut them over to the new definition of their
170+
upstream dependencies after the swap. This mirrors the guidance in [blue/green
171+
deployments with dbt](/manage/dbt/blue-green-deployments/).
176172
{{< /warning >}}
177173
178-
### 1. Detach Materialize and create the deployment
179-
180-
1. `DROP SOURCE` in Materialize to release the replication slot. Confirm
181-
`pg_replication_slots` no longer lists Materialize's slot. **The freshness
182-
gap starts here.**
183-
1. Create the Blue/Green deployment, targeting the new major version with a
184-
parameter group that has `rds.logical_replication = 1`. Wait for the green
185-
instance to become **Available** (a major-version upgrade can take tens of
186-
minutes; it scales with database size).
187-
188-
The publication and `REPLICA IDENTITY` settings are copied from blue into green.
189-
190-
### 2. Attach Materialize to green
191-
192-
The green instance is a logical replica that is itself a primary
193-
(`pg_is_in_recovery()` returns `false`), so it can host Materialize's slot.
194-
Create a source against **green's temporary endpoint** and let it snapshot and
195-
catch up:
196-
197-
```mzsql
198-
CREATE SOURCE my_source
199-
FROM POSTGRES CONNECTION pg_green (PUBLICATION 'mz_source')
200-
FOR ALL TABLES;
201-
```
202-
203-
**The freshness gap ends when this source has caught up.** As in Approach A,
204-
snapshot time scales with data volume.
205-
206-
### 3. Switch over and repoint the source
207-
208-
Trigger the Blue/Green switchover. AWS promotes green **in place** — it takes
209-
over the production endpoint, while the old blue is demoted and renamed. The
210-
switchover completes in tens of seconds, during which the application has a
211-
brief write pause.
212-
213-
Because green is promoted in place (same instance, same LSN timeline), its
214-
replication slot is preserved. Repoint the source's connection at the
215-
production endpoint so Materialize resumes from the **same slot with no
216-
re-snapshot**:
217-
218-
```mzsql
219-
ALTER CONNECTION pg_green SET (HOST = 'PRODUCTION_ENDPOINT') WITH (VALIDATE = true);
220-
```
221-
222-
After the switchover, green's temporary endpoint is removed; a source restart
223-
would otherwise fail to resolve it, so run the `ALTER CONNECTION` promptly.
224-
225174
{{< note >}}
226-
This seamless resume works **only** because a Blue/Green switchover promotes
227-
green in place, preserving the slot and LSN timeline. Repointing a source at an
228-
unrelated instance (one that doesn't carry the same slot) forces a fresh
229-
snapshot instead.
175+
Any active [`SUBSCRIBE`](/sql/subscribe/) commands attached to the swapped
176+
cluster(s) will break at the swap. On retry, the client automatically connects
177+
to the newly deployed cluster.
230178
{{< /note >}}
231179
232-
### 4. Clean up
180+
### 5. Decommission
181+
182+
Drop the subscription on the new primary, drop the old source in Materialize,
183+
and decommission the old primary.
233184
234-
Delete the Blue/Green deployment and the demoted old instance.
185+
## Why managed Blue/Green deployments don't work
186+
187+
[Amazon RDS Blue/Green deployments](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/blue-green-deployments.html)
188+
let AWS build the upgraded "green" instance, keep it in sync, and swap endpoints
189+
at switchover. This is an appealing shortcut, but it is **not compatible with an
190+
attached Materialize source**.
191+
192+
Creating the deployment fails at `CREATING_READ_REPLICA_OF_SOURCE` with *"external
193+
replication on the blue primary instance"*. One of the documented prerequisites
194+
is that the DB instance is not the source or target of external replication, and
195+
Materialize's logical replication slot is external replication. No RDS setting
196+
bypasses this.
197+
198+
Releasing the slot — by [dropping the source](/sql/drop-source/) — does let the
199+
deployment be created, but that is not a viable production step:
200+
201+
- `DROP SOURCE` defaults to `RESTRICT` and fails when the source has dependents.
202+
Forcing it with `CASCADE` drops the entire downstream dataflow — tables, views,
203+
materialized views, indexes, and sinks — which means rebuilding and rehydrating
204+
your environment, not a brief pause.
205+
- Even after the switchover, Materialize's original slot on blue is unusable, so
206+
the new source must snapshot from scratch against green.
207+
208+
The result is a staleness window that begins when you detach from blue and lasts
209+
through provisioning and a full re-snapshot. Use the [parallel
210+
source](#upgrade-with-a-parallel-source) procedure above instead: it keeps the
211+
existing source serving fresh for the entire upgrade, and the only coordinated
212+
pause is the write freeze at cutover.
235213

236214
## Considerations
237215

238-
- **DDL is not replicated.** For Approach A, pre-create the target schema on the
239-
new instance before subscribing.
216+
- **DDL is not replicated.** Pre-create the target schema on the new instance
217+
before subscribing.
240218
- **Sequences are not advanced on the target** by native logical replication.
241-
Synchronize them at cutover (Approach A) or RDS handles it at switchover
242-
(Approach B).
219+
Synchronize them at cutover.
243220
- **`REPLICA IDENTITY FULL`** must be set on replicated tables so Materialize
244221
captures all column values on updates and deletes.
245-
- **`ALTER CONNECTION ... SET (HOST = ...)`** resumes from the existing slot
246-
only when the target instance preserves that slot (an in-place promotion).
247-
Otherwise Materialize re-snapshots.
222+
- **A publication can feed multiple subscribers**, so Materialize's slot and the
223+
upgrade subscription's slot coexist on the old primary without conflict.
248224

249225
## Related pages
250226

251227
- [Ingest data from Amazon RDS](/ingest-data/postgres/amazon-rds/)
252228
- [Guide: Ingest from a dedicated PostgreSQL replica](/ingest-data/postgres/logical-replica/)
253229
- [Guide: Handle upstream schema changes with zero downtime](/ingest-data/postgres/source-versioning/)
254-
- [`ALTER CONNECTION`](/sql/alter-connection/)
230+
- [Blue/green deployments with dbt](/manage/dbt/blue-green-deployments/)
231+
- [`ALTER SCHEMA`](/sql/alter-schema/)

0 commit comments

Comments
 (0)