You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Customers run operational workloads on Materialize.
@@ -12,6 +14,7 @@ To date, we have managed optimizer changes using feature flags (e.g., `enable_ca
12
14
Not every feature can be feature flagged (e.g., changing `MirRelationExpr` to hold `Repr*` types), though, and we do not have much in the way of tooling for feature flags.
13
15
14
16
It is hard for us to make changes in the optimizer that won't cause some customers to have a bad time---even if some customers have a much better time with those changes.
17
+
(Whence the name, "the customer trade-off problem".)
15
18
We need a way to change the optimizer without disrupting customer workloads.
16
19
17
20
## Success Criteria
@@ -45,7 +48,8 @@ It is very important that we solve the "optimizer image" problem (you should be
45
48
46
49
-**`optimizer-versions`** Separate optimizer versions, settable per-cluster using a system-level privilege.
47
50
-**`feature-flags`** Feature flag everything, building tooling to support eng, field eng, and customers.
48
-
-**`plan-pinning`** Offer an explicit way to fix a query plan.
51
+
-**`plan-pinning-v1`** Offer an explicit way to fix a query plan in a per-cluster way.
52
+
-**`plan-pinning-v2`** All plans are pinned by default.
49
53
-**`query-hints`** Offer query hints or special syntax to control query plans.
50
54
51
55
What are the pros and cons of each approach?
@@ -85,22 +89,36 @@ Cons:
85
89
- Who flips the bits? If it's us: high support burden. If it's someone else: what if they break things?
86
90
- Unknown support windows, and we have not historically done a good job managing feature flags.
87
91
88
-
### `plan-pinning`
92
+
### `plan-pinning-v1`
89
93
90
94
Pros:
91
95
96
+
+ Per-cluster control.
92
97
+ Ties in neatly with related ideas of "production clusters", guarantees, and auto-scaling.
93
98
+ Ties in neatly with related ideas of "DDIR" or some other stable, low-level interface.
94
99
+ Offers the most reliable possible experience---a fixed LIR plan would be stable even if bugfixes in MIR cause queries to change.
95
100
96
101
Cons:
97
102
98
103
- Any changes to the plan and you lose your pin. (Mitigation: use MVs on different clusters to separate the units you care about.)
104
+
- Any changes to other objects on the cluster.
99
105
- LIR is a not currently stored anywhere (but is a stable interface between MIR and rendering). DDIR does not actually exist.
100
106
- Once we are committed, may be hard to back out of. (Mitigation: deploy this is as an unstable feature with a customer partner.)
101
107
- More durable state.
102
108
- Need to manage migrations for LIR. (Mitigation: best effort.)
103
109
110
+
### `plan-pinning-v2`
111
+
112
+
Pros:
113
+
114
+
+ All of the positives of `plan-pinning-v1`.
115
+
+ Defaulting to pinned helps keep operational workloads operational.
116
+
+ Clean factoring of mechanism ("LIR is durable") from tools for its management (`COPY CLUSTER`, `EXPLAIN REPLAN ...`).
117
+
118
+
Cons:
119
+
120
+
- All of the cons of `plan-pinning-v1`, _except_ it's easier to make changes to objects on the cluster.
121
+
104
122
### `query-hints`
105
123
106
124
Pros:
@@ -120,15 +138,17 @@ Cons:
120
138
121
139
## Solution Proposal
122
140
123
-
We propose using **`plan-pinning`**.
141
+
We propose using **`plan-pinning-v2`**.
124
142
The ability to pin plans _exactly_ solves the **`our-bad`** problem.
125
143
It's also superior to the alternatives.
126
144
We see it as superior to **`feature-flags`** because we can work more flexibly (change types!) with less uncertainy (known configs!).
127
145
(Feature flags will of course continue to exist!)
128
146
We see it as superior to **`query-hints`** because we don't want to add query hints.
129
147
130
148
A prior version of this design doc and [the prior design doc in #30233](https://github.com/MaterializeInc/materialize/pull/30233) proposed **`optimizer-versions`**.
131
-
Why have we changed our minds?
149
+
A [prior version of this design doc in #35441](https://github.com/MaterializeInc/materialize/pull/35441) proposed **`plan-pinning-v1`**.
150
+
151
+
### Why did we switch to plan pinning?
132
152
133
153
`**optimizer-versions**` overfits to particular engineering challenges (wanting to make certain AST changes).
134
154
But recent work on repr types has shown that we can change the tires while the car is moving---we simply have to be careful.
@@ -137,21 +157,18 @@ Refactoring to have a clean optimizer crate is a good idea, but versioning is a
137
157
138
158
The balance tips further in **`plan-pinning`**'s favor when we consider that pinned plans are not merely a useful way for customers to have more confidence in Materialize, they are a way to help us identify clusters that are candidates for autoscaling and immediate incident escalation---production clusters.
139
159
140
-
##Minimal Viable Prototype
160
+
### Why did we switch to `plan-pinning-v2`?
141
161
142
-
We will pin plans at the level of clusters.
162
+
During broader conversations about the "cluster lifecycle", it became clear that it's much easier to simply treat _all_ plans as durable.
163
+
This obviates questions about freezing/unfreezing clusters and what to do when dependent objects change---we can reuse existing logic around `ALTER`, `DROP`, and `CREATE`.
164
+
During a business logic change blue/green swap, old definitions will be dropped and new plans will be written---but storing LIR plans by default means unchanged objects will keep their plans.
165
+
Treating `COPY CLUSTER` and `EXPLAIN REPLAN` as orthogonal but complementary features makes the design more compelling.
143
166
144
-
Two new DDL commands:
145
-
146
-
```sql
147
-
ALTER CLUSTER foo FREEZE;
148
-
ALTER CLUSTER foo UNFREEZE;
149
-
```
167
+
## Minimal Viable Prototype
150
168
151
-
We will store the LIR for all of the dataflows on `foo`, and automatically use those LIR plans on reboot.
152
-
These plans will be stored in the catalog.
153
-
No changes can be made to `foo`: no new dataflows, no removals.
154
-
It will not be part of this work, but it seems sensible to limit other actions on frozen clusters, e.g., you many only run fast-path `SELECT`s and `SUBSCRIBE`s (with the possible exception of queries that touch introspection sources).
169
+
Clusters will pin LIR plans (`DataflowDescription<LirRelationExpr>`s) by default.
170
+
These plans will be stored in persist shards, referenced in the catalog.
171
+
When a cluster starts up, it will attempt to read existing LIR plans and deploy _those_, rather than recompiling plans.
155
172
156
173
### What is the SLA?
157
174
@@ -160,107 +177,73 @@ At any point, we may simply throw up our hands and replan.
160
177
Users should be notified if pinned plans are replanned, but it should not necessarily rise to the level of pinging an on-call engineer---say, an escalation rather than an incident.
161
178
A possible success metric for plan pinning (beyond e.g., overall usage/number of pinned plans) is how _few_ replans are forced to occur.
162
179
163
-
### What can change?
164
-
165
-
Suppose we have the following dependency diagram, where `S` means "source", `V` means "view", `MV` means "materialized view", and `C` means cluster:
166
-
167
-
```mermaid
168
-
flowchart LR
169
-
subgraph C1
170
-
S1
171
-
end
172
-
subgraph C2
173
-
MV1
174
-
end
175
-
subgraph C3
176
-
MV2
177
-
end
178
-
179
-
S1 --> MV1 --> V2 --> MV2
180
-
```
181
-
182
-
If we freeze `C3`, we certainly can't make changes to `C3`.
183
-
What about `V2` (which is inlined into the definition of `MV2`)?
184
-
What about `MV1` (which is read from persist)?
185
-
What about `S1`?
186
-
We don't need to fix opinions permanently on these questions up front, but we will need to _have_ opinions to start.
187
-
188
-
As a first cut, it seems safe to say:
189
-
1. Frozen clusters block `DROP ... CASCADE` and must be unfrozen first.
190
-
2. Anything but `V2` may change in business logic; schema alterations have to be additive (cf. `ALTER MATERIALIZED VIEW`).
191
-
That is, we would treat persist as a barrier: a frozen cluster will block changes to things it depends on that are not persisted.
192
-
If `S1` or `MV1` change by altering a computation (but not its type) or by adding a new column, that should be fine (though we may need to generate an intermediate dataflow to project out the new column, since we will not want to change the pinned LIR plan of `MV2`).
193
-
194
-
It is important for us and for customers to know how things would be replanned if clusters were unfrozen.
195
-
In the long term, we will want to know which plans are pinned and how far those pins have drifted from what we would output.
196
-
It is merely engineering for us to know about drift in cloud---replan, emitting a diff of the new plan and the pinned one to store---but less so in self-managed.
197
-
(These diffs will not address the question of "which MZ changes caused the plan changes", though, and we will only get information at release time.
198
-
A proactive, Mztrail-like thing would help as well.)
199
-
We would want to make `mz-debug` aware of frozen clusters.
200
-
Some kind of `COPY CLUSTER` comand may help users experiment: copy a frozen cluster, unfreeze it, find a new plan you like, freeze _that_, and then green/blue the new one into place.
201
-
202
-
When using DBT, users should receive an error when attempting to make changes to a frozen cluster.
203
-
Whether that error is soft---i.e., the DBT run continues---or not is less clear.
204
-
205
-
### Why at the cluster level?
206
-
207
-
We propose freezing at the cluster level.
208
-
The environment and organization level is far too coarse.
209
-
The replica and dataflow levels are too fine---freezing these but not the rest of the cluster seems like a recipe for confusion (two replicas on the same cluster with different plans? multiple versions of dependencies?).
210
-
211
-
### How does LIR change?
180
+
### What happens at an upgrade?
212
181
213
-
LIR is the interface between the optimizer and rendering.
214
-
While "stable", it's not persisted and has a purely internal contract.
182
+
During a 0dt upgrade, the new environmentd will read the catalog and spin up dataflows for the plans recorded in persist shards.
183
+
This way, the new environment will continue to operate with the existing plans---stability!
184
+
We may need to migrate these plans if, e.g., there was a change in the LIR definition.
185
+
(We use a [schema registry](https://github.com/MaterializeInc/materialize/pull/37814) to track the need for these migrations.)
215
186
216
-
LIR has a large surface---`MirScalarExpr` (and with it, `UnaryFunc`, `BinaryFunc`, `VariadicFunc`, and `UnmaterializableFunc`), `AggregateExpr` and `AggregateFunc`, `TableFunc`, `Row`, etc.---and any changes across that surface could cause a pinned plan to no longer be runnable.
217
-
It would be unwise to freeze things as they are in place: there is no support for migrations, and `MirScalarExpr` and the various `*Func` would be locked in time.
218
-
Worse still, several of these `*Func` types reference external types, like `regex::Regex`---if we froze these things, upgrading the `regex` crate would break pinned plans and/or force a migration.
187
+
### What happens when business logic changes?
219
188
220
-
We propose the following shift:
189
+
Whenever a planned object (materialized view; index) changes, it must generally be dropped and then recreated.
190
+
Here, dropping loses the old pinned plan; recreating the object stores a new one.
191
+
There is a challenge, however: what if only a few objects need to change, but other plans should stay the same?
221
192
222
-
- Generate `Lir` versions of the various `*Func`s. This will require a preparatory PR to improve the associated macros, but should not be too complex (even though it generates a fair bit of code; we can engineer that code to point to common implementations of `eval`, so we won't get major code duplication). We will likely want to fold this in to work that parameterizes type holding `*Func`s to hold _either_`Sql*Type` or `Repr*Type`.
223
-
+ Some of the `TableFunc`s hold on to types for typechecking, but don't need those types for evaluation. We could leave these and `AggregateFunc`s alone.
224
-
+ Many of the other `*Func`s have functions that use their stored types in their `eval` method... they _must_ hold `Sql` types.
The proposed solution is a `COPY CLUSTER foo TO bar` command that creates a cluster `bar` that is identical to `foo`, i.e., it has the same plans.
194
+
One can then alter objects on `bar`, dropping and recreating only what's needed.
195
+
When `bar` is satisfactory, a blue/green swap deploys the new configuration.
196
+
Such an interactive usage should be permissible, but will not be best practice: one should really use a tool like mzdeploy to do a "slim deployment".
227
197
228
-
- Write `LirScalarExpr`, which omits `CallUnmaterializable` (because it should be resolved before LIR).
229
-
- Ensure that we only serialize the MZ-controlled bits. Wherever `Lir*Func` would include an external structure, ensure we store the information that lets us regenerate that structure using `#[serde(default=...)]` (e.g., store the original regex string, not the `regex::Regex` value).
230
-
- Add a version number to top-level `LIR`.
231
-
- Use `serde_reflection` to generate a schema (using `Registry`) that lives in the repo. Any change in LIR serialization will yield a schema change. We can programmatically require migrations/LIR version bumps for certain kinds of schema changes.
232
-
+ We _must_ eventually tolerate schema changes like new LIR operators, new `*Func`s, new fields on existing structures, or pins will break too often.
198
+
We will need to adapt mzdeploy to use `COPY CLUSTER` for slim deployments.
199
+
We may want to do the same for DBT.
200
+
In either case, we will likely want `COPY CLUSTER` to create a cluster with _no_ replicas.
201
+
This way, the user can arrange objects the way they like and _then_ create replicas---simulating a full hydration, giving them confidence that their definitions fit in the cluster.
202
+
There is some footgun risk here, and we propose that bare uses of `COPY CLUSTER` create replicas, but that slim deployments should run `COPY CLUSTER ... WITH (REPLICATION FACTOR = 0)`, i.e., carefully ensure that no dataflows will actually be created at first, and later running `ALTER CLUSTER` to set a larger replication factor.
233
203
234
-
A `DataflowDescription<LIR>` will be stored as JSON in a persist shard pointed to by the catalog.
235
-
Splitting up the LIR in this way means that (a) the catalog doesn't scale (as much) with object plan size and (b) we can parallelize parsing of LIR plans.
204
+
### How do users get improvements?
236
205
237
-
## Open questions
206
+
As the optimizer improves, existing pinned plans may lag behind our best possible performance.
207
+
How do we ensure that users can see the improvements we make?
238
208
239
-
### What is the concrete syntax?
209
+
Users changing business logic will necessarily use the latest version of the optimizer, so new objects will enjoy the benefits.
210
+
We should also support a user-facing `EXPLAIN REPLAN` for individual objects (what would my plan look like now?) and `EXPLAIN REPLAN CLUSTER` (how have my plans "drifted" from what I would get if I replanned today?).
211
+
In an ideal world, we could offer insight into the diffs of the replanned objects.
240
212
241
-
We should pick ergonomic, sensible DDL syntax for this.
242
-
How will users interact with it in DBT, etc.?
213
+
Users may also get improvements when we cannot load the LIR for some reason, e.g., a deliberate backwards incompatible migration, a bug.
243
214
244
-
### How do alterations work?
215
+
### How does LIR change?
245
216
246
-
Consider the dependency diagram above (["What can change?"](#what-can-change)).
247
-
What are the pragmatics of making changes to `S1` and `MV1`?
248
-
These changes could be changes to business logic or the addition of a column.
217
+
Several PRs have rearranged MIR and LIR.
218
+
- New structures
219
+
+[#36544 Defined a stable LIR scalar expression](https://github.com/MaterializeInc/materialize/pull/36544)
+[#37814 LIR schema registry to support migrations](https://github.com/MaterializeInc/materialize/pull/37814)
249
227
250
-
- What does the workflow look like for changes to business logic upstream? _Should be fine._
251
-
- What about additive schema changes? _Should be fine._
252
-
- What about changes to a hypothetical `MV3` what depends on `MV2`? _Should be fine._
228
+
We cut things such that MIR and LIR share the `*Func` definitions and other infrastructure, though there is some work towards pulling them further apart (e.g., [#37409 measure how often we need to dip back into MIR to propagate literal constraints in LIR](https://github.com/MaterializeInc/materialize/pull/37409)).
253
229
254
-
Suppose the logic in `MV2` is broken.
255
-
What is the workflow for fixing that?
256
-
_Any_ change to `MV2` will lead to replanning, and there is no possibility of "spot" fixes.
257
-
How do we communicate to customers that pinning is "best effort"?
230
+
A `DataflowDescription<LIR>` will be stored as JSON in a persist shard pointed to by the catalog.
231
+
Splitting up the LIR in this way means that (a) the catalog doesn't scale (as much) with object plan size and (b) we can parallelize parsing of LIR plans.
258
232
259
233
### Where does LIR live in the catalog?
260
234
261
-
It could go in `CatalogPlans` (all currently keyed by global ID), in `CatalogState` (alongside per-cluster metadata), or as a sidecar (like the `ExpressionCacheHandle`).
262
-
This part of the catalog seems to be in flux, but `CatalogState` seems right---plans for frozen clusters should be stored alongside those clusters.
263
-
That is, whether or not a cluster is frozen is not a boolean---it's an `Option<PersistShard pointer>`, with `Some(...)` holding the name of the shard that holds plans.
235
+
It in `CatalogState` (alongside the `CatalogItem`'s entry), or as a sidecar in `Catalog` itself (like the `ExpressionCacheHandle`).
236
+
This part of the catalog seems to be in flux, but `CatalogState` seems right---pinned plans should be stored with their associated IDs.
237
+
238
+
Storing a pointer to a persist shard holding the JSON-serialized LIR plan instead of putting the full plan in the catalog itself offers several benefits.
239
+
First, we can decode and migrate plans in parallel.
240
+
Second, we don't need to send a serialized plan to clusters---we can just point them at persist.
241
+
Third, it means that the catalog doesn't scale with object _size_ (though of course it scales with object _count_).
242
+
243
+
Plans should be written eagerly.
244
+
Migration can occur when we load a plan, but that means writing a new persist shard with the migrated plan.
245
+
246
+
## Open questions
264
247
265
248
### What happens when we move to DDIR?
266
249
@@ -271,7 +254,15 @@ If we later decide to save the DDIR rather than LIR, that will be as transparent
271
254
272
255
### What happens to `EXPLAIN OPTIMIZED PLAN` for pinned LIR plans?
273
256
257
+
`EXPLAIN OPTIMIZED PLAN` shows MIR by... compiling to MIR.
258
+
But an LIR plan pinned six months ago may have nothing to do with the MIR we get now---and so this `EXPLAIN` is showing the explain of the _replan_, not the original plan.
259
+
(`EXPLAIN PHYSICAL PLAN` is the default, and will not have this problem.)
260
+
274
261
Right now, I believe only Gábor uses `EXPLAIN OPTIMIZED PLAN`.
275
262
But in the event someone would like to see MIR for a pinner LIR plan, we would simply not have it.
276
263
We _could_ store a cached version, either a text (with fixed options) or as some kind of structure---though our aim was to _not_ have to serialize MIR.
277
264
I think the best approach here is to improve the default LIR-based `EXPLAIN PLAN` enough so that Gábor stops using `EXPLAIN OPTIMIZED PLAN`.
265
+
266
+
### How does this interact with the expression cache?
267
+
268
+
We will likely be able to use pinned LIR to deprecate the expression cache, but there should be no interference at first---though we will want to carefully prioritize which we consult (LIR first, then fall back to the cache).
0 commit comments