Skip to content

Commit 3dfcb36

Browse files
committed
adapter: let the synchronous cut-over change the replication factor
Changing a cluster's replication factor while a graceful reconfiguration is in progress is refused, because the record captures a target factor when written and the controller's cut-over writes it back to the realized config a tick later. A factor change applied independently in the meantime would be silently clobbered, with nothing in the ALTER's response to say so. That reasoning does not reach the synchronous cut-over. It folds the factor into the target it transacts and retires the record in the same transaction, and no consumer reads a retired record's target: `desired_replicas`, `target_hydrated` and the controller's own cut-over all sit behind `is_in_progress()`. There is no later cut-over left to clobber the write, so exempt it. The refusal cost the escape hatch its most useful shape. Forcing a wedged resize through while scaling down had to be two statements, and between them the cluster ran the old factor at the new size, the most expensive combination available, exactly while an operator is firefighting. `fold_reconfiguration_target` already computed the right answer for a re-targeted factor; only the guard made that branch unreachable. The escape hatch is the only exemption. A zero-timeout rollback still leaves a record in flight to settle on a tick, so it keeps the refusal, and the error hint now points at the cut-over as the third way out. The cut-over section gains the wedged case: the refusal still fires without a `WAIT`, and a combined size-and-factor cut-over lands both dimensions, converges the replica set, and settles the record `cancelled` because the record's own target factor was abandoned. The existing factor case ran with no record in flight, which is why this gap held. Also drops `Op::UpdateClusterReplicaConfig`. Deleting the staged reconfiguration machine removed its last producer, leaving the variant, its apply arm and its audit match arm dead.
1 parent dfd8fe6 commit 3dfcb36

5 files changed

Lines changed: 48 additions & 31 deletions

File tree

src/adapter/src/catalog/transact.rs

Lines changed: 0 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -242,11 +242,6 @@ pub enum Op {
242242
/// this write, alongside the record carried in `config`.
243243
burst_audit: Option<BurstAudit>,
244244
},
245-
UpdateClusterReplicaConfig {
246-
cluster_id: ClusterId,
247-
replica_id: ReplicaId,
248-
config: ReplicaConfig,
249-
},
250245
UpdateItem {
251246
id: CatalogItemId,
252247
name: QualifiedItemName,
@@ -2990,24 +2985,6 @@ impl Catalog {
29902985
)?;
29912986
}
29922987
}
2993-
Op::UpdateClusterReplicaConfig {
2994-
replica_id,
2995-
cluster_id,
2996-
config,
2997-
} => {
2998-
let replica = state.get_cluster_replica(cluster_id, replica_id).to_owned();
2999-
info!("update replica {}", replica.name);
3000-
tx.update_cluster_replica(
3001-
replica_id,
3002-
mz_catalog::durable::ClusterReplica {
3003-
cluster_id,
3004-
replica_id,
3005-
name: replica.name.clone(),
3006-
config: config.clone().into(),
3007-
owner_id: replica.owner_id,
3008-
},
3009-
)?;
3010-
}
30112988
Op::UpdateItem { id, name, to_item } => {
30122989
// A non-temporary item must not depend on a temporary one.
30132990
// Temporary objects are session-scoped and never persisted, so

src/adapter/src/coord/ddl.rs

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1374,7 +1374,6 @@ impl Coordinator {
13741374
| Op::UpdateOwner { .. }
13751375
| Op::RevokeRole { .. }
13761376
| Op::UpdateClusterConfig { .. }
1377-
| Op::UpdateClusterReplicaConfig { .. }
13781377
| Op::UpdateSourceReferences { .. }
13791378
| Op::UpdateSystemConfiguration { .. }
13801379
| Op::ResetSystemConfiguration { .. }

src/adapter/src/coord/sequencer/inner/cluster.rs

Lines changed: 16 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -297,7 +297,17 @@ impl Coordinator {
297297
// flight would be silently clobbered at cut-over. Refused even when the
298298
// same statement also re-targets the shape, so a record's target
299299
// replication factor is always the one it started with.
300-
if reconfiguration_in_flight && !matches!(options.replication_factor, Unchanged) {
300+
//
301+
// The synchronous cut-over is exempt: it folds the factor into the
302+
// target it transacts and retires the record in the same transaction,
303+
// so there is no later cut-over left to clobber the write. Without the
304+
// exemption the escape hatch could not force a wedged reshape through
305+
// and change the factor in one statement, which is the shape of the
306+
// request when a stuck resize is being scaled down to cut cost.
307+
if reconfiguration_in_flight
308+
&& !matches!(options.replication_factor, Unchanged)
309+
&& !requests_immediate_cut_over(strategy)
310+
{
301311
return Err(AdapterError::AlterClusterReplicationFactorWhileReconfiguring);
302312
}
303313

@@ -2290,11 +2300,11 @@ fn alter_reconfiguration_target(
22902300
/// AZ-only) from silently reverting the in-flight transition along every dimension
22912301
/// it did not mention.
22922302
///
2293-
/// Replication factor folds the same way, but only matters for the
2294-
/// nothing-in-flight case: a change to it while a reconfiguration is in
2295-
/// flight is refused before an `ALTER` reaches here, so
2296-
/// `unchanged.replication_factor` is always `true` when `in_flight` is
2297-
/// `Some`.
2303+
/// Replication factor folds the same way. A change to it while a reconfiguration
2304+
/// is in flight reaches here only on the synchronous cut-over, which transacts
2305+
/// the folded target and retires the record together. Every other path refuses
2306+
/// such a change, so `unchanged.replication_factor` is `false` under an
2307+
/// `in_flight` target only for a cut-over.
22982308
fn fold_reconfiguration_target(
22992309
in_flight: Option<&ReconfigurationTarget>,
23002310
new_target: ReconfigurationTarget,

src/adapter/src/error.rs

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -746,7 +746,9 @@ impl AdapterError {
746746
),
747747
AdapterError::AlterClusterReplicationFactorWhileReconfiguring => Some(
748748
"Cancel the reconfiguration by altering the cluster back to its current \
749-
configuration, or wait for it to settle, then change the replication factor."
749+
configuration, or wait for it to settle, then change the replication factor. \
750+
To commit the reconfiguration and change the replication factor in one \
751+
statement, add WITH (WAIT FOR '0s')."
750752
.to_string(),
751753
),
752754
AdapterError::AlterClusterScheduleWhileReconfiguring => Some(

test/testdrive/cluster-controller.td

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1526,6 +1526,35 @@ scale=1,workers=2 2
15261526
> SELECT r.size, count(*) FROM mz_cluster_replicas r JOIN mz_clusters c ON r.cluster_id = c.id WHERE c.name = 'cc_cutover' GROUP BY r.size
15271527
scale=1,workers=2 2
15281528

1529+
# A factor change while a record is in flight stays refused on every path that
1530+
# leaves the record in flight: the record's own cut-over would write its target
1531+
# factor a tick later and silently clobber the change.
1532+
> ALTER CLUSTER cc_cutover SET (SIZE 'scale=1,workers=4')
1533+
1534+
> SELECT recon.status FROM mz_internal.mz_cluster_reconfigurations recon JOIN mz_clusters ON mz_clusters.id = recon.cluster_id WHERE mz_clusters.name = 'cc_cutover'
1535+
in-progress
1536+
> SELECT r.size, count(*) FROM mz_cluster_replicas r JOIN mz_clusters c ON r.cluster_id = c.id WHERE c.name = 'cc_cutover' GROUP BY r.size
1537+
scale=1,workers=2 2
1538+
scale=1,workers=4 2
1539+
1540+
! ALTER CLUSTER cc_cutover SET (REPLICATION FACTOR 3)
1541+
contains:cannot change replication factor while a reconfiguration is in progress
1542+
1543+
# The synchronous cut-over is exempt: it folds the factor into the target it
1544+
# transacts and retires the record in the same transaction, so nothing survives
1545+
# to clobber the write. This is the escape hatch forcing a wedged resize through
1546+
# and scaling down in one statement, rather than paying the old factor at the new
1547+
# size until a second ALTER lands. The record wanted factor 2, so its target was
1548+
# abandoned and it settles `cancelled` rather than `finalized`.
1549+
> ALTER CLUSTER cc_cutover SET (SIZE 'scale=1,workers=4', REPLICATION FACTOR 1) WITH (WAIT FOR '0s')
1550+
1551+
> SELECT size, replication_factor FROM mz_clusters WHERE name = 'cc_cutover'
1552+
scale=1,workers=4 1
1553+
> SELECT r.size, count(*) FROM mz_cluster_replicas r JOIN mz_clusters c ON r.cluster_id = c.id WHERE c.name = 'cc_cutover' GROUP BY r.size
1554+
scale=1,workers=4 1
1555+
> SELECT recon.status FROM mz_internal.mz_cluster_reconfigurations recon JOIN mz_clusters ON mz_clusters.id = recon.cluster_id WHERE mz_clusters.name = 'cc_cutover'
1556+
cancelled
1557+
15291558
> DROP CLUSTER cc_cutover CASCADE
15301559
> DROP TABLE cc_cutover_t
15311560

0 commit comments

Comments
 (0)