storage: bail on concurrent shard modification for non-upsert ingestions - #38361
Closed
peterdukelarsen wants to merge 1 commit into
Closed
storage: bail on concurrent shard modification for non-upsert ingestions#38361peterdukelarsen wants to merge 1 commit into
peterdukelarsen wants to merge 1 commit into
Conversation
The source persist sink tolerates a compare_and_append upper mismatch by trimming its batch to the shard's actual upper and continuing, on the assumption that whoever advanced the upper wrote the data we would have written. That assumption is only sound for pipelines running the continual feedback upsert operator, which reads the shard back and emits corrections, so concurrent instances converge. The leniency was gated on the global storage_use_continual_feedback_upsert flag (default true), which made it apply to every ingestion, including ENVELOPE NONE CDC pipelines that have no such reconciliation. For those pipelines a concurrent writer is a replaced incarnation of the same ingestion whose output can encode a different consistency point. A network partition between the controller and a cluster leaves the old dataflow running as a zombie while its snapshot batch sits in indeterminate blob-write retries, and the reconciled replacement snapshots again at a later upstream position. If the zombie's batch lands first, trimming fuses the zombie's snapshot with the replacement's rewinds and replication into a state that never existed upstream. Observed as SS-427 in the invariants chaos harness: the conservation checker on a newly added MySQL table read a total off by exactly one upstream transaction, durable in the table's history, converging only as later updates rewrote the affected rows. Scope the leniency to exports whose envelope actually runs the feedback upsert operator, mirroring the operator selection in render::sources. Everything else returns to the pre-feedback-upsert behavior on mismatch: fail the append, restart the dataflow, and resume from the shard's actual committed state, which is consistent no matter which incarnation's prefix landed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NJj8MYhExdJXqMc37iftuN
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Results of poking around at SS-427 with claude.
The source persist sink tolerates a compare_and_append upper mismatch by trimming its batch to the shard's actual upper and continuing, on the assumption that whoever advanced the upper wrote the data we would have written. That assumption is only sound for pipelines running the continual feedback upsert operator, which reads the shard back and emits corrections, so concurrent instances converge. The leniency was gated on the global storage_use_continual_feedback_upsert flag (default true), which made it apply to every ingestion, including ENVELOPE NONE CDC pipelines that have no such reconciliation.
For those pipelines a concurrent writer is a replaced incarnation of the same ingestion whose output can encode a different consistency point. A network partition between the controller and a cluster leaves the old dataflow running as a zombie while its snapshot batch sits in indeterminate blob-write retries, and the reconciled replacement snapshots again at a later upstream position. If the zombie's batch lands first, trimming fuses the zombie's snapshot with the replacement's rewinds and replication into a state that never existed upstream. Observed as SS-427 in the invariants chaos harness: the conservation checker on a newly added MySQL table read a total off by exactly one upstream transaction, durable in the table's history, converging only as later updates rewrote the affected rows.
Scope the leniency to exports whose envelope actually runs the feedback upsert operator, mirroring the operator selection in render::sources. Everything else returns to the pre-feedback-upsert behavior on mismatch: fail the append, restart the dataflow, and resume from the shard's actual committed state, which is consistent no matter which incarnation's prefix landed.