CASSANDRA-20383 #4385
Draft
CASSANDRA-20383 #4385
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
TRANSFER PARTS FROM CHERRY-PICK REBASE-FIXES: MOVING AWAY FROM BOUNDARIES TOWARDS OFFSETS Rebase: misc build fixes REBASE 20383 Attach transfer IDs to SSTables, so transfers can be read-reconciled Empty commit: progress report below The current state of this branch has plenty of TODOs and review notes I'm thinking through, and I'm still debugging why MutationTrackingTest#testTrackedImport is failing. The design I've landed on for Bulk Transfer looks like this: 1. Transfers are streamed to replicas, then once replicas have received their streams durably, the coordinator activates the streams by assigning a Transfer ID (a plain MutationId) for that stream, shared across all replicas. Mutation Tracking Invariant: Any values present in a data response must either be present in a summary, or be part of the repaired set. If we just imported data on one instance without assigning a MutationId, we'd violate that invariant. 2. Transfer IDs are kept in SSTables, rather than CoordinatorLogs When we do a data read, we need to include Transfer IDs in the summary for any transferred SSTables visible in the View at the time of the read. Concurrent reads can reference different Views, and we need to avoid writes tearing between the LSM and summaries. We could have used another Shard / CoordinatorLog concept to track Transfer IDs, but then we'd need to synchronize LifecycleTransaction's changes to the live set with changes to the CoordinatorLog index. A vast majority of the load (from CQL writes) will flow through CoordinatorLog, and synchronizing to SSTable changes is something we want to avoid. Open Questions: - Avoiding the race condition between TransferActivation and ownership changes in TCM. I know TCM's ownership changes are initially additive, but is there a barrier between expanding the write voting group? Best solution is probbaly to attach an epoch on the TransferActivation preflight and let replicas reject if they know about an upcoming conflicting ownership change. - How do we want to retain bulk transfers on witnesses? We don't want to stream on the read path, but we could stream as part of background reconciliation. But transfers will already fail if they can't be persisted durably on all replicas, before activation, so we gain very little by replicating transfers everywhere. Fix MutationTrackingTest#testTrackedImport with two-phase activation Add test for activation on the read-reconcile path Add failing test for resurrection of pending imports after bounce WIP: refactor test Fix resurrection of pending tracked imports after bounce Add failing test for import with node down Rename transferId->activationId, since a transfer needs its own ID since it contains multiple plan IDs, and we don't want to assign an activationId until ready to activate WIP: Refactor transfer streaming to support replicas down during import WIP: Refactor transfer streaming to support replicas down during import WIP: Replay missing transfers on startup, bad approach Revert "WIP: Replay missing transfers on startup, bad approach" This reverts commit ea3025a. Test fixes Implement recovery after import when a node is down
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.