feat: implement ReplacingMergeTree migration for batches and batch_blocks #1102
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ocks Implements automatic deduplication for batches and batch_blocks tables using ReplacingMergeTree engine to address duplicate data issues from gap detection and backfill operations. Key changes: - Migration tracking system with schema_migrations table (migration 019) - ReplacingMergeTree shadow tables with optimized indexing (migration 020) - ORDER BY (batch_id) for optimal deduplication - Secondary indexes on inserted_at and l1_block_number for query performance - Data migration with argMax() deduplication (migration 021, MANUAL) - Atomic table swap with rollback plan (migration 022, MANUAL) - Automated cleanup of old tables (migration 023) Performance optimizations: - Primary key (batch_id) optimizes most common JOIN patterns - Secondary index on inserted_at maintains monitoring query performance - Secondary index on l1_block_number maintains L1-based query performance - Preserved projections for l2_block_number lookups Migration strategy: - Safe startup migrations (019, 020, 023) run automatically - Data-intensive operations (021, 022) require manual execution - MANUAL suffix prevents automatic execution during startup - Comprehensive documentation and rollback procedures included 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.