[storage/archive] Add Lite Index + Remove Journal Padding #2707
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Adds a benchmark for archive get operations with 64KB values to better represent realistic block-sized workloads. The existing 32-byte value benchmark heavily favors buffer pool caching (~500 values per page), which is not representative of actual blockchain storage patterns. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
Adds a benchmark that uses a tiny buffer pool (10 pages = 160KB) to simulate realistic cache contention scenarios. This validates that the split storage design performs better under cache pressure. Results show spike is 3-4x faster than main for serial reads when the cache is too small to hold the working set, confirming the hypothesis that bypassing the buffer pool is beneficial when cache hit rates are low. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
commonware-mcp | 5de9e25 | Jan 08 2026, 01:29 AM |
patrick-ogrady
commented
Jan 6, 2026
patrick-ogrady
commented
Jan 6, 2026
Extracts common blob management logic from fixed.rs and variable.rs into a reusable BlobManager component. This eliminates ~340 lines of duplicate code for init, sync, prune, destroy, and rewind operations. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
Deploying monorepo with
|
| Latest commit: |
5de9e25
|
| Status: | ⚡️ Build in progress... |
…journal replay clone_blob() returns a Future that needs to be awaited, while clone() directly clones the Append wrapper which is what we need for the stream. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
patrick-ogrady
commented
Jan 6, 2026
patrick-ogrady
commented
Jan 6, 2026
patrick-ogrady
commented
Jan 6, 2026
patrick-ogrady
commented
Jan 6, 2026
patrick-ogrady
commented
Jan 6, 2026
patrick-ogrady
commented
Jan 6, 2026
patrick-ogrady
commented
Jan 6, 2026
patrick-ogrady
commented
Jan 6, 2026
patrick-ogrady
commented
Jan 6, 2026
patrick-ogrady
commented
Jan 6, 2026
patrick-ogrady
commented
Jan 6, 2026
… module - Rename blob_manager.rs to manager.rs and BlobManager to Manager - Add BufferFactory trait with AppendFactory (pool caching) and WriteFactory (no caching) - Make Manager generic over BufferFactory to support both buffer types - Move archive/blob.rs to journal/segmented/glob.rs using Manager with WriteFactory - Rename glob::Blob to glob::Glob - Update freezer and prunable archive to use new glob module - Each journal (fixed, variable, glob) now has its own Config struct - Remove unused section verification code during replay - Update documentation references 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
Introduce a new `Oversized` module that combines `FixedJournal` + `Glob` with automatic crash recovery. This ensures index entries always point to valid glob data after an unclean shutdown. Key changes: - Add `OversizedEntry` trait for entries that reference oversized values - Add `Oversized` struct with crash recovery validation during init - Migrate `archive/prunable` to use `Oversized` instead of separate journals - Migrate `freezer` to use `Oversized` instead of separate journals - Add `rewind_section` method to `FixedJournal` for crash recovery - Add `key_index_replay_buffer` config field to freezer The crash recovery algorithm validates each index entry's value location against the actual glob size, rewinding the index journal to exclude any trailing invalid entries that reference non-existent glob data. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
…O(entries) The previous recovery algorithm loaded ALL index entries into memory to validate them against glob sizes. This was O(total entries) in both memory and time. The new algorithm only checks the last entry in each section. Since entries are appended sequentially and value offsets are monotonically increasing within a section, if the last entry is valid then all earlier entries must be valid too. Changes: - Rewrite Oversized::recover() to only check last entry per section - Add sections() method to fixed::Journal - Remove replay_buffer from Oversized::Config (no longer needed) - Remove key_index_replay_buffer from Freezer::Config - Add 9 comprehensive recovery tests covering edge cases 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
- Expand Glob, Oversized, and related types to use u64 for value offset and size instead of u32, removing the ~4GB per-section limit - Add conformance hash tests to fixed.rs and glob.rs to detect accidental format changes - Fix Oversized::prune() to return bool for API consistency with Manager::prune() Breaking change: on-disk format changed for KeyEntry and IndexEntry. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
Add 16-byte alignment to glob storage, allowing u32 offsets to address ~64GB per section while saving 8 bytes per entry compared to u64. Changes: - Add shared `align` module with alignment utilities - Update glob.rs to write entries at aligned positions - Revert OversizedEntry trait to (u32, u32) for offset/size - Revert Cursor from (u64, u64, u64) to (u64, u32, u32) - Revert KeyEntry and IndexEntry to use u32 for value location 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
- Change prune() from parallel to sequential (index first, glob second) to ensure crash safety. If crash occurs after index prune but before glob prune, orphan data in glob is acceptable and cleaned up on next prune. - Add explicit warning log when recover() detects glob has pruned a section that index still has (indicates crash during prune). - Add three new recovery tests: - test_recovery_glob_pruned_but_index_not - test_recovery_index_partition_deleted - test_recovery_index_synced_but_glob_not 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
patrick-ogrady
commented
Jan 7, 2026
Contributor
Author
patrick-ogrady
left a comment
patrick-ogrady
left a comment
There was a problem hiding this comment.
almost there!
danlaine
reviewed
Jan 7, 2026
danlaine
reviewed
Jan 7, 2026
danlaine
reviewed
Jan 7, 2026
danlaine
reviewed
Jan 7, 2026
danlaine
previously approved these changes
Jan 7, 2026
Codecov Report❌ Patch coverage is @@ Coverage Diff @@
## main #2707 +/- ##
==========================================
+ Coverage 92.91% 93.14% +0.22%
==========================================
Files 363 368 +5
Lines 108104 110795 +2691
==========================================
+ Hits 100450 103196 +2746
+ Misses 7654 7599 -55
... and 12 files with indirect coverage changes Continue to review full report in Codecov by Sentry.
🚀 New features to boost your workflow:
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary (from Claude)
Redesigns archive and freezer storage to split data across two backends:
Key Changes
Managerabstraction to deduplicate blob lifecycle management across segmented journalsBufferFactorytrait withAppendFactory(uses buffer pool) andWriteFactory(direct I/O)Globjournal for sequential value storage without buffer pool overheadOversizedabstraction combining fixed index journal + glob with automatic crash recoveryarchive/prunable,archive/immutable, andfreezerto new split architectureappendmethods to pre-allocate buffers and avoid memory copiesCrash Recovery
The
Oversizedmodule handles unclean shutdown by validating index entries against actual glob sizes:Config Changes
Archive and freezer configs now have separate partitions and write buffers for key/index and value storage:
key_partition/value_partition(was singlepartition)key_write_buffer/value_write_buffer(was singlewrite_buffer)key_buffer_poolfor index journal caching