Improve performance of delta generation for packs #1720
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Implements the TODO for multi-threading in pack deltification with three separate implementations and automatic thread count optimization. - Support three deltification strategies: no-deltas, single-threaded, multi-threaded - Adaptive thread count based on object count (1-8 threads) - Configurable threading parameters (num_threads, batch_size)
Implement a high-performance Rust version of the create_delta function that provides significant performance improvements for pack deltification.
Implement support for additional Git pack configuration options: - pack.allowPackReuse: Controls whether existing pack data can be reused when generating new packs (defaults to true)
- Implement deltify_pack_objects in Rust with single-threaded approach - Fix create_delta to use minimum match length of 3 bytes - Use Python's UnpackedObject class for compatibility - Enable Rust implementation as default for deltify_pack_objects - All pack tests now pass
Implement parallel delta compression using Rayon with intelligent thresholding and selective parallelization for optimal performance. Hide behind a (default enabled) "parallel" feature flag.
- Add C Git's sorting heuristics with name hashing for better delta grouping - Implement memory-aware delta window management (256MB limit) - Add delta chain depth limiting (max depth 50, matching Git's default) These optimizations improve pack performance by: 1. Grouping similar files together for better delta compression 2. Dynamically adjusting window size based on object sizes and memory 3. Preventing excessively deep delta chains that hurt access performance
jelmer
force-pushed
the
create-pack-delta-perf
branch
2 times, most recently
from
2763277 to
1abb07f
Compare
Added memory pool for buffer reuse during delta creation: - MemoryPool struct with buffer management - Buffer reuse for delta operations to reduce allocation overhead - Capacity limits to prevent memory bloat (64KB max delta, 32KB temp)
jelmer
force-pushed
the
create-pack-delta-perf
branch
from
1abb07f to
fbb7da2
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.