feature : rocksdb phase4 consolidated - #8208
Draft
LegendPei wants to merge 35 commits into
Draft
Conversation
feature : rocksdb phase1 config and spi
Feat/rocksdb phase2 engine
Feat/rocksdb phase3 query index
Add production configuration controls, diagnostics, maintenance foundations, snapshot support, and benchmark coverage.
Add configurable WAL synchronization, shutdown durability handling, recovery safeguards, and focused tests.
Add fair bounded status scans, storage tuning controls, verification modes, and workload comparison coverage.
Optimize high fan-out transaction cleanup and add resumable orphan cleanup, crash recovery, and interference coverage.
Add lock-index verification and repair planning, resumable progress, safety gates, and maintenance tests.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## 2.x #8208 +/- ##
============================================
- Coverage 73.25% 73.12% -0.13%
- Complexity 1146 1148 +2
============================================
Files 1153 1153
Lines 42348 42507 +159
Branches 5061 5063 +2
============================================
+ Hits 31022 31085 +63
- Misses 8845 8942 +97
+ Partials 2481 2480 -1
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ⅰ. Describe what this PR did
This PR continues to optimize RocksDB FileMode in #8149, focusing on handling the upper bound of background workload, Seata lifecycle performance, WAL/recovery semantics, benchmark evidence, and protected maintenance processes.
The main content is as follows:
noneandperiodicWAL sync strategies, and supplement with clean-shutdown marker, strict sync before shutdown, exception-safe resource release, scheduling shutdown race handling, and process-crash recovery harness.maxTotalWalSize, compaction, and WAL-related parameters, but did not upgrade profiles that failed the benchmark gate to their default values.GLOBAL_REMOVE, branch/lock cleanup, and lock release within the lifecycle boundary of xid. Use RangeDelete within a provably safe and ordered index range, and continue to use a singleWriteBatchto maintain atomicity of state and index updates.Main benchmark results:
Machine parameters:
-Xms2g -Xmx4g;a small number of early matrices use-Xms1g -Xmx3gglobal_remove_with_branches,1M×2×2,triplicated+53.4%global_remove,1M×2×2,triplicated+56.8%,p99-34.0%lock.release_branch,1M×2×2,triplicated+41.0%,p99-26.3%lock.release_global,1M×2×2,triplicated+45.7%,p99-28.0%GLOBAL_REMOVE,50 branch×10 lock+30.7%,p99-26.6%The performance improvement mentioned here is derived from repeated tests conducted on the same machine, with the same scale, workload, and independent JVM. Since bounded scan and full scan yield different return sizes, it is only used to demonstrate the upper bound of a single workload, without calculating the multiple improvement.
Ⅱ. Does this pull request fix one issue?
fixes#8114
Ⅲ. Why don't you add test cases (unit test/integration test)?
Test coverage:
WriteBatch.Ⅳ. Describe how to verify it
Ⅴ. Special notes for reviews
This PR is a follow-up optimization of #8149 and depends on #8149