Skip to content

feature : rocksdb phase4 consolidated - #8208

Draft
LegendPei wants to merge 35 commits into
apache:2.xfrom
LegendPei:feat/rocksdb-phase4-consolidated
Draft

feature : rocksdb phase4 consolidated#8208
LegendPei wants to merge 35 commits into
apache:2.xfrom
LegendPei:feat/rocksdb-phase4-consolidated

Conversation

@LegendPei

Copy link
Copy Markdown
Contributor

Ⅰ. Describe what this PR did

This PR continues to optimize RocksDB FileMode in #8149, focusing on handling the upper bound of background workload, Seata lifecycle performance, WAL/recovery semantics, benchmark evidence, and protected maintenance processes.

The main content is as follows:

  1. Extend the existing diagnostics, checkpoint, and benchmark harnesses in feature : support rocksdb in filemode #8149. Add independent JVM repeat, A/B sequence control, real payload, scan/read/delete counts, p50/p95/p99, SST/WAL/compaction, and foreground interference models to the benchmark.
  2. Introduce none and periodic WAL sync strategies, and supplement with clean-shutdown marker, strict sync before shutdown, exception-safe resource release, scheduling shutdown race handling, and process-crash recovery harness.
  3. Change status/begin/full scan and Coordinator retry/end-state queries to protected scans with limit, deadline, cursor, and scan stats; use k-way merge with resumable scanning for multi-state queries to avoid long-term occupation of the first page by failed sessions.
  4. Change the full cleanup of orphan locks during the startup phase to a background task with persistent cursors, batch size limits, inter-round rate limits, and interruption recovery, while retaining the conservative cleanup semantics after a crash.
  5. Exposed block cache, DB/CF write buffer, maxTotalWalSize, compaction, and WAL-related parameters, but did not upgrade profiles that failed the benchmark gate to their default values.
  6. Optimize GLOBAL_REMOVE, branch/lock cleanup, and lock release within the lifecycle boundary of xid. Use RangeDelete within a provably safe and ordered index range, and continue to use a single WriteBatch to maintain atomicity of state and index updates.
  7. Add PAGE/FULL verify, repair plan, dry-run, quota execution, checkpoint pre-protection, progress persistence, and fault recovery to the feature : support rocksdb in filemode #8149 basic verify and snapshot interfaces. High-cost maintenance actions will not enter the normal transaction hot path.

Main benchmark results:

Machine parameters:

Item Value
OS Ubuntu 24.04.2 LTS,Linux 6.8
CPU QEMU/KVM,12 vCPU
Memory 15GiB
Disk Virtual mechanical disk/shared storage
JDK OpenJDK 21.0.11
Maven 3.9.6/3.9.x
RocksDB JNI 10.10.1.1
Common JVM -Xms2g -Xmx4g;a small number of early matrices use -Xms1g -Xmx3g
Scenario Baseline Phase 4 Candidate Result
cleanup global_remove_with_branches,1M×2×2,triplicated 16,455.966 ops/s 25,245.184 ops/s 吞吐 +53.4%
write global_remove,1M×2×2,triplicated 14,929.260 ops/s,p99 0.159ms 23,407.043 ops/s,p99 0.105ms Throughput +56.8%,p99 -34.0%
lock.release_branch,1M×2×2,triplicated 13,047.133 ops/s,p99 0.152ms 18,402.559 ops/s,p99 0.112ms Throughput +41.0%,p99 -26.3%
lock.release_global,1M×2×2,triplicated 13,640.268 ops/s,p99 0.143ms 19,869.311 ops/s,p99 0.103ms Throughput +45.7%,p99 -28.0%
status bounded scan,100K→200K,total overdue fixed at 1,000 1,001 rows scanned 1,001 rows scanned Single scan volume remains bounded when total data doubles
high fan-out GLOBAL_REMOVE,50 branch×10 lock 318.951 ops/s,p99 4.951ms 416.840 ops/s,p99 3.636ms Throughput+30.7%,p99 -26.6%

The performance improvement mentioned here is derived from repeated tests conducted on the same machine, with the same scale, workload, and independent JVM. Since bounded scan and full scan yield different return sizes, it is only used to demonstrate the upper bound of a single workload, without calculating the multiple improvement.

Ⅱ. Does this pull request fix one issue?

fixes#8114

Ⅲ. Why don't you add test cases (unit test/integration test)?

Test coverage:

  • RocksDB parameter parsing, illegal value verification, DB/CF options application, and diagnostic snapshot.
  • WAL periodic sync, shutdown strict sync, clean marker, sync failure, executor close race, and crash recovery harness.
  • status/begin/multi-status cursor, deadline, bounded scan stats, Coordinator round-robin fairness, and stale index.
  • Equivalent semantics of RangeDelete and scan-delete, duplicate deletion, restart recovery, branch/lock fan-out, and atomic WriteBatch.
  • Persistence of cursor, rate limiting, restart continuation, exception recovery, and foreground interference for orphan lock background cleanup.
  • checkpoint, PAGE/FULL verify, repair dry-run, guard, fault injection, progress recovery, and Raft snapshot.
  • Benchmark parameter parsing, CSV header, repeat summary, actual rows/pointReads, fixed expiration time, and probe startup synchronization.

Ⅳ. Describe how to verify it

Ⅴ. Special notes for reviews

This PR is a follow-up optimization of #8149 and depends on #8149

LegendPei added 30 commits June 3, 2026 17:03
Add production configuration controls, diagnostics, maintenance foundations, snapshot support, and benchmark coverage.
Add configurable WAL synchronization, shutdown durability handling, recovery safeguards, and focused tests.
Add fair bounded status scans, storage tuning controls, verification modes, and workload comparison coverage.
Optimize high fan-out transaction cleanup and add resumable orphan cleanup, crash recovery, and interference coverage.
Add lock-index verification and repair planning, resumable progress, safety gates, and maintenance tests.
@codecov

codecov Bot commented Aug 22, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 33.33333% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 73.12%. Comparing base (dbea185) to head (898e8cf).

Files with missing lines Patch % Lines
...erver/console/entity/param/GlobalSessionParam.java 0.00% 4 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##                2.x    #8208      +/-   ##
============================================
- Coverage     73.25%   73.12%   -0.13%     
- Complexity     1146     1148       +2     
============================================
  Files          1153     1153              
  Lines         42348    42507     +159     
  Branches       5061     5063       +2     
============================================
+ Hits          31022    31085      +63     
- Misses         8845     8942      +97     
+ Partials       2481     2480       -1     
Files with missing lines Coverage Δ
...ava/org/apache/seata/common/ConfigurationKeys.java 0.00% <ø> (ø)
...in/java/org/apache/seata/common/DefaultValues.java 100.00% <ø> (ø)
...n/java/org/apache/seata/common/store/LockMode.java 94.11% <100.00%> (+0.36%) ⬆️
...ta/spring/boot/autoconfigure/StarterConstants.java 100.00% <ø> (ø)
...ot/autoconfigure/loader/SeataPropertiesLoader.java 54.54% <ø> (-8.62%) ⬇️
...configure/SeataServerEnvironmentPostProcessor.java 100.00% <ø> (ø)
...e/properties/server/store/StoreFileProperties.java 48.55% <ø> (-51.45%) ⬇️
...erver/console/entity/param/GlobalSessionParam.java 0.00% <0.00%> (ø)

... and 3 files with indirect coverage changes

Impacted file tree graph

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant