feat: implement parallel execution #33
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
feat: add parallel transaction execution infrastructure Implement execute_best_transactions_parallel with mutex-protected shared state for validating parallel execution structure: - Add ParallelExecutionState with mutex-protected db, info, and best_txs - Add ParallelExecutionMetrics with atomic counters for lock-free metrics - Build receipts and record transactions inside thread for cancellation safety - Call mark_invalid directly from threads via mutex-protected best_txs - Remove post-processing results loop - all work happens inline Currently uses NUM_THREADS=1 and mutex serialization for validation. Structure is in place for future Block-STM style optimistic concurrency. feat: Implement Block-STM parallel transaction execution This commit introduces Block-STM style parallel execution for transaction processing, enabling true parallelism in payload building. Key components: - block_stm/types.rs: Core types (TxnIndex, Incarnation, Version, etc.) - block_stm/mv_hashmap.rs: Multi-version data structure for versioned state - block_stm/captured_reads.rs: Read set tracking for conflict detection - block_stm/scheduler.rs: Task dispatch, abort management, in-order commits - block_stm/db_adapter.rs: VersionedDatabase implementing revm::Database - block_stm/view.rs: WriteSet and EVM state key/value types Execution flow: 1. Worker threads execute transactions in parallel using VersionedDatabase 2. Reads route through MVHashMap, fall back to base state 3. Writes tracked for conflict detection and validation 4. Scheduler handles re-execution on conflicts 5. Sequential commit phase builds receipts and applies state The implementation uses shared read-only access to the base database during parallel execution, with sequential commits after all workers complete. Revert change docs: Add Block-STM architecture documentation Comprehensive documentation covering: - Architecture overview with ASCII diagram - Component descriptions (MVHashMap, Scheduler, VersionedDatabase, etc.) - Execution flow walkthrough - Conflict detection explanation - EVM state mapping - Performance considerations - Future improvements update
- Add --builder.parallel-threads CLI flag (env: BUILDER_PARALLEL_THREADS) - Default to std::thread::available_parallelism() (number of CPU cores) - Falls back to 4 threads if parallelism detection fails - Thread configuration flows through BuilderConfig -> OpPayloadBuilderCtx - Update Block-STM README to document the configuration option
When parallel_threads == 1: - Uses sequential execution via execute_best_transactions - Restores CachedReads optimization for better repeated-read performance - Wraps database with cached_reads.as_db_mut() for state caching When parallel_threads > 1: - Uses Block-STM parallel execution via execute_best_transactions_parallel - Requires DatabaseRef bound for concurrent reads - Does not use CachedReads (incompatible with parallel access) Changes: - Add execute_sequential and build_sequential methods to OpBuilder - Update build_payload to branch based on parallel_threads - Keep flashblocks always using parallel execution (designed for incremental building) - Update README to document the behavior
BlockHash removal: - Block hashes are immutable within a block (like code by hash) - No need to track them as dependencies for conflict detection - Simplify block_hash_ref to read directly from base_db Race condition fix in scheduler: - Use compare_exchange for atomic commit slot claiming - Prevents double-counting of total_commits - Only the thread that wins the CAS updates status and stats
Implements commutative balance delta support in Block-STM for parallel fee accumulation (e.g., to coinbase). This avoids read-write conflicts when multiple transactions pay fees to the same address. Key changes: - Add BalanceDelta, VersionedDelta, ResolvedBalance types - Add balance_deltas storage in MVHashMap with: - write_balance_delta(): Store delta without conflict - resolve_balance(): Sum deltas when balance is read - has_pending_deltas(): Check for pending increments - Update WriteSet with add_balance_delta() method - Add CapturedResolvedBalance to track delta resolution reads - Update db_adapter to resolve balances with pending deltas - Update scheduler to: - Apply balance deltas separately from regular writes - Validate resolved balance reads with contributor tracking - Integrate with context.rs: - Extract pending_balance_increments as deltas - Use LazyEvmState.resolve_full_state() for commit Design: - Delta writes do NOT conflict with each other (commutative) - Delta reads (resolution) create dependencies on all contributors - If any contributor re-executes, reader is invalidated Tests: - 16 new tests covering: - Parallel accumulation without conflicts - Resolution correctness - Contributor re-execution invalidation - Mixed read/delta scenarios - Stress testing with multiple threads - Multiple addresses - Aborted contributor handling
meyer9
changed the title
meyer9
force-pushed
the
meyer9/parallel-execution-revm-main
branch
from
390292e to
bc9912e
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
📝 Summary
💡 Motivation and Context
✅ I have completed the following steps:
make lintmake test