Proposal: Support Local Replay (LRP) for Mainnet Batches

[KyrinCode]

Background:

X Layer requires Local Replay (LRP) functionality to replay transaction history from a specific range of mainnet batches locally. The primary goals of this feature are:

1. Performance Optimization: Provide a realistic testing environment for targeted performance improvements.
2. Version Comparison: Benchmark execution efficiency between old and new sequencer versions by replaying the same historical blocks.
3. Release Validation: Ensure the new version does not fork historical transactions by replaying all historical mainnet blocks before major releases.

Goal:

Partial replay: Enable local replay of mainnet transactions for a specified range of batches, defined as [fromBatchNumber, toBatchNumber].

Full replay: Support replaying long batch history until the latest batchNo. Since sequential full replay is undoubtably time consuming, we need to separate the tasks into segments and run in parallel.

Research on Potential Approaches (Resequence-Based Approach Selected)

Basic Idea of reusing resequence logic:

At first, we have original mainnet data, and the state of chaindata and datastream is:

• chaindata: currentBatchNo
• datastream: currentBatchNo

After unwinding to fromBatchNo-1:

• chaindata: fromBatchNo-1
• datastream: currentBatchNo

Resequence to toBatchNo

• First read [fromBatchNo, toBatchNo] from datastream to memory and truncate these from datastream

  • chaindata: fromBatchNo-1
  • datastream: fromBatchNo-1

• Then replay batches [fromBatchNo, toBatchNo] until

  • chaindata: toBatchNo
  • datastream: toBatchNo

Example of Current Usage of Mainnet LRP in X Layer

make lrp-config  # configure LRP parameters
make lrp-mainnet # deploy

Current version supports:

• range replay
• faster read batches from datastream
• real-time mismatch detection
• computing instant TPS
• resume replay from the breakpoint
• chaindata compaction
• user separation
• vmtouch
• multi-process replay

Check more to dive into okx#277

Welcome to any feedback. Any more insights or recommendations would be appreciated.

[revitteth]

Thanks, we're checking up on this!