Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PoS sync #8429

Draft
wants to merge 98 commits into
base: main
Choose a base branch
from
Draft
Changes from 1 commit
Commits
Show all changes
98 commits
Select commit Hold shift + click to select a range
f1e1113
first try
pinges Dec 19, 2024
9abc4dd
fix NPE
pinges Dec 19, 2024
b0d15fc
no logsWithMetadata available during sync with SyncBlock. Fixes NPE
pinges Dec 20, 2024
f2acc7e
add logging when importing
pinges Dec 20, 2024
da9cff8
merge main
pinges Jan 21, 2025
c133616
added transaction indexing and log meta data
pinges Jan 22, 2025
cece7a4
make last steps in pipeline not ordered and convert ImportStep
pinges Jan 23, 2025
15aa667
increase import step parallelism
pinges Jan 23, 2025
ec796ef
make all ordered
pinges Jan 23, 2025
faa1b1b
make import shorter
pinges Jan 24, 2025
87b5eb4
Merge branch 'main' of github.com:hyperledger/besu into SyncBlockBody
pinges Jan 28, 2025
1e50b17
replace Body with SyncBody in Fastsync
pinges Jan 28, 2025
cd2beef
Merge branch 'main' of github.com:hyperledger/besu into SyncBlockBody
pinges Jan 31, 2025
4a0ccd5
Merge branch 'main' of github.com:hyperledger/besu into SyncBlockBody
pinges Feb 2, 2025
baf668f
add some tests and some minor changes
pinges Feb 4, 2025
709ff3a
Merge branch 'main' of github.com:hyperledger/besu into SyncBlockBody
pinges Feb 4, 2025
1b93d01
Merge branch 'main' of github.com:hyperledger/besu into SyncBlockBody
pinges Feb 5, 2025
6521fb2
Merge branch 'main' of github.com:hyperledger/besu into SyncBlockBody
pinges Feb 6, 2025
a89f402
Merge branch 'main' of github.com:hyperledger/besu into SyncBlockBody
pinges Feb 6, 2025
b2be1ec
Merge branch 'main' of github.com:hyperledger/besu into SyncBlockBody
pinges Feb 7, 2025
4cdb549
clean up after review
pinges Feb 10, 2025
995f74a
reduce header request size to 120
pinges Feb 11, 2025
f682d48
Merge branch 'main' of github.com:hyperledger/besu into SyncBlockBody
pinges Feb 11, 2025
c61feb6
fix AGBFPT
pinges Feb 11, 2025
b347e1e
undo change of header request size
pinges Feb 11, 2025
b5ed59d
Merge branch 'main' of github.com:hyperledger/besu into SyncBlockBody
pinges Feb 11, 2025
5a34ec8
minor changes for readability
pinges Feb 13, 2025
b4e7898
merge conflicts
pinges Feb 13, 2025
7bdc04b
Merge branch 'main' of github.com:hyperledger/besu into SyncBlockBody
pinges Feb 13, 2025
424d27e
add usage of legacy bodies download
pinges Feb 18, 2025
5769c0a
Merge branch 'main' of github.com:hyperledger/besu into SyncBlockBody
pinges Feb 18, 2025
1840bb3
add some logging
pinges Feb 19, 2025
e3e1c2f
Merge branch 'main' of github.com:hyperledger/besu into SyncBlockBody
pinges Feb 19, 2025
9f1b6d3
add logging
pinges Feb 21, 2025
403038f
fix bodies download
pinges Feb 24, 2025
2702138
Merge branch 'main' of github.com:hyperledger/besu into SyncBlockBody
pinges Feb 24, 2025
044b052
undo change in SyncBlockBody
pinges Feb 25, 2025
a8ddebe
add useful comment
pinges Feb 25, 2025
0a75004
Merge branch 'main' of github.com:hyperledger/besu into SyncBlockBody
pinges Feb 25, 2025
56fbc62
clean up
pinges Feb 26, 2025
d9a6207
remove log at info
pinges Feb 28, 2025
f6d6496
Merge branch 'main' of github.com:hyperledger/besu into SyncBlockBody
pinges Feb 28, 2025
dbd8206
Merge branch 'main' of github.com:hyperledger/besu into SyncBlockBody
pinges Mar 17, 2025
3c6d87a
add importBlockForSyncing method back in
pinges Mar 17, 2025
efa6a17
pos sync mode
jframe Mar 17, 2025
8c9230d
body download done, receipts and finish to go
pinges Mar 17, 2025
37a06e9
sync headers first
jframe Mar 17, 2025
3c54cee
improve header logging
jframe Mar 17, 2025
8c7647e
improve header logging - reset accumulated time
jframe Mar 17, 2025
145f43b
use header request size for header syncSource
jframe Mar 17, 2025
2a4e08b
changing block to header in logging
jframe Mar 17, 2025
8a06908
use throttled logger in ImportHeadersStep
jframe Mar 17, 2025
6f66b02
log formatting
jframe Mar 17, 2025
63c5dbb
Change worker subtasks to use service executor to avoid executor queues
jframe Mar 18, 2025
c0289ba
add receipts step
pinges Mar 18, 2025
0aeeb25
merge
pinges Mar 18, 2025
835e56b
Use header parallelism size for header loading
jframe Mar 18, 2025
de1a987
Don't validate headers during download
jframe Mar 18, 2025
846bc70
add finishing step
pinges Mar 18, 2025
7fbede2
Merge branch 'PoSSync-Hackathon' of github.com:pinges/besu into PoSSy…
pinges Mar 18, 2025
ac42769
Temporarily stop downloading headers at 22,000,000
jframe Mar 19, 2025
cafd860
Add header target cmd line option to override syncing headers to gene…
jframe Mar 19, 2025
3583408
validate header ranges during header download
jframe Mar 19, 2025
04f49f8
fix some problems and add logging
pinges Mar 19, 2025
3f0ec65
fix empty headers
pinges Mar 19, 2025
6c5c9f5
fix rlp
pinges Mar 19, 2025
03593c0
fix compile
pinges Mar 19, 2025
91ffdee
Revert "validate header ranges during header download" since we downl…
jframe Mar 19, 2025
98e3397
fix empty headers
pinges Mar 19, 2025
5ec358e
Merge branch 'PoSSync-Hackathon' of github.com:pinges/besu into PoSSy…
pinges Mar 19, 2025
79d636c
fix rec download
pinges Mar 20, 2025
47f9362
add override
pinges Mar 20, 2025
af1d9e6
fallback to downloading headers if we don't have them in the db
jframe Mar 20, 2025
f8b3593
fix starting point
pinges Mar 20, 2025
541d832
Merge branch 'PoSSync-Hackathon' of github.com:pinges/besu into PoSSy…
pinges Mar 20, 2025
4523036
log header download percentage correctly
jframe Mar 20, 2025
1f2bda6
Merge branch 'PoSSync-Hackathon' of github.com:pinges/besu into PoSSy…
jframe Mar 20, 2025
8967385
fix last step
pinges Mar 20, 2025
1722527
Merge branch 'PoSSync-Hackathon' of github.com:pinges/besu into PoSSy…
pinges Mar 20, 2025
2c47ada
log header download percentage with 2 decimal places to be consistent…
jframe Mar 20, 2025
ad94eff
Merge branch 'PoSSync-Hackathon' of github.com:pinges/besu into PoSSy…
jframe Mar 20, 2025
a9def6b
parallelism config for bodies and receipts
jframe Mar 20, 2025
11deb21
parallelism config for bodies and receipts
jframe Mar 20, 2025
b68263b
parallelism config for bodies and receipts
jframe Mar 20, 2025
403ccd0
fix difficulty
pinges Mar 20, 2025
08a1a6a
merge
pinges Mar 20, 2025
f60913e
try to fix MAX
pinges Mar 20, 2025
98d33d8
make body download async
pinges Mar 20, 2025
bb4fd99
Skip header download if already completed
jframe Mar 20, 2025
8ef1b99
Merge branch 'PoSSync-Hackathon' of github.com:pinges/besu into PoSSy…
jframe Mar 20, 2025
8211a80
Skip header check to use child block when genesis block as we don't d…
jframe Mar 20, 2025
23a8e20
Upgrade rocksdb to 9.10 and increase maxWriteBufferNumber
jframe Mar 21, 2025
dbf8eed
Remove config for maxWriteBufferNumber
jframe Mar 21, 2025
3f33463
Revert "Upgrade rocksdb to 9.10 and increase maxWriteBufferNumber"
jframe Mar 21, 2025
2ed4aa0
RocksDB config to use maxWriteBufferNumber
jframe Mar 21, 2025
d5c4c69
do multiple txs to update blocks and receipts to avoid rocksdb error
jframe Mar 21, 2025
e066eef
fix off by one header download
pinges Mar 21, 2025
35ef36c
fix start problem of +1
pinges Mar 24, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix starting point
Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net>
pinges committed Mar 20, 2025
commit f8b359317b0287276e385327adbce9165aa15113
Original file line number Diff line number Diff line change
@@ -76,6 +76,7 @@ public GetSyncReceiptsFromPeerTask(

isPoS = protocolSchedule.anyMatch((ps) -> ps.spec().isPoS());
}

@Override
public int getRetriesWithOtherPeer() {
return retries;
Original file line number Diff line number Diff line change
@@ -51,11 +51,12 @@ public FinishPosSyncStep(
final ProtocolContext protocolContext,
final EthContext ethContext,
final BlockHeader pivotHeader,
final boolean transactionIndexingEnabled) {
final boolean transactionIndexingEnabled,
final long startingBlock) {
this.protocolContext = protocolContext;
this.blockchain = ethContext.getBlockchain();
final long chainHeadBlockNumber = blockchain.getChainHeadBlockNumber();
this.nextLowestBlockNumber = (int) (chainHeadBlockNumber + 1);
this.nextLowestBlockNumber = startingBlock;
if (chainHeadBlockNumber != 0) {
final Optional<BlockHeader> chainHeadBlockHeader =
blockchain.getBlockHeader(chainHeadBlockNumber);
@@ -67,6 +68,11 @@ public FinishPosSyncStep(

@Override
public void accept(final List<BlockHeader> blockHeaderRange) {
LOG.atInfo()
.setMessage("Next lowest block number: {}, lowest block number: {}")
.addArgument(nextLowestBlockNumber)
.addArgument(blockHeaderRange.getFirst().getNumber())
.log();
final BlockHeaderRange newRange =
new BlockHeaderRange(
blockHeaderRange.getFirst(),
Original file line number Diff line number Diff line change
@@ -146,10 +146,10 @@ public Pipeline<SyncTargetNumberRange> createDownloadPipelineForSyncTarget(
final int downloaderParallelism = syncConfig.getDownloaderParallelism();
final int headerRequestSize = syncConfig.getDownloaderHeaderRequestSize();

final long startingBlock = getCommonAncestor(target) + 1;
final PosSyncSource syncSource =
new PosSyncSource(
getCommonAncestor(target)
+ 1, // TODO remove the +1 when check in DefaultBlockChain is fixed
startingBlock, // TODO remove the +1 when check in DefaultBlockChain is fixed
() -> fastSyncState.getPivotBlockHeader().get().getNumber(),
headerRequestSize,
false);
@@ -166,7 +166,8 @@ public Pipeline<SyncTargetNumberRange> createDownloadPipelineForSyncTarget(
protocolContext,
ethContext,
fastSyncState.getPivotBlockHeader().get(),
syncConfig.getSnapSyncConfiguration().isSnapSyncTransactionIndexingEnabled());
syncConfig.getSnapSyncConfiguration().isSnapSyncTransactionIndexingEnabled(),
startingBlock);

return PipelineBuilder.createPipelineFrom(
"fetchCheckpoints",