Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf(tree): chunk multiproofs #14800

Merged
merged 49 commits into from
Mar 6, 2025
Merged

perf(tree): chunk multiproofs #14800

merged 49 commits into from
Mar 6, 2025

Conversation

shekhirin
Copy link
Collaborator

@shekhirin shekhirin commented Mar 3, 2025

Towards #14417

Problem

Given proof targets with 100 accounts, it's preferred to spawn 10 multiproofs with 10 accounts each, instead of spawning one multiproof for all 100 accounts at once. The reason for that is that 10 multiproofs can be executed in parallel (even considering that they may do redundant work because of the intersecting trie nodes), while one multiproof will be executed serially for all accounts.

Same idea can be applied to storage slots: proof targets with 1 account and 100 storage slots is better spawned as 10 multiproofs for 10 storage slots each, rather than one multiproof for all 100 storage slots at once.

Solution

This PR adds chunking on the MultiProofTask side.

  1. In on_prefetch_proof, proof targets are chunked and spawned as separate multiproofs.
  2. In on_state_update, state update is chunked, proof targets are created for each chunk, and spawned as separate multiproofs.

Benchmark

image

@shekhirin shekhirin added C-perf A change motivated by improving speed, memory usage or disk footprint A-engine Related to the engine implementation labels Mar 3, 2025
@shekhirin shekhirin force-pushed the alexey/chunk-proof-targets branch from 5cbcb6f to 8cd112a Compare March 3, 2025 14:29
@shekhirin shekhirin force-pushed the alexey/chunk-proof-targets branch from 8cd112a to 8f2ff1d Compare March 3, 2025 14:35
@shekhirin shekhirin force-pushed the alexey/chunk-proof-targets branch from f688e1e to 40eaed8 Compare March 3, 2025 14:56
@shekhirin shekhirin force-pushed the alexey/chunk-proof-targets branch 4 times, most recently from 94bfa41 to 3a09209 Compare March 3, 2025 15:34
@shekhirin shekhirin force-pushed the alexey/chunk-proof-targets branch from 3a09209 to e70357a Compare March 3, 2025 15:35
Copy link

codspeed-hq bot commented Mar 3, 2025

CodSpeed Performance Report

Merging #14800 will not alter performance

Comparing alexey/chunk-proof-targets (f9ecff0) with main (53e72c8)

Summary

✅ 77 untouched benchmarks

@shekhirin shekhirin marked this pull request as ready for review March 3, 2025 16:31
@shekhirin shekhirin requested a review from rkrasiuk as a code owner March 3, 2025 16:31
@shekhirin shekhirin force-pushed the alexey/chunk-proof-targets branch from 807dcd2 to a45b67a Compare March 5, 2025 20:12
@shekhirin shekhirin force-pushed the alexey/chunk-proof-targets branch from 93b989d to 1690584 Compare March 5, 2025 21:16
@shekhirin shekhirin force-pushed the alexey/chunk-proof-targets branch from c8ca55b to ed0829a Compare March 5, 2025 21:37
@shekhirin shekhirin requested a review from rkrasiuk March 6, 2025 10:47
@shekhirin shekhirin force-pushed the alexey/chunk-proof-targets branch from 56970f8 to 688c4b5 Compare March 6, 2025 11:41
@shekhirin shekhirin force-pushed the alexey/chunk-proof-targets branch from 688c4b5 to 37e6919 Compare March 6, 2025 11:46
Copy link
Collaborator

@mattsse mattsse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can follow the logic here, defer to @rkrasiuk re chunking

looks sound to me

@shekhirin shekhirin enabled auto-merge March 6, 2025 15:34
@shekhirin shekhirin added this pull request to the merge queue Mar 6, 2025
Merged via the queue into main with commit a765af9 Mar 6, 2025
42 checks passed
@shekhirin shekhirin deleted the alexey/chunk-proof-targets branch March 6, 2025 15:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-engine Related to the engine implementation C-perf A change motivated by improving speed, memory usage or disk footprint
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

3 participants