Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

block scanning should use block acceleration features of the TCT #1576

Closed
Tracked by #1376
TalDerei opened this issue Jul 28, 2024 · 3 comments
Closed
Tracked by #1376

block scanning should use block acceleration features of the TCT #1576

TalDerei opened this issue Jul 28, 2024 · 3 comments
Assignees
Labels
performance Related to speed improvements refactor Improving existing system with new design sync V2 features required after mainnet

Comments

@TalDerei
Copy link
Contributor

TalDerei commented Jul 28, 2024

clients stream compact blocks and construct a filtered instance of the SCT. Currently, we're trial-decrypting every encrypted payload for every block retrieved from the full node. However, our wasm view server implementation doesn't yet take advantage of the TCT acceleration features detailed in Penumbra issue #842. In that context, we can use the fact that penumbra commits the intermediate root hash of every block, and client synchronization can stream the root hashes of each block without needing to know the individual commitments. This approach would reduce the number of Merkle hashing operations needed to insert a block into the SCT.

our scan_block implementation was built using mock_client as a template. Instead, we should employ the TCT's API, specifically the insert_block and insert_epoch methods, to accelerate block syncing.

we've essentially built this highly optimized merkle tree (Tierd Commitment Tree) designed for efficient client sync, but we're not taking advantage of it's fundamental properties.

@TalDerei TalDerei added refactor Improving existing system with new design performance Related to speed improvements sync labels Jul 28, 2024
@TalDerei TalDerei self-assigned this Aug 15, 2024
@hdevalence
Copy link
Member

hdevalence commented Aug 19, 2024

the relevant code in the rust view server is here: https://github.com/penumbra-zone/penumbra/blob/main/crates/view/src/sync.rs#L111-L117

the optimization is: if the result of scanning the block is that there were no decryptions, use insert_block.

there's no need to use the epoch-level insertion features at this time as that is more complicated to employ (scanning goes block by block, so doing epoch-level insertion would require buffering an entire epoch, which is also the reason it's not used in the native view server implementation).

@hdevalence
Copy link
Member

hdevalence commented Aug 19, 2024

this optimization isn't completely trivial to port to the wasm view server, because it inlines together the trial decryption and the SCT insertions into one sequence of operations:

https://github.com/penumbra-zone/web/blob/main/packages/wasm/crate/src/view_server.rs#L123-L130

instead, these operations can be cleanly separated by using in-memory advice, as in the native code: https://github.com/penumbra-zone/penumbra/blob/main/crates/view/src/sync.rs#L86-L109 : first, build a lookup table of successful decryptions (the "advice"), then use it when processing each note payload. that way, it's easy to write code that skips over an empty block, as all the trial decryption is done up-front.

the native code's behavior of always asking the storage for advice is not necessary and probably undesirable to replicate, since it means waiting for an extra read regardless of whether there are any rolled-up payloads in the compact block. most blocks don't have these, so the wasm code's current behavior of only fetching from storage when there is known to be something to fetch seems good to preserve.

@hdevalence
Copy link
Member

Added some code that ports the native rust code's advice mechanism to the wasm crate in #1716 but it needs testing to check that it doesn't mess up the SCT synchronization

@TalDerei TalDerei added the V2 features required after mainnet label Sep 4, 2024
@TalDerei TalDerei closed this as completed Sep 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
performance Related to speed improvements refactor Improving existing system with new design sync V2 features required after mainnet
Projects
Archived in project
Development

No branches or pull requests

2 participants