client: save preimages feature #3143

gabrocheleau · 2023-11-05T16:01:38Z

This PR will add a savePreimages option to the client, which will save a key-value map of keyHashes to their preimages (pre-hashed keys). This is a required preliminary step to eventually implement the merkle->verkle transition logic.

Could also be somewhat related to #614

codecov · 2023-11-05T16:04:14Z

Codecov Report

Attention: Patch coverage is 70.91837% with 57 lines in your changes are missing coverage. Please review.

Project coverage is 86.88%. Comparing base (b8f5b6d) to head (7a0842d).

Additional details and impacted files

Flag	Coverage Δ
block	`88.34% <ø> (ø)`
blockchain	`91.61% <ø> (ø)`
client	`84.77% <84.84%> (+0.04%)`	⬆️
common	`98.25% <100.00%> (+<0.01%)`	⬆️
devp2p	`82.12% <ø> (ø)`
ethash	`∅ <ø> (∅)`
evm	`74.25% <50.00%> (-0.09%)`	⬇️
genesis	`99.98% <ø> (ø)`
rlp	`∅ <ø> (∅)`
statemanager	`77.00% <75.00%> (-0.02%)`	⬇️
trie	`89.24% <ø> (ø)`
tx	`95.45% <ø> (ø)`
util	`89.13% <ø> (ø)`
vm	`79.62% <50.00%> (-0.58%)`	⬇️
wallet	`88.35% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

jochem-brouwer · 2023-11-05T19:40:47Z

I would actually add this flag into Trie. Then, before you hash the key, save it into another DB (as preimage). This would only require an extra option for the DB to write too, though. But adding it into Trie means that we capture all trie items which are ever put, and we can be 100% (?) sure that we capture all preimages.

gabrocheleau · 2023-11-06T07:31:02Z

I would actually add this flag into Trie. Then, before you hash the key, save it into another DB (as preimage). This would only require an extra option for the DB to write too, though. But adding it into Trie means that we capture all trie items which are ever put, and we can be 100% (?) sure that we capture all preimages.

So I haven't pushed the whole WIP, as I was still playing around with some lower-level (vm/statemanager/trie) handling. I don't think we can strictly have that at the trie level, as it needs to be at least one-level higher up to handle the transition logic from trie->verkle-trie. Also, after discussing with gajinder, we though that the metaDB from the client seemed like a good place to put this. Nothing set in stone though, happy to hear your thoughts/ideas

My current plan is to have the following structure:

the stateManager stores every preimage in memory. If the option is activated at the stateManager level, then anytime there is a putAccount operation, we add the preimage to the stateManager key/value map. I think that should cover all "touched accounts", right?
then, at the runBlock level, once all the txs from the block have been executed, we retrieve those stored preimages from the statemanager and dump them. We would return the array of hashedKeys->preimages in RunBlockResult
RunBlockResult can then be accessed by the client, and the preimages can be added to the metadb.

jochem-brouwer · 2023-11-06T10:59:26Z

the stateManager stores every preimage in memory. If the option is activated at the stateManager level, then anytime there is a putAccount operation, we add the preimage to the stateManager key/value map. I think that should cover all "touched accounts", right?

Yes, upon first thoughts I think this is right. What about the idea of Verkle where touched-but-deleted is different than in Merkle where touched-but-deleted does not change the trie? I think I recall this is dropped?

To test if all preimages are indeed stored in the StateManager I think we can add an extra test to the blockchain/state tests, what we do is we indeed copy all the MetaDB pre-image items, then after runBlock we copy the Trie and set the root to the root before the runBlock was ran. Then, we re-dump all these preimages in the copied trie and check if the root is the same reported root after runBlock.

…ations

holgerd77 · 2024-02-26T14:23:56Z

Updated this via UI

jochem-brouwer

Two final comments :) Thanks for the update!

jochem-brouwer · 2024-02-26T23:27:56Z

packages/client/test/rpc/engine/preimages.spec.ts

+        '0x4600000000000000000000000000000000000000',
+        '0x4700000000000000000000000000000000000000',
+      ],
+      // no txs


Are there no txs in this block? It seems that there are?

~~Should there also be a test where a contract is invoked and a storage slot is written to?~~ This is tested into client

@g11tech would know more about those cases than me, would you mind clarifying gajinder?

yes there are, we need to run the test with debug enabled for evm and add the list of things accessed here

I logged the accesses and added some tx preimages to the two blocks that have transactions.

packages/vm/src/types.ts

g11tech

lgtm

jochem-brouwer

LGTM

* trie: export "Path" interface * Move over Path interface export to types for consistency --------- Co-authored-by: Holger Drewes <[email protected]>

…ty range result (#3047) * Use no-elements proof for final check in account fetcher * Use no-elements proof for final check in storage fetcher * Check inputs after zero element proof code section * Cleanup * Reject peer if zero-element proof fails * Add tests for zero-element proof * Remove proofTrie usage * Use hashing function in static version of verifyRangeProof * Include useKeyHashing in call to fromProof so that hashing function is used in proof verification * Use appliedKey for hashing function instead of the function directly passed in TrieOpts * Pass in hashing function for use in static proof verification calls * Fix static range proof verification errors * Use custom hashing function from opts if available * Add test to check if zero element range proof verification fails with remaining elements to the right * Check if parameters are as expected for zero-element proof * Fix linting issues --------- Co-authored-by: Scotty <[email protected]> Co-authored-by: Holger Drewes <[email protected]> Co-authored-by: Indigo Alpha <[email protected]>

* Update to official trusted setup * Remove devnet6 * Add kzg-wasm * Update block tests to use kzg-wasm * Update tests * Add more 4844 tests to browser run * Initial integration of kzg-wasm on git * Update kzg-wasm build * Fix linter weirdness * Move initKzg to `runTests` * Fix tests * More cleanup * Goodbye c-kzg * fix kzg references * Replace c-kzg with kzg-wasm in package.json * Update kzg wasm commit and vm tester config * Update initKzg to createKZG * fix copy pasta * Fix more copy pasta * update kzg-wasm to npm release * One last bit of copy pasta * Address feedback * client: remove try/catch blocks createKZG() and remove the initKZG stale comment --------- Co-authored-by: Jochem Brouwer <[email protected]>

jochem-brouwer

LGTM!

Great work have removed my preimage -> preImage stuff :)

holgerd77 · 2024-03-01T10:45:42Z

Seems browser tests are not going through here?

…mjs-monorepo into client/save-preimages

holgerd77 · 2024-03-01T12:03:03Z

packages/evm/src/types.ts

    addAlwaysWarmAddress(address: string, addToAccessList?: boolean): void
    addAlwaysWarmSlot(address: string, slot: string, addToAccessList?: boolean): void
    startReportingAccessList(): void
+    startReportingPreimages(): void


Oh, just seeing on release notes writing: this needs to be optional for backwards compatibility.

* Add 4844-browser-readiness and KZG WASM setup instructions to tx CHANGELOG and README * Add all CHANGELOG files * Bump versions for block, blockchain, client and common, give internal verkle dependencies a caret range * Bump versions for devp2p, ethash, evm and genesis * Bump versions for statemanager, trie, tx and util * Bump versions for verkle, vm and wallet * Add changes from Preimage PR #3143 * Added monorepo-wide docs:build command to root package.json * Rebuild docs * Update README and example files * Rebuild package-lock.json * Add various continued work PRs to the CHANGELOG entries * Bump EVM to v3.0.0 and VM to v8.0.0 (in-between breaking releases due to new EVM async create() constructor) * Rebuild package-lock.json * Add additional EVM/VM breaking release CHANGELOG notes, slight simplification of EVM.create() constructor calling * Add additional EVM/VM breaking release README notes * Rebuild docs * Some TypeScript test fixes * More CHANGELOG additions * Update CHANGELOG files (small PRs) * Update KZG related CHANGELOG and README entries and example code * EVM: make main constructor protected

client: PreimageManager and Preimage storage in MetaDB

0906767

gabrocheleau added type: feature package: client labels Nov 5, 2023

gabrocheleau added package: vm package: evm labels Nov 6, 2023

acolytec3 and others added 21 commits November 7, 2023 10:56

Merge branch 'master' into client/save-preimages

1409a05

common: add getAppliedKey method to StateManagerInterface

5e76ea0

evm: add preimage reporting

c2938eb

statemanager: add getAppliedKey methods to the statemanager implement…

3e1984a

…ations

vm: improve some types

76a3876

vm: implement preimage reporting

63a9e61

client: add preimage saving

9f0bf8e

Merge branch 'master' into client/save-preimages

d6af5a9

evm: update to unprefixedHexToBytes

fe8022a

Merge branch 'master' into client/save-preimages

dbbd07f

vm: add reportPreimages test to runTx tests

50d85b8

vm: pass down reportPreimages arg from block to tx

6c32301

client: pass down reportPreimages from client to vm through runBlock

b338b99

client: unify config options naming

c9ed40a

client: preimageManager test

e0a4baf

client: test preimage, wip

8787ac5

common: add todo to gasPrices from 6800

157debe

merge with master

88c85af

statemanager: fix type issues

5b2cf25

dev accessWitness class on the lines of geth impl

3c63902

integrate accesswitness in evm/vm/runtx flow

7954bb6

gabrocheleau added 2 commits February 25, 2024 19:33

common: make getAppliedKey optional

6bbd14f

evm/vm: handle optional getAppliedKey

0e589de

gabrocheleau mentioned this pull request Feb 26, 2024

Breaking/Deprecation Tracking Issue #3216

Open

21 tasks

holgerd77 removed the PR state: needs changes label Feb 26, 2024

jochem-brouwer reviewed Feb 26, 2024

View reviewed changes

g11tech previously approved these changes Mar 1, 2024

View reviewed changes

jochem-brouwer previously approved these changes Mar 1, 2024

View reviewed changes

jochem-brouwer and others added 5 commits March 1, 2024 10:05

client: add preimage doc

1c413fa

trie: export "Path" interface (#3292)

d5a6ca8

* trie: export "Path" interface * Move over Path interface export to types for consistency --------- Co-authored-by: Holger Drewes <[email protected]>

Make trustedSetupPath in Util kzg module optional (#3296)

0b366a2

jochem-brouwer dismissed stale reviews from g11tech and themself via 1903314 March 1, 2024 09:05

jochem-brouwer force-pushed the client/save-preimages branch from 14ffd9e to 1903314 Compare March 1, 2024 09:05

jochem-brouwer approved these changes Mar 1, 2024

View reviewed changes

client: add tx preimages to preimage test cases

1408e29

jochem-brouwer force-pushed the client/save-preimages branch from ee09f29 to 1408e29 Compare March 1, 2024 11:09

jochem-brouwer added 3 commits March 1, 2024 12:09

Merge branch 'master' into client/save-preimages

ad72130

Merge branch 'master' into client/save-preimages

a50ee60

Merge branch 'client/save-preimages' of github.com:ethereumjs/ethereu…

7a0842d

…mjs-monorepo into client/save-preimages

jochem-brouwer merged commit f01264c into master Mar 1, 2024
45 of 46 checks passed

holgerd77 reviewed Mar 1, 2024

View reviewed changes

holgerd77 added a commit that referenced this pull request Mar 1, 2024

Add changes from Preimage PR #3143

8ebcb47

holgerd77 added a commit that referenced this pull request Mar 11, 2024

Add changes from Preimage PR #3143

8d23def

holgerd77 added a commit that referenced this pull request Mar 18, 2024

Add changes from Preimage PR #3143

87a3a73

holgerd77 deleted the client/save-preimages branch July 11, 2024 08:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

client: save preimages feature #3143

client: save preimages feature #3143

gabrocheleau commented Nov 5, 2023 •

edited

Loading

codecov bot commented Nov 5, 2023 •

edited

Loading

jochem-brouwer commented Nov 5, 2023

gabrocheleau commented Nov 6, 2023

jochem-brouwer commented Nov 6, 2023

holgerd77 commented Feb 26, 2024

jochem-brouwer left a comment •

edited

Loading

jochem-brouwer Feb 26, 2024

jochem-brouwer Feb 26, 2024 •

edited

Loading

gabrocheleau Feb 27, 2024

g11tech Feb 28, 2024

gabrocheleau Feb 29, 2024

g11tech left a comment

jochem-brouwer left a comment

jochem-brouwer left a comment

holgerd77 commented Mar 1, 2024

holgerd77 Mar 1, 2024

client: save preimages feature #3143

client: save preimages feature #3143

Conversation

gabrocheleau commented Nov 5, 2023 • edited Loading

codecov bot commented Nov 5, 2023 • edited Loading

Codecov Report

jochem-brouwer commented Nov 5, 2023

gabrocheleau commented Nov 6, 2023

jochem-brouwer commented Nov 6, 2023

holgerd77 commented Feb 26, 2024

jochem-brouwer left a comment • edited Loading

Choose a reason for hiding this comment

jochem-brouwer Feb 26, 2024

Choose a reason for hiding this comment

jochem-brouwer Feb 26, 2024 • edited Loading

Choose a reason for hiding this comment

gabrocheleau Feb 27, 2024

Choose a reason for hiding this comment

g11tech Feb 28, 2024

Choose a reason for hiding this comment

gabrocheleau Feb 29, 2024

Choose a reason for hiding this comment

g11tech left a comment

Choose a reason for hiding this comment

jochem-brouwer left a comment

Choose a reason for hiding this comment

jochem-brouwer left a comment

Choose a reason for hiding this comment

holgerd77 commented Mar 1, 2024

holgerd77 Mar 1, 2024

Choose a reason for hiding this comment

gabrocheleau commented Nov 5, 2023 •

edited

Loading

codecov bot commented Nov 5, 2023 •

edited

Loading

jochem-brouwer left a comment •

edited

Loading

jochem-brouwer Feb 26, 2024 •

edited

Loading