Upstream v2.60.2 #106
Merged
Upstream v2.60.2 #106
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: stayweek <[email protected]>
…es (ledgerwatch#9805) we recently changed field which used to measure progress from `t.BytesCompleted()` to `t.Stats().BytesReadData` - and seems it doesn't include "re-hashing" progress - so progress doesn't grow in logs and looks like it's stuck. argument to change - was: `t.BytesCompleted()` is not recommended to use for "download rate calculation". We don't use this field for rate calc, we using it for progress calc
…edgerwatch#9857) For better integration with GH Actions added overview in the end of reports and exit with code 1 in case when met some errors.
This pr implements following beacon api https://ethereum.github.io/beacon-APIs/#/Validator/prepareBeaconCommitteeSubnet Steps: 1. once receive committee subscription request, record this subscription info associated with validator index. 2. if `isAggregator` flag is on, prepare internal data structure to collect and aggregate incoming attestation. 3. set expiry and begin to subscribe `beacon_attestation_{subnet}`, so we are capable of receiving attestation data from peer. 4. whenever receive attestation data, try aggregate it if needed.
Fix ledgerwatch#9834 Let's check and validate blob configurations only if the txn is BlobTx Signed-off-by: jsvisa <[email protected]>
Basically, now in order to mark blobs of a block as downloaded, re-check the blocks payload used to download it against the blob store.
# Sync Committee Contribution pooling
First of all, a distinction:
* Sync Committee Messages are single unaggregated messages from a
specific validator
* Sync Committee Contributions are aggregated messages from many
validators.
We get these 2 messages from 2 different gossip sources, and then after
validating the Gossip rules, we send everything to the pool which
aggregate the `sync committee` or `contribution` into another aggregate.
## Sync Committee subscription:
/eth/v1/validator/sync_committee_subscriptions
The subscription just starts to subscribe to the `Sync Committee`'s
Gossip channel. it is actually really simple, here is the pseudo-code
for how the subscription happen for each `ValidatorIndex` requested:
```vb
Function PostEthV1ValidatorSyncCommitteeSubscriptions
Request: ValidatorIndicies []uint64
Get the head state of the blockchain
For each subRequest in ValidatorIndicies do
Compute the subnets for the validator index from subRequest using the headState
For each subnet in syncnets do
Construct a topic name for the sync committee subnet
Subscribe to the topic with the sync committee subnet the topic with the calculated expiry time in sentinel
End Function
```
### Extras
* /eth/v1/validator/contribution_and_proofs - Submit to the node a
contribution to be republished on the Gossip.
* /eth/v1/validator/sync_committee_contribution - Retrieve a
contribution (from the pool) that we have aggregated from the gossip.
* /eth/v1/beacon/pool/sync_committees - Submit to the node a sync sync
committee messages to be republished on the Gossip.
---------
Co-authored-by: Kewei <[email protected]>
Basically pruning was overpruning useful blobs which were needed for processing of blocks at chain tip, thus leading to degraded performance. Regressions fixed: 1) Make a copy of `GossipData` so that we do not risk republishing gossip bogus/mem corrupted data as published. (`PublishGossip` takes hold of the data bytes for a long time) 2) Blobs download not showing progress 3) Increase request dial timeout to specs 4) Made `Attestations` routine recoverable 5) Fixed inconsistency in the total active balance cache which caused a panic
we did make `downloader` more passive (erigon-driven), but it did break `--seedbox` flag also
- new method `DownloadAndSaveTorrentFile` did download data files instead of .torrent files because `WebSeeds.ByName()` return data urls - not .torrent urls - calling wrong url to download .torrent - save value to disk without `.torrent` suffix - size attack prevention - did return nil err and it stopped webseed scanning
…ts with modified bytecode (ledgerwatch#9438) In this PR I'm introducing so-called `overlays` for doing ad-hoc state simulations of contracts which replay the state with a modified bytecode for any given block range. The new feature adds two new RPC methods: `overlay_callConstructor` and `overlay_getLogs` I've also added a few bench tests and a new postman collection which checks both new methods for the WETH contract in original and modified version. ## Motivation Overlays allow you to add your custom logic to already deployed contracts and simulate on top of them. With overlays you can create new view functions, modify existing ones, change field visibility, emit new events and query the historical data of any contract with your modified source code. Similar commercial products exist in the space and I believe making this feature opensource in erigon will make it easier for everyone to tinker with it and build new cool things on top of it ✨ ## Usage 1) compile your contract to obtain the new deployment bytecode, append the constructor args at the end, if any 2) call `overlay_callConstructor` with the result from 1) to receive the runtime bytecode 3) call `overlay_getLogs` to simulate new logs, passing 2) as state overrides ([example](https://github.com/crebsy/erigon/blob/devel/cmd/rpcdaemon/postman/Overlay_Testing.json)) 4) call `eth_call` to simulate new view function calls passing 2) as state overrides ## Ad-hoc query performance & historical back-filling The current implementation for ad-hoc range queries of `overlay_getLogs` is optimized so that irrelevant blocks are skipped entirely from the simulation context. This can lead to very fast results of a few minutes or even seconds for simulating the entire historical logs of a contract which makes any infra for back-filling redundant. However, such shortcuts are only effectively working for contracts which are not touching too many blocks of the chain's history. So depending on the contract you're trying to patch, you still might need to create a custom back-filling script which simulates the new logs with e.g. 1000 or 10000 block ranges. As a reference for a "busy" contract the simulation rate for the [Uniswap V2 router ](https://etherscan.io/address/0x7a250d5630b4cf539739df2c5dacb4c659f2488d#code) was ~ 280 blocks/s on a host with 24 cores. So while small block ranges returned relatively quickly, the simulation of the entire historical logs for this contract took a few hours on a single node. I'm not collecting any perf metrics yet for this but this could be easily added in the future for tracking better implementations and insights about contract <-> block distribution. ## Configuration 1) Enable the `ots,overlay` api namespaces by adding it to the `--http.api` flag 2) The implementation of `overlay_getLogs` is parallelized so it's good practice to increase the `--db.read.concurrency` to a higher value, e.g. 256 so other erigon goroutines are not getting blocked. ## Supported networks Tested with ethereum and polygon nodes
Fixes problems found during bor-mainnet sync tests: - FetchHeaders: paging-like fetching of headers from a peer - assumption was that a peer response will contain all e.g. 1024 headers since they can fit in < 2 MB however despite that some peers respond with only 16 headers so we need to send further requests until we get all e.g. 1024 headers - It looks like another assumption didn't hold - when we penalize a peer sentry doesn't generate a disconnect even. For that this PR creates a tracking penalizer that updates the peer tracker with a disconnect. - Removes another wrong incorrect - ErrEmptyBody validation because there are blocks which are empty, e.g. first 100,000k blocks - Sentry SentMessageById was disallowing GetBlockBodiesMsg, this PR fixes this
This PR separates the data collection layer from the `Antiquary` object to the `beaconStatesCollector`, which hopefully makes the code cleaner. It is still messy, but now we have a proper separation between the logic and the data collection layer. The reason why we have internal buffers here is that MDBX locks the entire database when writing to it, so we need to minimize the time spent in the write lock. So instead of writing the historical states on write transactions and keeping the lock for hours, we accumulate them in memory and write them in a single write transaction.
Fixes problems found during bor-mainnet sync tests: - sync.Run was missing a for loop for initial sync convergences - temporarily disabling ForkChoiceUpdate execution since we are yet to solve and add missing state sync events and spans - no need to flush storage inside block downloader (better async performance) - logging improvements
…erwatch#9882) - fixes an issue where heimdall polling goroutines get terminated upon a heimdall http err which is incorrect - these polling routines should be kept alive upon errors and errors should simply be logged - adds logging - reduces cyclomatic complexity by extracting logic for polling events, spans, checkpoints --------- Co-authored-by: battlmonstr <[email protected]>
- collecting info for "Headers" stage
Implemented a network speed test to run every 180 seconds. This information is crucial for diagnostic tools. It helps us identify if a client is reaching their maximum capacity.
Fixes a whim reported in ledgerwatch#9783
Do not open files which are bigger than 32 bytes but "has no" words in it.
Cherry pick PR ledgerwatch#10636 to Erigon 2
…ion workflows (ledgerwatch#10778) This PR backports improvements that we added to the E3 tests: recording runner name and db version used for testing on MongoDB database.
Merged all the work done from main branch to keep diagnostics up to date.
Cherry pick PR ledgerwatch#10093 into the release branch Co-authored-by: Minhyuk Kim <[email protected]>
Cherry pick PR ledgerwatch#10825 into the release branch Co-authored-by: Minhyuk Kim <[email protected]>
jyellick
reviewed
Jul 2, 2024
| @@ -27,6 +27,8 @@ func MarshalReceipt( | |||
| if t.Protected() { | |||
| chainId = types.DeriveChainId(&t.V).ToBig() | |||
| } | |||
| case *types.DepositTx: | |||
There was a problem hiding this comment.
Why was this case removed? I believe that GetChainID will panic on deposit tx?
There was a problem hiding this comment.
This is a new file. Yes, deposit tx doesn't have a chain Id.
mmontour1306
approved these changes
Jul 3, 2024
jyellick
approved these changes
Jul 8, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.