Signed-off-by: stayweek <[email protected]>

…es (erigontech#9805)

we recently changed field which used to measure progress from
`t.BytesCompleted()` to `t.Stats().BytesReadData` - and seems it doesn't
include "re-hashing" progress - so progress doesn't grow in logs and
looks like it's stuck.

argument to change - was: `t.BytesCompleted()` is not recommended to use
for "download rate calculation".

We don't use this field for rate calc, we using it for progress calc

…rigontech#9857)

For better integration with GH Actions added overview in the end of
reports and exit with code 1 in case when met some errors.

This pr implements following beacon api

https://ethereum.github.io/beacon-APIs/#/Validator/prepareBeaconCommitteeSubnet
Steps:
1. once receive committee subscription request, record this subscription
info associated with validator index.
2. if `isAggregator` flag is on, prepare internal data structure to
collect and aggregate incoming attestation.
3. set expiry and begin to subscribe `beacon_attestation_{subnet}`, so
we are capable of receiving attestation data from peer.
4. whenever receive attestation data, try aggregate it if needed.

Fix erigontech#9834

Let's check and validate blob configurations only if the txn is BlobTx

Signed-off-by: jsvisa <[email protected]>

Basically, now in order to mark blobs of a block as downloaded, re-check
the blocks payload used to download it against the blob store.

…rigontech#9866)

missing goroutine

# Sync Committee Contribution pooling

First of all, a distinction:
* Sync Committee Messages are single unaggregated messages from a
specific validator
* Sync Committee Contributions are aggregated messages from many
validators.

We get these 2 messages from 2 different gossip sources, and then after
validating the Gossip rules, we send everything to the pool which
aggregate the `sync committee` or `contribution` into another aggregate.

## Sync Committee subscription:
/eth/v1/validator/sync_committee_subscriptions

The subscription just starts to subscribe to the `Sync Committee`'s
Gossip channel. it is actually really simple, here is the pseudo-code
for how the subscription happen for each `ValidatorIndex` requested:

```vb
Function PostEthV1ValidatorSyncCommitteeSubscriptions
    Request: ValidatorIndicies []uint64


    Get the head state of the blockchain


    For each subRequest in ValidatorIndicies do

        
        Compute the subnets for the validator index from subRequest using the headState

        For each subnet in syncnets do
            Construct a topic name for the sync committee subnet
                    Subscribe to the topic with the sync committee subnet the topic with the calculated expiry time in sentinel
            
End Function
```

### Extras

* /eth/v1/validator/contribution_and_proofs - Submit to the node a
contribution to be republished on the Gossip.
* /eth/v1/validator/sync_committee_contribution - Retrieve a
contribution (from the pool) that we have aggregated from the gossip.
* /eth/v1/beacon/pool/sync_committees - Submit to the node a sync sync
committee messages to be republished on the Gossip.

---------

Co-authored-by: Kewei <[email protected]>

Basically pruning was overpruning useful blobs which were needed for
processing of blocks at chain tip, thus leading to degraded performance.

Regressions fixed:

1) Make a copy of `GossipData` so that we do not risk republishing
gossip bogus/mem corrupted data as published. (`PublishGossip` takes
hold of the data bytes for a long time)
2) Blobs download not showing progress
3) Increase request dial timeout to specs
4) Made `Attestations` routine recoverable
5) Fixed inconsistency in the total active balance cache which caused a
panic

we did make `downloader` more passive (erigon-driven), but it did break
`--seedbox` flag also

- new method `DownloadAndSaveTorrentFile` did download data files
instead of .torrent files because `WebSeeds.ByName()` return data urls -
not .torrent urls
- calling wrong url to download .torrent
- save value to disk without `.torrent` suffix 
- size attack prevention - did return nil err and it stopped webseed
scanning

…ts with modified bytecode (erigontech#9438)

In this PR I'm introducing so-called `overlays` for doing ad-hoc state
simulations of contracts which replay the state with a modified bytecode
for any given block range.

The new feature adds two new RPC methods: `overlay_callConstructor` and
`overlay_getLogs`

I've also added a few bench tests and a new postman collection which
checks both new methods for the WETH contract in original and modified
version.

## Motivation
Overlays allow you to add your custom logic to already deployed
contracts and simulate on top of them. With overlays you can create new
view functions, modify existing ones, change field visibility, emit new
events and query the historical data of any contract with your modified
source code.

Similar commercial products exist in the space and I believe making this
feature opensource in erigon will make it easier for everyone to tinker
with it and build new cool things on top of it ✨

## Usage
1) compile your contract to obtain the new deployment bytecode, append
the constructor args at the end, if any
2) call `overlay_callConstructor` with the result from 1) to receive the
runtime bytecode
3) call `overlay_getLogs` to simulate new logs, passing 2) as state
overrides
([example](https://github.com/crebsy/erigon/blob/devel/cmd/rpcdaemon/postman/Overlay_Testing.json))
4) call `eth_call` to simulate new view function calls passing 2) as
state overrides

## Ad-hoc query performance & historical back-filling
The current implementation for ad-hoc range queries of `overlay_getLogs`
is optimized so that irrelevant blocks are skipped entirely from the
simulation context. This can lead to very fast results of a few minutes
or even seconds for simulating the entire historical logs of a contract
which makes any infra for back-filling redundant. However, such
shortcuts are only effectively working for contracts which are not
touching too many blocks of the chain's history. So depending on the
contract you're trying to patch, you still might need to create a custom
back-filling script which simulates the new logs with e.g. 1000 or 10000
block ranges.

As a reference for a "busy" contract the simulation rate for the
[Uniswap V2 router
](https://etherscan.io/address/0x7a250d5630b4cf539739df2c5dacb4c659f2488d#code)
was ~ 280 blocks/s on a host with 24 cores. So while small block ranges
returned relatively quickly, the simulation of the entire historical
logs for this contract took a few hours on a single node.

I'm not collecting any perf metrics yet for this but this could be
easily added in the future for tracking better implementations and
insights about contract <-> block distribution.

## Configuration
1) Enable the `ots,overlay` api namespaces by adding it to the
`--http.api` flag

2) The implementation of `overlay_getLogs` is parallelized so it's good
practice to increase the `--db.read.concurrency` to a higher value, e.g.
256 so other erigon goroutines are not getting blocked.


## Supported networks
Tested with ethereum and polygon nodes

Fixes problems found during bor-mainnet sync tests:
- FetchHeaders: paging-like fetching of headers from a peer - assumption
was that a peer response will contain all e.g. 1024 headers since they
can fit in < 2 MB however despite that some peers respond with only 16
headers so we need to send further requests until we get all e.g. 1024
headers
- It looks like another assumption didn't hold - when we penalize a peer
sentry doesn't generate a disconnect even. For that this PR creates a
tracking penalizer that updates the peer tracker with a disconnect.
- Removes another wrong incorrect - ErrEmptyBody validation because
there are blocks which are empty, e.g. first 100,000k blocks
- Sentry SentMessageById was disallowing GetBlockBodiesMsg, this PR
fixes this

This PR separates the data collection layer from the `Antiquary` object
to the `beaconStatesCollector`, which hopefully makes the code cleaner.
It is still messy, but now we have a proper separation between the logic
and the data collection layer.

The reason why we have internal buffers here is that MDBX locks the
entire database when writing to it, so we need to minimize the time
spent in the write lock. So instead of writing the historical states on
write transactions and keeping the lock for hours, we accumulate them in
memory and write them in a single write transaction.

Fixes problems found during bor-mainnet sync tests:

- sync.Run was missing a for loop for initial sync convergences
- temporarily disabling ForkChoiceUpdate execution since we are yet to
solve and add missing state sync events and spans
- no need to flush storage inside block downloader (better async
performance)
- logging improvements

…ontech#9882)

- fixes an issue where heimdall polling goroutines get terminated upon a
heimdall http err which is incorrect - these polling routines should be
kept alive upon errors and errors should simply be logged
- adds logging
- reduces cyclomatic complexity by extracting logic for polling events,
spans, checkpoints

---------

Co-authored-by: battlmonstr <[email protected]>

- collecting info for "Headers" stage

Implemented a network speed test to run every 180 seconds. This
information is crucial for diagnostic tools. It helps us identify if a
client is reaching their maximum capacity.

…rigontech#9892)

https://ethereum.github.io/beacon-APIs/#/Validator/getAggregatedAttestation

Fixes a whim reported in erigontech#9783

Do not open files which are bigger than 32 bytes but "has no" words in
it.

P2P fails on restart because rawdb.ReadCurrentHeader returns a nil
header. It looks like ReadHeadHeaderHash fails to find the current
header hash. However the correct hash is returned by ReadHeadBlockHash.

Let's use ReadHeadBlockHash, because the status needs to report a header for which we have a full block body.

Brings down time to fetch all checkpoints from 4 hours to 42 seconds in
combination with maticnetwork/heimdall#1154

```
DBUG[04-09|21:12:39.754] [bor.heimdall] fetching all checkpoints 
DBUG[04-09|21:13:10.355] [bor.heimdall] fetching all checkpoints progress page=5 len=40000
DBUG[04-09|21:13:22.061] [bor.heimdall] fetching all checkpoints done len=60474 duration=42.307967083s
```

…#9315)" (erigontech#9900)

we have this PR in release and don't have in devel

Co-authored-by: Andrew Ashikhmin <[email protected]>

… files without .torrent (erigontech#9782)

```
if !prohibited(file) && !exists(file) {
   create(file)
}
```
such pattern is always race - even if `prohibited`/`exists()`/`create()`
are atomic inside.
to eliminate race - need move towards analog of `CompareAndSwap` -
`CreateIfNotProhibited` which in 1 mutex lock will: check_lock,
existence, create.

It seems like you need context when implementing
`GetModifiedAccountsByHash`.
However, it is not specified in the interface.

Fixes erigontech#9898

When not specified, gas fee cap should default to base fee to prevent
errors where gas fee cap is less than block base fee.

Fixes erigontech#9870

endpoint: `/eth/v1/beacon/pool/attestations`
https://ethereum.github.io/beacon-APIs/#/Beacon/submitPoolAttestations

fixes two minor race conditions

fix `IsTopicBeaconAttestation ` and slightly revise notifier.

Starts Erigon from scratch, monitors the snapshot download process and
waits for it to complete within 4 hours, reporting times for download
and indexing phases. Fails if the download phase doesn't finish on time

- added updating prometheus sync metrics once in 180 seconds log loop
erigontech#8438

This PR adds headers which prevent the cloudflare firewall from banning
the downloaders http requests.

At the moment these are not obfuscated in the codebase.

This should be reviewed before this change is committed to the core
repo.

staking - it's some feature to sum 2 graphics - have no idea when it can be useful - only confusing now.

The Gossip Manager is responsible for handling incoming messages from
the network and handling those accordingly, if the handling goes well,
these messages are then re-gossiped to the rest of the network. The
Gossip Manager registers a `Service` object which is responsible for
handling the incoming messages.

Below is a small graph on how the Gossip Manager works:
<img width="754" alt="gossipmanager"
src="https://github.com/ledgerwatch/erigon/assets/29233688/57fddf79-b417-4792-8aac-1a08534c1207">

- Rename
  - state.AggregatorV3Context -> state.FilesRoTx
  - MakeContext() -> BeginFilesRo()
  - aggCtx -> aggTx
  - InvertedIndexContext -> InvertedIndexRoTx, ic -> iit
  - HistoryContext -> HistoryRoTx, hc -> ht
  - DomainContext -> DomainRoTx, dc -> dt

Remove
- Aggregator class

…ech#9902)

See https://github.com/gnosischain/configs

I also added `EthereumClock` which is an abstraction over ETH 2.0
reliance on `unix.Nano()` for better testing in the future. I just added
the `struct` this object is not used anywhere. also had to regenerate
some mocks.

Improves upon erigontech#9913
Closes erigontech#9889

…9926)

* fix GrpcServer.p2pServer race
* fix HeaderDownload.latestMinedBlockNumber race

see erigontech#8704

…ontech#9934)

…igontech#9928)

Problematic situation
`runPeer` blocks on `rw.ReadMsg()`, however in the meantime the peer
gets penalised.

Expected behaviour
the peer to get disconnected and for sentry to generate a Disconnect
event

Actual behaviour
no disconnect event gets generated, peer is stuck in `rw.ReadMsg()`

Fix
call `pi.peer.Disconnect(reason)` as part of `peerInfo.Remove(reason)`
during `Penalize`

1. `Disconnect` sends a disc reason to `p.disc` channel
2. `p.disc` channel is read in `Peer.run` -
https://github.com/ledgerwatch/erigon/blob/devel/p2p/peer.go#L279
3. it causes the function to exit and in its defer call close `p.closed`
channel
4. `p.closed` channel is used as a closing channel in the
`protoRW.closed` in both `ReadMsg` and `WriteMsg` so once it is closed
those functions exit

After the fix done in erigontech#9928
trackingPenalizer is no longer needed

…#9947)

minor tidy up of code I wrote last week

…rigontech#9946)

```
Gotten unexpected root                   got="[25 97 1 2 224 23 12 23 98 201 172 124 211 121 115 79 189 245 64 97 147 7 148 171 53 37 124 29 33 89 66 161]" expected=0xf09b99bb8c94f0c631625f020805ff21f111fac6f889fe78b9b91e71c6cf16ce
```

see
erigontech/silkworm-go@2e4f273

fixes erigontech#9299

…ch#9961)

Set default values for "debug.addrs" and "diagnostics.addr" to make
support command compact:
OLD -> ./build/bin/erigon support --debug.addrs localhost:6060
--diagnostics.addr localhost:8080 --diagnostics.sessions 15947373
NEW -> ./build/bin/erigon support --diagnostics.sessions 15947373

Introduces `ErrCompressedFileCorrupted`.

Those errors should be handled properly inside aggregator and inside
different utilities.
At least we do not open wrong compressed files but we sometimes want to
do some extra on those files (eg during run of a `downloader`)

* Added cf headers to webseed requests

* Extended torrent tequest transport to retry 500+ status responses and
count progress and fails

* Added internal torrent status logging for stopped and no-progress
files

Co-authored-by: awskii <[email protected]>

…rigontech#9957)

### Change ### 
Adds a `disableBlockDownload` boolean flag to current implementation of
sentry multi client to disable built in header and body download
funcitonality.

### Long Term ###
Long term we are planning to refactor sentry multi client and de-couple
it from custom header and body download logic.

### Context ### 
Astrid uses its own body download logic which is de-coupled from sentry
multi client.

When both are used at the same time (using `--polygon.sync=true`) there
are 2 problematic scenarios:
- restarting Astrid takes a very long time due to the init logic of
sentry multi client. It calls `HeaderDownload.RecoverFromDb` which is
coupled to the Headers stage in the stage loop. So if Astrid has fetched
1 million headers but hasn't committed execution yet then this will
result in very slow start up since all 1 million blocks have to be read
from the DB. Example logs:
```
[INFO] [04-16|12:55:42.254] [downloader] recover headers from db     left=65536
...
[INFO] [04-16|13:03:42.254] [downloader] recover headers from db     left=65536
```

- debug log messages warning about sentry consuming being slow since
Astrid does not use `HeaderDownload` and `BodyDownload` so there is
nothing consuming the headers and bodies from these data structures.
This has no logical impact, however clogs resources. Example logs:
```
[DBUG] [04-16|14:03:15.311] [sentry] consuming is slow, drop 50% of old messages msgID=BLOCK_HEADERS_66
[DBUG] [04-16|14:03:15.311] [sentry] consuming is slow, drop 50% of old messages msgID=BLOCK_HEADERS_66
```

Existing prune had some confusing logic (thanks to me).

In PR, change the logic to:
if receipt, or any individual log in the receipt contains an address in
`noPruneContracts` the entire log must be stored and indexed.

Also fixes a regression that logs were not getting indexed - this was
due to
```
if l.BlockNumber < pruneBlock && cfg.noPruneContracts != nil && !cfg.noPruneContracts[l.Address] {
```
Since the individual log object didn't have the block number - something
that shouldn't matter.
In PR, change the logic to:
Use `blockNum` from outer loop

…oint (erigontech#9971)

--reset clears all the tables associated with a stage, while we just
want to reset the prune_at progress to 0 for some situations

…#9970)

Fixes panic and log prints
```
DBUG[04-17|03:03:11.204] [agg] History.openFiles                  err="compressed file \"v1-accounts.1344-1408.v\" dictionary is corrupted: size 9.3 GB but no words in it" f=/mnt/erigon/snapshots/history/v1-accounts.1344-1408.v
panic: History(code).openFiles: compressed file "v1-code.1344-1408.v" dictionary is corrupted: size 3.9 MB but no words in it

goroutine 1 [running]:
github.com/ledgerwatch/erigon/cmd/integration/commands.allSnapshots.func1()
	github.com/ledgerwatch/erigon/cmd/integration/commands/stages.go:1743 +0x4c8
sync.(*Once).doSlow(0xc00102fbc0?, 0xc00102b870?)
	sync/once.go:74 +0xc2
sync.(*Once).Do(...)
	sync/once.go:65
github.com/ledgerwatch/erigon/cmd/integration/commands.allSnapshots({0x1ebbe30?, 0x2c90460?}, {0x1ec32a0?, 0xc0004d68c0?}, {0x1ec6508?, 0xc000519f00?})
	github.com/ledgerwatch/erigon/cmd/integration/commands/stages.go:1717 +0x90
github.com/ledgerwatch/erigon/cmd/integration/commands.openDB({{0x1ec6508, 0xc0000f1800}, 0xc000df6410, 0x1cb3c98, {0xc000a25290, 0x15}, 0x0, 0x20000000000, 0x40000000, 0xffffffffffffffff, ...}, ...)
	github.com/ledgerwatch/erigon/cmd/integration/commands/root.go:108 +0x4a5
github.com/ledgerwatch/erigon/cmd/integration/commands.init.func11(0x2b5cbe0, {0x1a0b51f?, 0x4?, 0x1a0b3df?})
	github.com/ledgerwatch/erigon/cmd/integration/commands/stages.go:366 +0xea
github.com/spf13/cobra.(*Command).execute(0x2b5cbe0, {0xc000895470, 0x3, 0x3})
	github.com/spf13/[email protected]/command.go:987 +0xab1
github.com/spf13/cobra.(*Command).ExecuteC(0x2b5c340)
	github.com/spf13/[email protected]/command.go:1115 +0x3ff
github.com/spf13/cobra.(*Command).Execute(...)
	github.com/spf13/[email protected]/command.go:1039
github.com/spf13/cobra.(*Command).ExecuteContext(0x0?, {0x1ebbf90?, 0xc000df63c0?})
	github.com/spf13/[email protected]/command.go:1032 +0x47
main.main()
	github.com/ledgerwatch/erigon/cmd/integration/main.go:15 +0xe6
```

…ontech#9983)

- put initial speed test inside go routine
- passed context to download, upload, latency tests to stop if parent
context cancelled
- increased test interval to 30 minutes

remove some deprecated linters and fields:
https://golangci-lint.run/product/changelog/

…nced node (erigontech#9984)

for erigontech#9940

`stats.MetadataReady++` was called outside if and inside

also created erigontech#9997

- existence check and creation must be inside same `mutex.Lock`
- use ".tmp + rename" trick

Example of possible race: 
```
[EROR] [04-19|03:33:12.908] Erigon startup                           err="rename /mnt/erigon/snapshots/v1-000900-001000-transactions.seg.torrent.tmp /mnt/erigon/snapshots/v1-000900-001000-transactions.seg.torrent: no such file or directory"
```

This should fix erigontech#9854

This task is mostly implemented to be used in
`erigon/erigon-lib/downloader/mdbx_piece_completion.go` and maybe in
`nodesDB` (where we need many parallel RwTx)

I was agains adding this "trick"/"api" last years, because thought that
we can implement our App to be more 1-big-rwtx-friendly. And we did it
in Erigon - StagedSync. TxPool also did, but with a bit less happy face
- by "map+mutex with periodic flush to db". But `anacrolix/torrent` is
external library and unlikely will survive such big mind-model-change.
Maybe it's time to add `db.Batch()`.

#### Batch Rw transactions

Each `DB.Update()` waits for disk to commit the writes. This overhead
can be minimized by combining multiple updates with the `DB.Batch()`
function:

```go
err := db.Batch(func(tx *bolt.Tx) error {
	...
	return nil
})
```

Concurrent Batch calls are opportunistically combined into larger
transactions. Batch is only useful when there are multiple goroutines
calling it.

The trade-off is that `Batch` can call the given
function multiple times, if parts of the transaction fail. The
function must be idempotent and side effects must take effect only
after a successful return from `DB.Batch()`.

For example: don't display messages from inside the function, instead
set variables in the enclosing scope:

```go
var id uint64
err := db.Batch(func(tx *bolt.Tx) error {
	// Find last key in bucket, decode as bigendian uint64, increment
	// by one, encode back to []byte, and add new key.
	...
	id = newValue
	return nil
})
if err != nil {
	return ...
}
fmt.Println("Allocated ID %d", id)
```


---- 

Implementation mostly taken from
https://github.com/etcd-io/bbolt/?tab=readme-ov-file#batch-read-write-transactions

Maybe in future can push-down it to
https://github.com/erigontech/mdbx-go

- PutUvarint can produce 10 bytes
- re-using buffer - faster and less gc

…or: See comment (erigontech#9972)

* Fixed and simplified unaggregated bits check.
* There are 2 bits on, one for the attester and one for the
End-of-bitlist, needed to account for end of bitlist bit
 * Wrong publishing topic for sync_committee_ messages
* Added more Ignore by receiving specific errors to avoid forwarding
useless data.
 * Replaced `validateAttestation` with full message processing
 * Fixed forwarding of sync committee aggregates
 * Fixed subnet announcements

---------

Co-authored-by: kewei <[email protected]>

…n't check .lock. Otherwise can't create .torrent for existing .seg files. (erigontech#10004)

…ch#10014)

This is for E2.

It implements the backward compatible output field for traces on
ots_traceTransaction: otterscan/execution-apis#1

It'll be consumed by Otterscan in an upcoming release of this feature:
otterscan/otterscan#1530

there was error:
```
prog.go:12:2: missing go.sum entry for module providing package github.com/golang/mock/mockgen/model; to add:
	go mod download github.com/golang/mock
```