-
Notifications
You must be signed in to change notification settings - Fork 719
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
lighthouse proposer-reorgs feature causes proposing missed #5437
Comments
Can you share some logs from mev-boost during this time? It actually seems like there should have been enough time for the block to propagate. 1.5s+ (from 2.3s to 4s) should be enough I'll check some more metrics tomorrow to see at what time the block arrived on other nodes. It might have been borderline on time The SSV thing might have slowed this down a bit too. I'll have a closer look at timestamps to see how long signing took |
No problem. If you need more detailed logs. I would like to post the files on discord.
|
Some more observations: At 1.642s into the slot, Lighthouse has the payload from the builder:
There is very little that needs to happen in the BN after this point, so I suspect the block was finished being built after ~1.6s. The broadcast warning is logged at 2.384s:
The
Looking at my nodes, I can see there's even more delay beyond 2.38s before the block is received by peers. The earliest it was received by the nodes I checked was 7.57s into the slot, i.e. 5.2s after your node attempted to publish it:
This 5s delay either came from:
I think the most likely is (3), but it would be good to rule out (1) and (2) with more mev-boost logs. Unfortunately the mev-boost logs you've posted are not useful as they don't show the request from mev-boost to the relay to unblind the block. Can you check again and send me more mev-boost logs? I'm |
Related (includes full mev-boost logs): |
@SimonSMH1015 do you mind sharing some SSV logs as well? It would be interesting to investigate the 700ms delay there too |
Sent you on Discord |
Closing this, as it should be resolved now that the bloxroute's issues are resolved. Please open a new issue or reopen this issue if you get another missed block. |
Description
Please provide a brief description of the issue.
Lighthouse client has a propose featrue that they prefer to do a reorg once some rules are satisfied. Last week (Mar-15-2024 17:14:11 UTC+0) at slot 8641569 we encounted a propose miss due to proposer-reorgs feature. We connected two ssv operator with aound 500 validators to a lighthose mainnet full node. And the validator with index 990976 are set to propose block at slot 8641569.
Because the previous slot was considered as a weak head, which triggerd the lighthouse proposer-reorgs feature, it attempt to do a reorg and took 500 milliseconds longer than usual to propose. At the same time, because our node is connected to the SSV operator, it is slower than the directly connected validator when requesting the blinded block . This lead to block was broadcast 2380 milliseconds after the start of the propose.
However, the next proposer, proposer of slot 8641570, which is alao a lighthouse client, should think our block as a late block and it should use the proposer-reorgs feature to do the reorg again. As a result, slot 8641569 was reorged and we got a propose miss.
Version
Please provide your Lighthouse and Rust version. Are you building from
stable
orunstable
, which commit?5.1.0 stable
Present Behaviour
Describe the present behaviour of the application, with regards to this
issue.
Lighthosue tried to do a reorg at 8641569.
but took longer time than usual, which caused a delayed broadcast.
And the next proposer with a lighthouse client sucessfully performed another reorg at slot 8641570
and lead to my validator got a propose miss at slot 8641569.
![proposeMiss](https://private-user-images.githubusercontent.com/39431803/314000218-ab97612b-dcf7-49e5-9ae3-6fa765882c0c.PNG?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ3ODA5NDIsIm5iZiI6MTcyNDc4MDY0MiwicGF0aCI6Ii8zOTQzMTgwMy8zMTQwMDAyMTgtYWI5NzYxMmItZGNmNy00OWU1LTlhZTMtNmZhNzY1ODgyYzBjLlBORz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MjclMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODI3VDE3NDQwMlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTFhMTgwZjVkY2E2ZTYxZjBhZTdjMTIzMmQwZjIzYTQ1ODAyMTQyN2Y1MTliYmQ5YTQzYWRjZjU3MGJmYWUwNjUmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.ewQ2gpPhp64Kjte02LSRP2kqbPevQ4i5asaqqx7bxnM)
Expected Behaviour
How should the application behave?
Not sure if we should not try a propose re-org at slot 8641569.
Steps to resolve
Please describe the steps required to resolve this issue, if known.
The text was updated successfully, but these errors were encountered: