Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automate Superchain Registry chain support #8105

Open
wants to merge 60 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
60 commits
Select commit Hold shift + click to select a range
1596721
Initial scripts
emlautarom1 Jan 22, 2025
9b40dfa
Remove `codeHashes` indirection
emlautarom1 Jan 23, 2025
372c69c
Use `BytesIO` instead of `StringIO` for binary data
emlautarom1 Jan 23, 2025
38efd5c
Merge into single script
emlautarom1 Jan 23, 2025
47c0c1e
Rename `folder` to `directory`
emlautarom1 Jan 23, 2025
80d6f33
Store downloads in temp folder
emlautarom1 Jan 23, 2025
577336f
Use tempdir for intermediate `.json` files
emlautarom1 Jan 23, 2025
b3eaa22
Import ordering
emlautarom1 Jan 23, 2025
91b38d9
Generate Nethermind runner configs
emlautarom1 Jan 23, 2025
706f51e
Use `listdir` instead of `walk`
emlautarom1 Jan 24, 2025
6f97b13
Initial Zstd-aware loader
emlautarom1 Jan 24, 2025
86642ec
Use `Limbo` logger
emlautarom1 Jan 24, 2025
d7fb8e3
Use file format aware loader everywhere
emlautarom1 Jan 24, 2025
d718e5e
Remove hardcoded `GenesisHash`
emlautarom1 Jan 24, 2025
215db37
Added copy generated files script
emlautarom1 Jan 24, 2025
4b70457
Generated files
emlautarom1 Jan 24, 2025
b83e942
Update bootnodes
emlautarom1 Jan 27, 2025
e3ff58a
Map `alloc` to `accounts`
emlautarom1 Jan 27, 2025
63abda3
Update autogen ChainSpec files
emlautarom1 Jan 27, 2025
af5520a
Remove `enr` entries
emlautarom1 Jan 27, 2025
785c1c6
Update autogen files
emlautarom1 Jan 27, 2025
3e63d56
Use provided serializer
emlautarom1 Jan 27, 2025
d47804f
Include `GenesisHash`
emlautarom1 Jan 28, 2025
f704108
Update autogen files
emlautarom1 Jan 28, 2025
700d891
Update autogen files
emlautarom1 Jan 28, 2025
486c486
Fix `genesis` fields
emlautarom1 Jan 28, 2025
cdf5d29
Use `4` spaces for Python files
emlautarom1 Jan 28, 2025
0328aec
Merge branch 'master' into feat/superchain-registry
emlautarom1 Jan 28, 2025
2124634
Cleanup script
emlautarom1 Jan 28, 2025
6b58743
Update autogen files
emlautarom1 Jan 28, 2025
238ae6d
Replace `zstd` with `zst`
emlautarom1 Jan 28, 2025
b855822
Formatting
emlautarom1 Jan 28, 2025
40baa40
Fix E2E tests
emlautarom1 Jan 28, 2025
1e8abec
Delete old `zstd` files
emlautarom1 Jan 28, 2025
ff0d242
Merge branch 'master' into feat/superchain-registry
emlautarom1 Jan 31, 2025
2cbd94c
Include licence in headers
emlautarom1 Feb 4, 2025
996590e
Merge branch 'master' into feat/superchain-registry
emlautarom1 Feb 5, 2025
ef45d4b
Hardcode op-mainnet genesis hash
emlautarom1 Feb 6, 2025
ab78b4d
Update autogen files
emlautarom1 Feb 6, 2025
8c04ba8
Include ancient barries in op-mainnet
emlautarom1 Feb 6, 2025
3a11633
Update autogen files
emlautarom1 Feb 6, 2025
a96995a
Add 'worldchain'
emlautarom1 Feb 6, 2025
40e31f9
Remove old `.json` chainspec files
emlautarom1 Feb 6, 2025
d2007e0
Remove snapshot for op-mainnet
emlautarom1 Feb 6, 2025
492f781
Update autogen files
emlautarom1 Feb 6, 2025
caff92b
Merge branch 'master' into feat/superchain-registry
emlautarom1 Feb 6, 2025
d8a11f5
Update `worldchain-sepolia_archive`
emlautarom1 Feb 7, 2025
9812204
Merge branch 'feat/superchain-registry' of https://github.com/Netherm…
emlautarom1 Feb 7, 2025
bd1afbd
Update `worldchain-mainnet_archive`
emlautarom1 Feb 7, 2025
ee03a9b
Update `base-mainnet_archive`
emlautarom1 Feb 7, 2025
55a453c
Update `base-sepolia_archive`
emlautarom1 Feb 7, 2025
6f9ade1
Update `op-sepolia_archive`
emlautarom1 Feb 7, 2025
ced119f
Update `op-mainnet_archive`
emlautarom1 Feb 7, 2025
08d132e
Support only Superchain chains
emlautarom1 Feb 7, 2025
496085c
Cleanup requests
emlautarom1 Feb 7, 2025
8cff3d8
Fix script args
emlautarom1 Feb 7, 2025
8d59cfd
Add superchain workflow
emlautarom1 Feb 7, 2025
28b17e4
Delete copy script
emlautarom1 Feb 7, 2025
433e2ad
Update autogen files
emlautarom1 Feb 7, 2025
af354f6
Rethrow inner exception
emlautarom1 Feb 7, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .editorconfig
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@ file_header_template = SPDX-FileCopyrightText: 2025 Demerzel Solutions Limited\n

[*.cs]
indent_size = 4
[*.py]
indent_size = 4

#### Naming styles ####

Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/update-fast-sync.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ jobs:
pip install setuptools
pip install emoji
- name: Update config files
run: python3 scripts/syncSettings.py ${{ secrets.ETHERSCAN_API_KEY }}
run: python3 scripts/syncSettings.py -k ${{ secrets.ETHERSCAN_API_KEY }}
- name: Create GitHub app token
id: gh-app
uses: actions/create-github-app-token@v1
Expand Down
56 changes: 56 additions & 0 deletions .github/workflows/update-superchain.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
name: Update OP Superchain chains

on:
schedule:
- cron: '0 0 * * 0'
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The workflow will run weekly. Does this sound OK @kamilchodola ?

workflow_dispatch:

permissions:
contents: write
pull-requests: write

jobs:
update-configs:
name: Update OP Superchain chains
runs-on: ubuntu-latest
steps:
- name: Check out repository
uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.13.1'
- name: Install dependencies
run: |
pip install zstandard
pip install safe-pysha3
pip install requests
pip install setuptools
pip install emoji
- name: Generate Chainspec and config files
run: python3 scripts/superchain.py -v
- name: Update fast sync settings
run: python3 scripts/syncSettings.py --superchain
- name: Copy generated files
run:
cp output/chainspec/* ./src/Nethermind/Chains
cp output/runner/* ./src/Nethermind/Nethermind.Runner/configs
- name: Create GitHub app token
id: gh-app
uses: actions/create-github-app-token@v1
with:
app-id: ${{ vars.APP_ID }}
private-key: ${{ secrets.APP_PRIVATE_KEY }}
- name: Create a pull request
env:
GH_TOKEN: ${{ steps.gh-app.outputs.token }}
run: |
head_branch=feature/update-superchain-${{ github.run_number }}-${{ github.run_attempt }}
git config user.name "${{ github.actor }}"
git config user.email "${{ github.actor }}@users.noreply.github.com"
git checkout -b $head_branch
git add -A
git commit -am "Update OP Superchain chains"
git push origin $head_branch
gh pr create -B master -H $head_branch -t "Update OP Superchain chains" \
-b "Generated Chainspec and config files" -l configuration -l sync
337 changes: 337 additions & 0 deletions scripts/superchain.py

Large diffs are not rendered by default.

76 changes: 52 additions & 24 deletions scripts/syncSettings.py
Original file line number Diff line number Diff line change
@@ -1,21 +1,14 @@
# SPDX-FileCopyrightText: 2022 Demerzel Solutions Limited
# SPDX-License-Identifier: LGPL-3.0-only

import argparse
import json
import subprocess
import emoji
import sys
import requests

configsPath = './src/Nethermind/Nethermind.Runner/configs'

key = sys.argv[1]

headers = {
'Content-type': 'application/json',
}

print(emoji.emojize("Fast Sync configuration settings initialization :white_check_mark: "))
CONFIGS_PATH = './src/Nethermind/Nethermind.Runner/configs'
APPLICATION_JSON = { 'Content-type': 'application/json' }
SUPERCHAIN_CHAINS = ["op-mainnet", "op-sepolia", "base-mainnet", "base-sepolia", "worldchain-mainnet", "worldchain-sepolia"]

configs = {
# fast sync section
Expand Down Expand Up @@ -97,6 +90,18 @@
"multiplierRequirement": 10000,
"isPoS": True
},
"worldchain-mainnet": {
"url": "https://worldchain-mainnet.g.alchemy.com/public",
"blockReduced": 8192,
"multiplierRequirement": 10000,
"isPoS": True
},
"worldchain-sepolia": {
"url": "https://worldchain-sepolia.g.alchemy.com/public",
"blockReduced": 8192,
"multiplierRequirement": 10000,
"isPoS": True
},
"linea-mainnet": {
"url": "https://rpc.linea.build",
"blockReduced": 8192,
Expand All @@ -113,22 +118,33 @@

def fastBlocksSettings(configuration, apiUrl, blockReduced, multiplierRequirement, isPoS):
if "etherscan" in apiUrl:
latestBlock = int(json.loads(subprocess.getoutput(
f'curl --silent "https://{apiUrl}/api?module=proxy&action=eth_blockNumber&apikey={key}"'))['result'], 16)
params = {
'module': 'proxy',
'action': 'eth_blockNumber',
'apikey': key,
}
response = requests.get(f'https://{apiUrl}/api', params=params)
latestBlock = int(json.loads(response.text)['result'], 16)
else:
data_req = '{"id":0,"jsonrpc":"2.0","method": "eth_blockNumber","params": []}'
response = requests.post(apiUrl, headers=headers, data=data_req).text
response = requests.post(apiUrl, headers=APPLICATION_JSON, data=data_req).text
latestBlock = int(json.loads(response)['result'], 16)

baseBlock = latestBlock - blockReduced
baseBlock = baseBlock - baseBlock % multiplierRequirement

if "etherscan" in apiUrl:
pivot = json.loads(subprocess.getoutput(
f'curl --silent "https://{apiUrl}/api?module=proxy&action=eth_getBlockByNumber&tag={hex(baseBlock)}&boolean=true&apikey={key}"'))
params = {
'module': 'proxy',
'action': 'eth_getBlockByNumber',
'tag': f'{hex(baseBlock)}',
'boolean': 'true',
'apikey': key,
}
pivot = json.loads(requests.get(f'https://{apiUrl}/api', params=params))
else:
data_req = '{"id":0,"jsonrpc":"2.0","method": "eth_getBlockByNumber","params": ["' + str(hex(baseBlock)) + '", false]}'
pivot = json.loads(requests.post(apiUrl, headers=headers, data=data_req).text)
data_req = f'{{"id":0,"jsonrpc":"2.0","method": "eth_getBlockByNumber","params": ["{hex(baseBlock)}", false]}}'
pivot = json.loads(requests.post(apiUrl, headers=APPLICATION_JSON, data=data_req).text)

pivotHash = pivot['result']['hash']
pivotTotalDifficulty = int(pivot['result'].get('totalDifficulty', '0x0'), 16)
Expand All @@ -139,18 +155,30 @@ def fastBlocksSettings(configuration, apiUrl, blockReduced, multiplierRequiremen
if not isPoS:
print(configuration + ' PivotTotalDifficulty: ' + str(pivotTotalDifficulty))

with open(f'{configsPath}/{configuration}.json', 'r') as mainnetCfg:
with open(f'{CONFIGS_PATH}/{configuration}.json', 'r') as mainnetCfg:
data = json.load(mainnetCfg)

data['Sync']['PivotNumber'] = baseBlock
data['Sync']['PivotHash'] = pivotHash

if not isPoS:
data['Sync']['PivotTotalDifficulty'] = str(pivotTotalDifficulty)

with open(f'{configsPath}/{configuration}.json', 'w') as mainnetCfgChanged:
with open(f'{CONFIGS_PATH}/{configuration}.json', 'w') as mainnetCfgChanged:
json.dump(data, mainnetCfgChanged, indent=2)

for config, value in configs.items():
print(emoji.emojize(f"{config.capitalize()} section :white_check_mark: "))
fastBlocksSettings(config, value['url'], value['blockReduced'], value['multiplierRequirement'], value['isPoS'])
if __name__ == "__main__":
parser = argparse.ArgumentParser(description="Fast Sync configuration settings")
parser.add_argument("-k", "--key", default="", help="etherscan API key")
parser.add_argument("--superchain", action="store_true", help="only process superchain chains")

args = parser.parse_args()
key = args.key

print(emoji.emojize("Fast Sync configuration settings initialization :white_check_mark: "))
for config, value in configs.items():
if args.superchain and config not in SUPERCHAIN_CHAINS:
continue

print(emoji.emojize(f"{config.capitalize()} section :white_check_mark: "))
fastBlocksSettings(config, value['url'], value['blockReduced'], value['multiplierRequirement'], value['isPoS'])
Binary file added src/Nethermind/Chains/arena-z-mainnet.json.zst
Binary file not shown.
Binary file added src/Nethermind/Chains/automata-mainnet.json.zst
Binary file not shown.
Loading
Loading