-
Notifications
You must be signed in to change notification settings - Fork 25
ci(perf): Track Firewood Performance via AvalancheGo Benchmarks #1493
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
25140e2
1ea99bb
1ce1481
8794250
4aa0e08
32fd06f
7a75021
e35fa5d
83a113a
18f4035
73fc781
a19f1e8
668cef3
6e65816
1e2bdd3
01ee24f
19d921e
1131a1d
5e32e9d
8008264
f2cf502
39c2050
ffcb333
02e7e94
7b16833
625049e
bb2c97e
c84dc10
37f1325
33e6921
268c05f
7055e9c
8b92cc6
98c9d00
2b128a8
a3966e6
65224f8
25cde31
e19e951
2b144f0
fb0a9cc
bfc7c8a
7d12619
09e9f46
c7cad22
84b381a
a8db04d
18d4047
79a2def
cb0f6be
5ef9313
809c8ea
2236393
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -58,7 +58,7 @@ | |
| "clippy.toml", | ||
| "**/tests/compile_*/**", | ||
| "justfile", | ||
| "scripts/run-just.sh", | ||
| "scripts/**", | ||
| ], | ||
| } | ||
| ] | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -30,6 +30,32 @@ jobs: | |
| run: | | ||
| cp -rv target/doc/* ./_site | ||
| cp -rv docs/assets ./_site | ||
| # GitHub Pages deploys from a single source (overwrites, doesn't merge). | ||
| # Benchmark history lives on benchmark-data branch (for append-only storage). | ||
| # We merge both into _site/ so a single deployment serves docs + benchmarks. | ||
| # See track-performance.yml for how benchmark data is collected and stored. | ||
| # | ||
| # Structure on benchmark-data branch (see track-performance.yml for how this is populated): | ||
| # bench/ - Official benchmark history (main branch only) | ||
| # dev/bench/{branch}/ - Feature branch benchmarks (experimental) | ||
| - name: Include benchmark data | ||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Build working: https://github.com/ava-labs/firewood/actions/runs/21334822723/job/61405063779 |
||
| run: | | ||
| # Fetch benchmark-data branch (may not exist on first run) | ||
| if ! git fetch origin benchmark-data 2>/dev/null; then | ||
| echo "No benchmark-data branch yet, skipping" | ||
| exit 0 | ||
| fi | ||
|
|
||
| # Extract benchmark directories from benchmark-data branch into current dir | ||
| # These may not exist if no benchmarks have been run yet for that category | ||
| git checkout origin/benchmark-data -- dev 2>/dev/null || echo "No dev/ directory (no feature branch benchmarks yet)" | ||
| git checkout origin/benchmark-data -- bench 2>/dev/null || echo "No bench/ directory (no main branch benchmarks yet)" | ||
|
|
||
| # Copy to _site - at least one must exist | ||
| [[ -d dev ]] && cp -rv dev _site/ | ||
| [[ -d bench ]] && cp -rv bench _site/ | ||
|
|
||
| [[ -d dev || -d bench ]] || { echo "::error::No benchmark data (dev/ or bench/) found"; exit 1; } | ||
| - uses: actions/upload-artifact@v4 | ||
| with: | ||
| name: pages | ||
|
|
||
RodrigoVillar marked this conversation as resolved.
Show resolved
Hide resolved
|
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,143 @@ | ||||||
| # Triggers AvalancheGo's C-Chain reexecution benchmark and publishes | ||||||
| # results to GitHub Pages for trend analysis. | ||||||
| name: C-Chain Reexecution Performance Tracking | ||||||
|
|
||||||
| on: | ||||||
| workflow_dispatch: | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Why is this manually dispatched? Shouldn't this happen automatically on pushes to main, or is this just an intermediate step for testing purposes? If the latter, please change the PR description to show the followup or create another PR/task and link it.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It's intermediate step for testing purposes. I created new issue for follow-up: #1639 and update PR description. |
||||||
| inputs: | ||||||
| firewood: | ||||||
| description: 'Firewood commit/branch/tag to test (leave empty to use the commit that triggered the workflow)' | ||||||
| default: '' | ||||||
| libevm: | ||||||
| description: 'libevm commit/branch/tag to test (leave empty to skip)' | ||||||
| default: '' | ||||||
Elvis339 marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
| avalanchego: | ||||||
| description: 'AvalancheGo commit/branch/tag to test against' | ||||||
| default: 'master' | ||||||
| test: | ||||||
| description: 'Predefined test (leave empty to use custom parameters below)' # https://github.com/ava-labs/avalanchego/blob/a85295d87193b30ff17c594680dadd6618022f5e/scripts/benchmark_cchain_range.sh#L63 | ||||||
| default: '' | ||||||
| config: | ||||||
| description: 'Config (e.g., firewood, hashdb)' | ||||||
| default: '' | ||||||
| start-block: | ||||||
| default: '' | ||||||
| end-block: | ||||||
| default: '' | ||||||
| block-dir-src: | ||||||
| description: 'Block directory source (e.g., cchain-mainnet-blocks-1m-ldb [without S3 path])' | ||||||
| default: '' | ||||||
| current-state-dir-src: | ||||||
| description: 'Current state directory source (e.g., cchain-mainnet-blocks-30m-40m-ldb [without S3 path])' | ||||||
| default: '' | ||||||
| runner: | ||||||
| description: 'Runner to use in AvalancheGo' | ||||||
| required: true | ||||||
| type: choice | ||||||
| options: | ||||||
| - avalanche-avalanchego-runner-2ti | ||||||
| - avago-runner-i4i-4xlarge-local-ssd | ||||||
| - avago-runner-m6i-4xlarge-ebs-fast | ||||||
| timeout-minutes: | ||||||
| description: 'Timeout in minutes' | ||||||
| default: '' | ||||||
|
|
||||||
| jobs: | ||||||
| record-benchmark-to-gh-pages: | ||||||
| runs-on: ubuntu-latest | ||||||
| permissions: | ||||||
| contents: write # Required for github-action-benchmark to push to gh-pages | ||||||
Elvis339 marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
| steps: | ||||||
| # NOTE: This checkout is only to get the bench-cchain-reexecution.sh script. | ||||||
| # We're not building or testing Firewood here—the script triggers AvalancheGo's | ||||||
| # workflow via API and passes FIREWOOD_REF to it. AvalancheGo is responsible | ||||||
| # for checking out and building Firewood at that ref. | ||||||
| - name: Checkout Firewood | ||||||
| uses: actions/checkout@v4 | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Doesn't this always check out the default branch? Shouldn't it be parameterized with something like: Otherwise I think this checks out main, and then the script later switches it to a different branch, which is less efficient and perhaps confusing.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The checkout here is only to get the So the checkout ref doesn't affect which Firewood version gets benchmarked that's controlled by
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Let's add some comments explaining this to avoid confusion later. In particular we should mention that the only reason to check this out is to get the script. One interesting side effect of this decision is that if you want to update the script, it won't use the new one until it lands in main I think.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Added clarifying comment to void confusion later.
Since the checkout doesn't specify a ref, it uses whichever branch you trigger from for manual runs. So you can test script changes by triggering the workflow from your feature branch. For scheduled runs (coming later), it will use main - which is the expected behavior for automated tracking. |
||||||
|
|
||||||
| - name: Trigger C-Chain Reexecution Benchmark | ||||||
| run: | | ||||||
| if [[ -n "${{ inputs.test }}" ]]; then | ||||||
| ./scripts/bench-cchain-reexecution.sh trigger "${{ inputs.test }}" | ||||||
| else | ||||||
| ./scripts/bench-cchain-reexecution.sh trigger | ||||||
| fi | ||||||
| env: | ||||||
| GH_TOKEN: ${{ secrets.FIREWOOD_AVALANCHEGO_GITHUB_TOKEN }} | ||||||
| # Custom mode (ignored when test is specified) | ||||||
| CONFIG: ${{ inputs.config }} | ||||||
| START_BLOCK: ${{ inputs.start-block }} | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It seems like if you set START_BLOCK you better also be setting CURRENT_STATE_DIR_SRC to let it know where to get the bootstrap database, is that correct? If so, we should verify that either neither is provided or both are.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Validation: firewood/scripts/bench-cchain-reexecution.sh Line 219 in cb0f6be
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If you want to start from genesis, then you either must set START_BLOCK to 0 (1?) or not set it. So, if START_BLOCK is not 0, then CURRENT_STATE_DIR_SRC must be set. Isn't that correct?
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Close,
|
||||||
| END_BLOCK: ${{ inputs.end-block }} | ||||||
| BLOCK_DIR_SRC: ${{ inputs.block-dir-src }} | ||||||
| CURRENT_STATE_DIR_SRC: ${{ inputs.current-state-dir-src }} | ||||||
| # Refs | ||||||
| FIREWOOD_REF: ${{ inputs.firewood || github.sha }} | ||||||
| AVALANCHEGO_REF: ${{ inputs.avalanchego }} | ||||||
| LIBEVM_REF: ${{ inputs.libevm }} | ||||||
| # Execution | ||||||
| RUNNER: ${{ inputs.runner }} | ||||||
| TIMEOUT_MINUTES: ${{ inputs.timeout-minutes }} | ||||||
|
|
||||||
| # github.ref controls where results are stored (not what gets benchmarked): | ||||||
| # - main branch → bench/ (official history) | ||||||
| # - feature branches → dev/bench/{branch}/ (experimental, won't pollute trends) | ||||||
| # inputs.firewood controls what gets benchmarked (passed to AvalancheGo). | ||||||
| - name: Determine results location | ||||||
| id: location | ||||||
| run: | | ||||||
| if [[ "${{ github.ref }}" == "refs/heads/main" ]]; then | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is this correct? I'm not following if Shouldn't this be doing a
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
The logic is:
This prevents a feature branch from accidentally (or intentionally) writing to official history by setting inputs.firewood=main.
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. A comment here would be helpful as it's not obvious.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. There was a comment:
|
||||||
| echo "data-dir=bench" >> "$GITHUB_OUTPUT" | ||||||
| else | ||||||
| echo "data-dir=dev/bench/$(echo '${{ github.ref_name }}' | tr '/' '-')" >> "$GITHUB_OUTPUT" | ||||||
| fi | ||||||
|
|
||||||
| - name: Publish benchmark results | ||||||
| id: store | ||||||
| uses: benchmark-action/github-action-benchmark@v1 | ||||||
| with: | ||||||
| name: C-Chain Reexecution with Firewood | ||||||
| tool: 'customBiggerIsBetter' | ||||||
| output-file-path: ./results/benchmark-output.json | ||||||
| github-token: ${{ secrets.GITHUB_TOKEN }} | ||||||
| summary-always: true | ||||||
| auto-push: true | ||||||
| fail-on-alert: true | ||||||
| comment-on-alert: false | ||||||
| gh-pages-branch: benchmark-data | ||||||
| benchmark-data-dir-path: ${{ steps.location.outputs.data-dir }} | ||||||
|
|
||||||
| - name: Summary | ||||||
| run: | | ||||||
| if [ "${{ steps.store.outcome }}" == "failure" ]; then | ||||||
| echo "::warning::Benchmark storage failed - results were not saved to GitHub Pages" | ||||||
Elvis339 marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
| fi | ||||||
|
|
||||||
| { | ||||||
| echo "## Firewood Performance Benchmark Results" | ||||||
| echo | ||||||
| echo "**Configuration:**" | ||||||
|
|
||||||
| if [ -n "${{ inputs.test }}" ]; then | ||||||
| echo "- Mode: Predefined test" | ||||||
| echo "- Test: \`${{ inputs.test }}\`" | ||||||
| else | ||||||
| echo "- Mode: Custom parameters" | ||||||
| echo "- Config: \`${{ inputs.config }}\`" | ||||||
| echo "- Blocks: \`${{ inputs.start-block }}\` → \`${{ inputs.end-block }}\`" | ||||||
| echo "- Block source: \`${{ inputs.block-dir-src }}\`" | ||||||
| echo "- State source: \`${{ inputs.current-state-dir-src }}\`" | ||||||
| fi | ||||||
|
|
||||||
| echo "- Firewood: \`${{ inputs.firewood || github.sha }}\`" | ||||||
| if [ -n "${{ inputs.libevm }}" ]; then | ||||||
| echo "- libevm: \`${{ inputs.libevm }}\`" | ||||||
| fi | ||||||
| echo "- AvalancheGo: \`${{ inputs.avalanchego }}\`" | ||||||
| echo "- Runner: \`${{ inputs.runner }}\`" | ||||||
| echo "- Timeout: \`${{ inputs.timeout-minutes }}\` minutes" | ||||||
| echo | ||||||
|
|
||||||
| echo "**Links:**" | ||||||
| echo "- [Performance Trends](https://ava-labs.github.io/firewood/${{ steps.location.outputs.data-dir }}/)" | ||||||
| } >> $GITHUB_STEP_SUMMARY | ||||||
|
|
||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added new benchmark script (bench-cchain-reexecution.sh) which
caused CI to fail with "Config does not cover the file". Shell
scripts aren't checked for license content (only .rs/.go/.h are),
but must be explicitly listed in the config. Exclude entire
scripts/ directory to avoid listing each script individually.
https://github.com/ava-labs/firewood/blob/main/.github/check-license-headers.yaml