-
Notifications
You must be signed in to change notification settings - Fork 25
ci(perf): Track Firewood Performance via AvalancheGo Benchmarks #1493
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
25140e2
1ea99bb
1ce1481
8794250
4aa0e08
32fd06f
7a75021
e35fa5d
83a113a
18f4035
73fc781
a19f1e8
668cef3
6e65816
1e2bdd3
01ee24f
19d921e
1131a1d
5e32e9d
8008264
f2cf502
39c2050
ffcb333
02e7e94
7b16833
625049e
bb2c97e
c84dc10
37f1325
33e6921
268c05f
7055e9c
8b92cc6
98c9d00
2b128a8
a3966e6
65224f8
25cde31
e19e951
2b144f0
fb0a9cc
bfc7c8a
7d12619
09e9f46
c7cad22
84b381a
a8db04d
18d4047
79a2def
cb0f6be
5ef9313
809c8ea
2236393
d5027de
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -58,7 +58,7 @@ | |
| "clippy.toml", | ||
| "**/tests/compile_*/**", | ||
| "justfile", | ||
| "scripts/run-just.sh", | ||
| "scripts/**", | ||
| ], | ||
| } | ||
| ] | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -30,6 +30,32 @@ jobs: | |
| run: | | ||
| cp -rv target/doc/* ./_site | ||
| cp -rv docs/assets ./_site | ||
| # GitHub Pages deploys from a single source (overwrites, doesn't merge). | ||
| # Benchmark history lives on benchmark-data branch (for append-only storage). | ||
| # We merge both into _site/ so a single deployment serves docs + benchmarks. | ||
| # See track-performance.yml for how benchmark data is collected and stored. | ||
| # | ||
| # Structure on benchmark-data branch (see track-performance.yml for how this is populated): | ||
| # bench/ - Official benchmark history (main branch only) | ||
| # dev/bench/{branch}/ - Feature branch benchmarks (experimental) | ||
| - name: Include benchmark data | ||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Build working: https://github.com/ava-labs/firewood/actions/runs/21334822723/job/61405063779 |
||
| run: | | ||
| # Fetch benchmark-data branch (may not exist on first run) | ||
| if ! git fetch origin benchmark-data 2>/dev/null; then | ||
| echo "No benchmark-data branch yet, skipping" | ||
| exit 0 | ||
| fi | ||
|
|
||
| # Extract benchmark directories from benchmark-data branch into current dir | ||
| # These may not exist if no benchmarks have been run yet for that category | ||
| git checkout origin/benchmark-data -- dev 2>/dev/null || echo "No dev/ directory (no feature branch benchmarks yet)" | ||
| git checkout origin/benchmark-data -- bench 2>/dev/null || echo "No bench/ directory (no main branch benchmarks yet)" | ||
|
|
||
| # Copy to _site - at least one must exist | ||
| [[ -d dev ]] && cp -rv dev _site/ | ||
| [[ -d bench ]] && cp -rv bench _site/ | ||
|
|
||
| [[ -d dev || -d bench ]] || { echo "::error::No benchmark data (dev/ or bench/) found"; exit 1; } | ||
| - uses: actions/upload-artifact@v4 | ||
| with: | ||
| name: pages | ||
|
|
||
RodrigoVillar marked this conversation as resolved.
Show resolved
Hide resolved
|
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,143 @@ | ||
| # Triggers AvalancheGo's C-Chain reexecution benchmark and publishes | ||
| # results to GitHub Pages for trend analysis. | ||
| name: C-Chain Reexecution Performance Tracking | ||
|
|
||
| on: | ||
| workflow_dispatch: | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Why is this manually dispatched? Shouldn't this happen automatically on pushes to main, or is this just an intermediate step for testing purposes? If the latter, please change the PR description to show the followup or create another PR/task and link it.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It's intermediate step for testing purposes. I created new issue for follow-up: #1639 and update PR description. |
||
| inputs: | ||
| firewood: | ||
| description: 'Firewood commit/branch/tag to test (leave empty to use the commit that triggered the workflow)' | ||
| default: '' | ||
| libevm: | ||
| description: 'libevm commit/branch/tag to test (leave empty to skip)' | ||
| default: '' | ||
Elvis339 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| avalanchego: | ||
| description: 'AvalancheGo commit/branch/tag to test against' | ||
| default: 'master' | ||
| test: | ||
| description: 'Predefined test (leave empty to use custom parameters below)' # https://github.com/ava-labs/avalanchego/blob/a85295d87193b30ff17c594680dadd6618022f5e/scripts/benchmark_cchain_range.sh#L63 | ||
| default: '' | ||
| config: | ||
| description: 'Config (e.g., firewood, hashdb)' | ||
| default: '' | ||
| start-block: | ||
| default: '' | ||
| end-block: | ||
| default: '' | ||
| block-dir-src: | ||
| description: 'Block directory source (e.g., cchain-mainnet-blocks-1m-ldb [without S3 path])' | ||
| default: '' | ||
| current-state-dir-src: | ||
| description: 'Current state directory source (e.g., cchain-mainnet-blocks-30m-40m-ldb [without S3 path])' | ||
| default: '' | ||
| runner: | ||
| description: 'Runner to use in AvalancheGo' | ||
| required: true | ||
| type: choice | ||
| options: | ||
| - avalanche-avalanchego-runner-2ti | ||
| - avago-runner-i4i-4xlarge-local-ssd | ||
| - avago-runner-m6i-4xlarge-ebs-fast | ||
| timeout-minutes: | ||
| description: 'Timeout in minutes' | ||
| default: '' | ||
|
|
||
| jobs: | ||
| record-benchmark-to-gh-pages: | ||
| runs-on: ubuntu-latest | ||
| permissions: | ||
| contents: write # Required for github-action-benchmark to push to gh-pages | ||
Elvis339 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| steps: | ||
| # NOTE: This checkout is only to get the bench-cchain-reexecution.sh script. | ||
| # We're not building or testing Firewood here—the script triggers AvalancheGo's | ||
| # workflow via API and passes FIREWOOD_REF to it. AvalancheGo is responsible | ||
| # for checking out and building Firewood at that ref. | ||
| - name: Checkout Firewood | ||
| uses: actions/checkout@v4 | ||
Elvis339 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| - name: Trigger C-Chain Reexecution Benchmark | ||
| run: | | ||
| if [[ -n "${{ inputs.test }}" ]]; then | ||
| ./scripts/bench-cchain-reexecution.sh trigger "${{ inputs.test }}" | ||
| else | ||
| ./scripts/bench-cchain-reexecution.sh trigger | ||
| fi | ||
| env: | ||
| GH_TOKEN: ${{ secrets.FIREWOOD_AVALANCHEGO_GITHUB_TOKEN }} | ||
| # Custom mode (ignored when test is specified) | ||
| CONFIG: ${{ inputs.config }} | ||
| START_BLOCK: ${{ inputs.start-block }} | ||
Elvis339 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| END_BLOCK: ${{ inputs.end-block }} | ||
| BLOCK_DIR_SRC: ${{ inputs.block-dir-src }} | ||
| CURRENT_STATE_DIR_SRC: ${{ inputs.current-state-dir-src }} | ||
| # Refs | ||
| FIREWOOD_REF: ${{ inputs.firewood || github.sha }} | ||
| AVALANCHEGO_REF: ${{ inputs.avalanchego }} | ||
| LIBEVM_REF: ${{ inputs.libevm }} | ||
| # Execution | ||
| RUNNER: ${{ inputs.runner }} | ||
| TIMEOUT_MINUTES: ${{ inputs.timeout-minutes }} | ||
|
|
||
| # github.ref controls where results are stored (not what gets benchmarked): | ||
| # - main branch → bench/ (official history) | ||
| # - feature branches → dev/bench/{branch}/ (experimental, won't pollute trends) | ||
| # inputs.firewood controls what gets benchmarked (passed to AvalancheGo). | ||
| - name: Determine results location | ||
| id: location | ||
| run: | | ||
| if [[ "${{ github.ref }}" == "refs/heads/main" ]]; then | ||
Elvis339 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| echo "data-dir=bench" >> "$GITHUB_OUTPUT" | ||
| else | ||
| echo "data-dir=dev/bench/$(echo '${{ github.ref_name }}' | tr '/' '-')" >> "$GITHUB_OUTPUT" | ||
| fi | ||
|
|
||
| - name: Publish benchmark results | ||
| id: store | ||
| uses: benchmark-action/github-action-benchmark@v1 | ||
| with: | ||
| name: C-Chain Reexecution with Firewood | ||
| tool: 'customBiggerIsBetter' | ||
| output-file-path: ./results/benchmark-output.json | ||
| github-token: ${{ secrets.GITHUB_TOKEN }} | ||
| summary-always: true | ||
| auto-push: true | ||
| fail-on-alert: true | ||
| comment-on-alert: false | ||
| gh-pages-branch: benchmark-data | ||
| benchmark-data-dir-path: ${{ steps.location.outputs.data-dir }} | ||
|
|
||
| - name: Summary | ||
| run: | | ||
| if [ "${{ steps.store.outcome }}" == "failure" ]; then | ||
| echo "::warning::Benchmark storage failed - results were not saved to GitHub Pages" | ||
Elvis339 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| fi | ||
|
|
||
| { | ||
| echo "## Firewood Performance Benchmark Results" | ||
| echo | ||
| echo "**Configuration:**" | ||
|
|
||
| if [ -n "${{ inputs.test }}" ]; then | ||
| echo "- Mode: Predefined test" | ||
| echo "- Test: \`${{ inputs.test }}\`" | ||
| else | ||
| echo "- Mode: Custom parameters" | ||
| echo "- Config: \`${{ inputs.config }}\`" | ||
| echo "- Blocks: \`${{ inputs.start-block }}\` → \`${{ inputs.end-block }}\`" | ||
| echo "- Block source: \`${{ inputs.block-dir-src }}\`" | ||
| echo "- State source: \`${{ inputs.current-state-dir-src }}\`" | ||
| fi | ||
|
|
||
| echo "- Firewood: \`${{ inputs.firewood || github.sha }}\`" | ||
| if [ -n "${{ inputs.libevm }}" ]; then | ||
| echo "- libevm: \`${{ inputs.libevm }}\`" | ||
| fi | ||
| echo "- AvalancheGo: \`${{ inputs.avalanchego }}\`" | ||
| echo "- Runner: \`${{ inputs.runner }}\`" | ||
| echo "- Timeout: \`${{ inputs.timeout-minutes }}\` minutes" | ||
| echo | ||
|
|
||
| echo "**Links:**" | ||
| echo "- [Performance Trends](https://ava-labs.github.io/firewood/${{ steps.location.outputs.data-dir }}/)" | ||
| } >> $GITHUB_STEP_SUMMARY | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added new benchmark script (bench-cchain-reexecution.sh) which
caused CI to fail with "Config does not cover the file". Shell
scripts aren't checked for license content (only .rs/.go/.h are),
but must be explicitly listed in the config. Exclude entire
scripts/ directory to avoid listing each script individually.
https://github.com/ava-labs/firewood/blob/main/.github/check-license-headers.yaml