[ci] Support running re-execution benchmark with arbitrary version of Firewood #4650
Conversation
There was a problem hiding this comment.
Pull request overview
This PR establishes infrastructure for tracking Firewood performance over time by enabling C-Chain reexecution benchmarks with custom Firewood builds. The workflow can be triggered from the Firewood repository with either published versions (for quick testing) or branch/commit references (for comprehensive testing with source builds).
Key changes:
- Adds a reusable workflow that accepts Firewood version/branch/commit as input
- Implements intelligent build strategy: uses
go getfor published versions, builds from source for branches/commits - Creates build script for compiling Firewood FFI from source using Nix
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
.github/workflows/c-chain-reexecution-benchmark-firewood.yml |
New workflow that orchestrates benchmark execution with custom Firewood builds and uploads results as artifacts |
graft/coreth/scripts/build_firewood.sh |
Shell script to clone, build, and optionally integrate Firewood FFI from source |
.github/workflows/c-chain-reexecution-benchmark-container.yml |
Removes container configuration (unrelated cleanup) |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…/avalanchego into es/enable-firewood-dev-workflow
…ution benchmarks Integrate firewood/libevm dependency overrides into existing workflows using polyrepo, eliminating the need for a separate firewood workflow. - Add LIBEVM_REF/FIREWOOD_REF to reexecute-cchain-range-with-copied-data - Update composite action and workflows to pass inputs - Remove redundant firewood workflow and build_firewood.sh script
…/avalanchego into es/enable-firewood-dev-workflow
…/avalanchego into es/enable-firewood-dev-workflow
Allow starting re-execution from genesis. Uses "genesis" sentinel value to bypass default state and create empty state directory.
…xecution task for discoverability and reducing magic string duplication
…/avalanchego into es/enable-firewood-dev-workflow
aaronbuchwald
left a comment
aaronbuchwald
left a comment
There was a problem hiding this comment.
I do not think that the re-execution benchmark action itself should be responsible for coupling firewood/libevm refs when any workflow invoking this can simply update the dependencies before invoking this action as a step. Separately, the action should not support a special case of archiving data just for Firewood.
These are simple pre- and post- execution steps, which can be easily added as a step before or after the action gets invoked within a workflow.
If Firewood wants to execute this separately, why not just define its own workflow to do so? If it wants to execute it within AvalancheGo, then we should just write the data to the default location using Firewood as the config.
…action inputs, add benchmark artifact upload step
…dencies for reexecution benchmarks
…nput handling for libevm, firewood refs, and refine artifact upload logic
Resolved offline. Root action now:
Dependency setup externalized to |
|
|
||
| setup-reexecution-deps: | ||
| desc: Setup custom deps for reexecution benchmarks. Use LIBEVM_REF and/or FIREWOOD_REF env vars. | ||
| cmds: |
There was a problem hiding this comment.
(No action required) Maybe update the polyrepo script to accept LIBEVM_REF and FIREWOOD_REF so that this task is just providing those env vars to the script?
| runs-on: ${{ matrix.runner }} | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| - name: Install Nix |
There was a problem hiding this comment.
Maybe make nix installation optional as well?
Also, what is the implication of calling this action twice (once here, once in the reexec action)?
| with: | ||
| name: benchmark-output-${{ inputs.run-id }}-${{ inputs.run-attempt }}-${{ inputs.job }}-${{ inputs.test }}-${{ inputs.runner_type }} | ||
| path: ${{ env.BENCHMARK_OUTPUT_FILE }} | ||
| retention-days: 30 |
There was a problem hiding this comment.
Maybe omit this? afaik 30 days is the default
| - name: Upload Benchmark Artifact | ||
| uses: actions/upload-artifact@v4 | ||
| with: | ||
| name: benchmark-output-${{ inputs.run-id }}-${{ inputs.run-attempt }}-${{ inputs.job }}-${{ inputs.test }}-${{ inputs.runner_type }} |
There was a problem hiding this comment.
What's the rational for embedding all these values in the name?
|
|
||
| First, set up dependencies using the `setup-reexecution-deps` task, then run the benchmark: | ||
|
|
||
| ```bash |
There was a problem hiding this comment.
(No action required) Maybe include a single example with both versions and document that both or either one can be provided? Same comment for the CI invocation examples.
| ./scripts/run_task.sh test-cchain-reexecution -- firewood-101-250k | ||
| ``` | ||
|
|
||
| ### How It Works |
There was a problem hiding this comment.
(No action required) Maybe focus this doc additions on usage rather than duplicating implementation details?
Why this should be merged
Enables Firewood to track performance over time by running C-Chain reexecution benchmarks with custom Firewood builds. This establishes the infrastructure for catching performance regressions before they reach production.
ava-labs/firewood#1494
How this works
with-dependenciesparametertask setup-reexecution-depsbefore calling the benchmark actionThe root action remains unaware of external dependency configuration - setup is handled by the workflow before invocation.
The same functionality is available locally for development:
Changes
setup-reexecution-depstask for dependency setup (externalized from action)How this was tested
gh workflow run "C-Chain Re-Execution Benchmark w/ Container"
--ref es/enable-firewood-dev-workflow
-f test=firewood-101-250k
-f with-dependencies="firewood=v0.0.15,libevm=v1.13.15-0.20251210210615-b8e76562a300"
-f runner=avalanche-avalanchego-runner-2ti
-f timeout-minutes=60
Need to be documented in RELEASES.md?
No