ci(perf): Track Firewood Performance via AvalancheGo Benchmarks #1493

Elvis339 · 2025-11-27T12:49:36Z

Why

Track C-Chain reexecution benchmark performance over time. Catch regressions before production.

Closes #1494

How

Firewood → triggers AvalancheGo benchmark → downloads results → publishes to GitHub Pages

Changes

scripts/bench-cchain-reexecution.sh - trigger, wait, download
.github/workflows/track-performance.yml - orchestrates benchmark + publish
justfile - bench, bench-status, bench-list, bench-help

Usage

just bench firewood-101-250k
FIREWOOD_REF=v0.1.0 just bench firewood-33m-40m
just bench-status <run_id>

This can hang if it never finishes, which is possible if someone introduces some infinite loop somewhere.

I think github will let it run for a day, which is way too long. 2h seems like plenty but perhaps a shorter time makes sense.

Suggested change

nix run ./ffi#gh -- run watch "{{ run_id }}" --repo ava-labs/avalanchego --exit-status

nix run ./ffi#gh -- timeout -k 1m 2h run watch "{{ run_id }}" --repo ava-labs/avalanchego --exit-status

firewood/justfile

Line 185 in 8008264

timeout -k 1m 2h nix run ./ffi#gh -- run watch "{{ run_id }}" --repo ava-labs/avalanchego --exit-status

rkuris · 2025-12-05T21:29:56Z

justfile

+    echo "  URL: https://github.com/ava-labs/avalanchego/actions/runs/$RUN_ID"
+    echo ""
+
+    echo "Waiting for benchmark completion..."


👍 This looks polished. I probably would have just thrown a "set -x" here and called it a day but I like this better.

Co-authored-by: rodrigo <77309055+RodrigoVillar@users.noreply.github.com> Signed-off-by: Elvis <43846394+Elvis339@users.noreply.github.com>

Co-authored-by: Ron Kuris <ron.kuris@avalabs.org> Signed-off-by: Elvis <43846394+Elvis339@users.noreply.github.com>

…ava-labs/firewood into es/enable-firewood-dev-workflow

Elvis339 · 2025-12-08T16:56:06Z

We need to get this merged first: ava-labs/avalanchego#4650 before merging this PR.

DO NOT MERGE.

rkuris · 2025-12-11T17:13:10Z

.github/workflows/gh-pages.yaml

          cp -rv docs/assets ./_site
+      - name: Include benchmark data
+        run: |
+          git fetch origin benchmark-data || true


why are we ignoring errors here?

rkuris · 2025-12-11T17:13:42Z

.github/workflows/gh-pages.yaml

+            git checkout origin/benchmark-data -- dev bench 2>/dev/null || true
+            cp -rv dev _site/ 2>/dev/null || true
+            cp -rv bench _site/ 2>/dev/null || true


same here. Seems like if these fail, the build should fail

rkuris · 2025-12-11T17:16:16Z

.github/workflows/track-performance.yml

+name: Track Performance
+
+on:
+  workflow_dispatch:


Why is this manually dispatched? Shouldn't this happen automatically on pushes to main, or is this just an intermediate step for testing purposes?

If the latter, please change the PR description to show the followup or create another PR/task and link it.

rkuris · 2025-12-11T17:16:37Z

.github/workflows/track-performance.yml

+          - c-chain-reexecution-firewood-101-250k
+          - c-chain-reexecution-firewood-33m-33m500k
+          - c-chain-reexecution-firewood-33m-40m


Why did you pick these particular ones?

rkuris · 2025-12-11T17:18:59Z

.github/workflows/track-performance.yml

+  c-chain-benchmark:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: write  # Required for github-action-benchmark to push to gh-pages


This comment was helpful as a good explanation of what this task does. IMO this would be better documented as a block comment or even better by changing the name from "c-chain-benchmark" to "record-bench-to-gh-pages".

rkuris · 2025-12-11T17:25:58Z

.github/workflows/track-performance.yml

+              -f runner="${{ inputs.runner }}" \
+              $LIBEVM_FLAG
+            
+            # Wait for GitHub API to register the workflow run (retry up to 3min)


Is this the same logic we use in just, or are we already out of sync with what just does?

rkuris · 2025-12-11T17:26:54Z

.github/workflows/track-performance.yml

+        id: download
+        shell: nix develop ./ffi --command bash {0}
+        run: |
+          just download-benchmark-results "${{ steps.trigger.outputs.run_id }}"


curious as to what happens if the benchmark fails to start at all

rkuris · 2025-12-11T17:27:28Z

.github/workflows/track-performance.yml

+      - name: Summary
+        run: |
+          if [ "${{ steps.store.outcome }}" == "failure" ]; then
+            echo "::warning::Benchmark storage failed - results were not saved to GitHub Pages"


looks like we keep going anyway. Is this expected?

Yes, this is intentional. The continue-on-error: true on the store step lets us still show the summary even if publishing to GitHub Pages fails.

The idea is: if the benchmark ran and we got results, we want to surface what was tested and provide links to the AvalancheGo run - even if we couldn't save to gh-pages. The warning ::warning::Benchmark storage failed... makes the failure visible without hiding the useful context.
Alternative would be failing the workflow, but then we lose the summary which makes debugging harder.

rkuris · 2025-12-11T17:33:20Z

.github/workflows/track-performance.yml

+          echo "## Firewood Performance Benchmark Results" >> $GITHUB_STEP_SUMMARY
+          echo "" >> $GITHUB_STEP_SUMMARY
+          echo "**Configuration:**" >> $GITHUB_STEP_SUMMARY
+          


Group all these together (probably with the ones below too):

Suggested change

echo "## Firewood Performance Benchmark Results" >> $GITHUB_STEP_SUMMARY

echo "" >> $GITHUB_STEP_SUMMARY

echo "**Configuration:**" >> $GITHUB_STEP_SUMMARY

{

echo "## Firewood Performance Benchmark Results"

echo

echo "**Configuration:**"

# ...

} > $GITHUB_STEP_SUMMARY

It's a little particular about spacing though. There must be a space after the opening { and a newline or a semicolon ; before the closing }.

rkuris · 2025-12-11T17:35:34Z

justfile

+
+    # Wait for GitHub to register the run (retry up to 3min)
+    for i in {1..18}; do
+        sleep 10


Can we consolidate this loop and reduce the possibility of these getting out of sync?

A great example might be lets assume we want to change this to check every 5 seconds. How many places have to be updated?

…stfile commands

- Add bench-cchain-reexecution.sh script to trigger and monitor AvalancheGo benchmarks - Add just commands: bench, bench-status, bench-list, bench-help - Add track-performance.yml workflow to publish results to GitHub Pages - Support predefined tests and custom block ranges via env vars

Elvis339 · 2026-01-22T17:54:00Z

justfile

+# just bench <test>                                    - predefined test
+# FIREWOOD_REF=v0.1.0 just bench firewood-101-250k     - with specific ref
+# Custom: START_BLOCK=101 END_BLOCK=250000 BLOCK_DIR_SRC=<s3> CURRENT_STATE_DIR_SRC=<s3> just bench
+bench test="":


just provides the golden path for developers to run benchmarks locally. Same commands work in CI.

Elvis339 · 2026-01-22T17:56:06Z

.github/workflows/track-performance.yml

+          fetch-depth: 0  # Needed for github-action-benchmark
+
+      - name: Trigger C-Chain Reexecution Benchmark
+        run: ./scripts/bench-cchain-reexecution.sh trigger "${{ inputs.test }}"


Not using just here it's a thin wrapper around this script. Avoids installing Nix/just in CI for no benefit.

Elvis339 added 3 commits November 27, 2025 16:39

ci: track performance

1ea99bb

test(ci): add PR label trigger for testing

1ce1481

Elvis339 added the run-benchmark label Nov 27, 2025

temp fix ci

8794250

Elvis339 removed the run-benchmark label Nov 27, 2025

Elvis339 changed the title ~~Es/enable firewood dev workflow~~ ci: Track Firewood Performance via AvalancheGo Benchmarks Nov 27, 2025

Elvis339 changed the title ~~ci: Track Firewood Performance via AvalancheGo Benchmarks~~ ci(perf): Track Firewood Performance via AvalancheGo Benchmarks Nov 27, 2025

Elvis339 marked this pull request as ready for review November 27, 2025 15:04

Elvis339 requested review from aaronbuchwald, demosdemon and rkuris as code owners November 27, 2025 15:04

Elvis339 mentioned this pull request Nov 27, 2025

Track Firewood Performance via AvalancheGo Reexecution Benchmarks #1494

Open

Elvis339 marked this pull request as draft November 27, 2025 15:11

Elvis339 self-assigned this Dec 1, 2025

Elvis339 added 4 commits December 2, 2025 22:13

ci: use switch CI token

4aa0e08

ci: lint

32fd06f

fix

7a75021

docs

e35fa5d

Elvis339 marked this pull request as ready for review December 2, 2025 20:32

Merge branch 'main' into es/enable-firewood-dev-workflow

83a113a

RodrigoVillar reviewed Dec 2, 2025

View reviewed changes

README.md Show resolved Hide resolved

RodrigoVillar reviewed Dec 4, 2025

View reviewed changes

.github/workflows/track-performance.yml Show resolved Hide resolved

Elvis339 added 3 commits December 4, 2025 20:54

ci: push performance to benchmark data

18f4035

ci(perf): add benchmark workflow with nix-based just commands

73fc781

Merge branch 'es/enable-firewood-dev-workflow' of https://github.com/…

a19f1e8

…ava-labs/firewood into es/enable-firewood-dev-workflow

Elvis339 requested a review from alarso16 as a code owner December 4, 2025 17:17

docs

668cef3

RodrigoVillar requested changes Dec 4, 2025

View reviewed changes

METRICS.md Outdated Show resolved Hide resolved

Elvis339 added 2 commits December 4, 2025 22:16

chore: remove "\n" that got URL encoded

6e65816

docs

1e2bdd3

RodrigoVillar reviewed Dec 5, 2025

View reviewed changes

rkuris requested changes Dec 5, 2025

View reviewed changes

Elvis339 and others added 9 commits December 8, 2025 18:56

ci(track-performance): remove if: always

01ee24f

address PR

19d921e

Update .github/workflows/track-performance.yml

1131a1d

Co-authored-by: rodrigo <77309055+RodrigoVillar@users.noreply.github.com> Signed-off-by: Elvis <43846394+Elvis339@users.noreply.github.com>

Update .github/workflows/track-performance.yml

5e32e9d

Co-authored-by: rodrigo <77309055+RodrigoVillar@users.noreply.github.com> Signed-off-by: Elvis <43846394+Elvis339@users.noreply.github.com>

Update justfile

8008264

Co-authored-by: Ron Kuris <ron.kuris@avalabs.org> Signed-off-by: Elvis <43846394+Elvis339@users.noreply.github.com>

fix(ci): replace sleep with retry loop, improve justfile commands

f2cf502

Merge branch 'es/enable-firewood-dev-workflow' of https://github.com/…

39c2050

…ava-labs/firewood into es/enable-firewood-dev-workflow

lint: descriptive link text

ffcb333

docs

02e7e94

Elvis339 requested review from RodrigoVillar and rkuris December 8, 2025 16:04

Elvis339 added the DO NOT MERGE This PR is not meant to be merged in its current state label Dec 8, 2025

Elvis339 and others added 2 commits December 8, 2025 20:56

Merge branch 'main' into es/enable-firewood-dev-workflow

7b16833

chore: fix syntax error for benchmark command

625049e

ava-labs deleted a comment from github-actions bot Dec 9, 2025

rkuris requested changes Dec 11, 2025

View reviewed changes

ci: update workflow for C-Chain reexecution benchmarks and improve ju…

bb2c97e

…stfile commands

Elvis339 marked this pull request as draft December 30, 2025 17:07

Elvis339 added 2 commits January 22, 2026 16:16

Merge branch 'main' into es/enable-firewood-dev-workflow

c84dc10

Elvis339 commented Jan 22, 2026

View reviewed changes

+                workflow_dispatch:
+                  inputs:
+                    firewood:
+                      description: 'Firewood commit/branch/tag to test (leave empty for current HEAD)'

+                        auto-push: true
+                        gh-pages-branch: benchmark-data
+                        benchmark-data-dir-path: ${{ steps.download.outputs.data-dir }}
+                        # Don't fail the workflow if there's an issue with benchmark storage

-    echo "Triggering reexecution benchmark in AvalancheGo..."
+    echo "Triggering reexecution benchmark in AvalancheGo with parameters:"
+    echo "  - Firewood: $FIREWOOD"
+    # ...

+                  echo "Triggering reexecution benchmark in AvalancheGo..."
+                  RUN_ID=$(just benchmark-trigger "$FIREWOOD" "{{ avalanchego }}" "{{ task }}" "{{ runner }}" "{{ libevm }}")
+                  echo "  Run ID: $RUN_ID"
+                  echo "  URL: https://github.com/ava-labs/avalanchego/actions/runs/$RUN_ID"

+              benchmark-wait run_id: check-nix
+                  #!/usr/bin/env bash
+                  set -euo pipefail
+                  nix run ./ffi#gh -- run watch "{{ run_id }}" --repo ava-labs/avalanchego --exit-status

+                        comment-on-alert: false
+                    - name: Summary
+                      if: always()

	echo " URL: https://github.com/ava-labs/avalanchego/actions/runs/$RUN_ID"
	echo " Monitor progress here: https://github.com/ava-labs/avalanchego/actions/runs/$RUN_ID"

	nix run ./ffi#gh -- run watch "{{ run_id }}" --repo ava-labs/avalanchego --exit-status
	nix run ./ffi#gh -- timeout -k 1m 2h run watch "{{ run_id }}" --repo ava-labs/avalanchego --exit-status

ci(perf): Track Firewood Performance via AvalancheGo Benchmarks #1493

Are you sure you want to change the base?

ci(perf): Track Firewood Performance via AvalancheGo Benchmarks #1493

Uh oh!

Conversation

Elvis339 commented Nov 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

How

Changes

Usage

Related

Uh oh!

Elvis339 commented Nov 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Elvis339 Dec 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Elvis339 commented Dec 8, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Elvis339 commented Nov 27, 2025 •

edited

Loading

Elvis339 commented Nov 27, 2025 •

edited

Loading

Elvis339 Dec 8, 2025 •

edited

Loading