ci(perf): Track Firewood Performance via AvalancheGo Benchmarks #1493

Elvis339 · 2025-11-27T12:49:36Z

Why

Track C-Chain reexecution benchmark performance over time. Catch regressions before production.

Closes #1494

How

Firewood → triggers AvalancheGo benchmark → downloads results → publishes to GitHub Pages

Changes

scripts/bench-cchain-reexecution.sh
- Trigger AvalancheGo's C-Chain reexecution benchmark
- Poll for workflow registration
- Wait for completion and download artifacts
.github/workflows/track-performance.yml
- Orchestrate benchmark trigger via the script
- Publish results to GitHub Pages (main → bench/, branches → dev/bench/{branch}/)

Usage

# Auth
nix run ./ffi#gh -- auth login
export GH_TOKEN=$(nix run ./ffi#gh -- auth token)

# Predefined test
./scripts/bench-cchain-reexecution.sh trigger firewood-101-250k

# With specific Firewood version
FIREWOOD_REF=v0.0.18 ./scripts/bench-cchain-reexecution.sh trigger firewood-101-250k

# Custom block range
RUNNER=avalanche-avalanchego-runner-2ti \
FIREWOOD_REF=v0.0.18 \
CONFIG=firewood \
START_BLOCK=1 \
END_BLOCK=100 \
BLOCK_DIR_SRC=cchain-mainnet-blocks-200-ldb \
./scripts/bench-cchain-reexecution.sh trigger

# Other commands
./scripts/bench-cchain-reexecution.sh tests    # list available tests
./scripts/bench-cchain-reexecution.sh list     # list recent runs
./scripts/bench-cchain-reexecution.sh status <run_id>

Set FIREWOOD_REF=v0.0.18 explicitly. Without it, the workflow builds from HEAD, which currently fails due to changes in FFI layer

Added new benchmark script (bench-cchain-reexecution.sh) which
caused CI to fail with "Config does not cover the file". Shell
scripts aren't checked for license content (only .rs/.go/.h are),
but must be explicitly listed in the config. Exclude entire
scripts/ directory to avoid listing each script individually.

https://github.com/ava-labs/firewood/blob/main/.github/check-license-headers.yaml

rkuris

Early review, could use another pass

There's a lot of code here and I barely was able to complete the review in my maximum review time. Please consider breaking this up for more timely reviews, especially if the review is anything larger than this.

I'm also a little confused about how we track which firewood and avalanchego versions we ran the test against.

Let's say the performance loss was due to a change in avalanchego, how can we know that?

.github/workflows/track-performance.yml

Remove local developer tooling (justfile recipe, flake.nix, METRICS.md) to reduce PR scope. These will be submitted in a follow-up PR after the CI workflow changes are merged.

Elvis339 · 2026-01-26T18:24:07Z

Early review, could use another pass

There's a lot of code here and I barely was able to complete the review in my maximum review time. Please consider breaking this up for more timely reviews, especially if the review is anything larger than this.

I'm also a little confused about how we track which firewood and avalanchego versions we ran the test against.

Let's say the performance loss was due to a change in avalanchego, how can we know that?

I've split into two PRs per your feedback:

This PR (CI): Workflows, benchmark script, license config
#1642 (Local tooling): justfile recipe, METRICS.md docs, flake.nix

On version tracking github-action-benchmark stores time-series data (mgas/s over time) without version metadata tracing a regression requires digging through run history.

This is a conscious tradeoff to get data flowing now with minimal setup, then iterate to something more robust (S3 storage, richer metadata, export for analysis). Planning to revisit after ~2 weeks of data collection.

For now, each run logs Firewood / Avalanchego refs in the GitHub Actions summary, so the info exists just not queryable. Added your concern to the tracking doc as something to think through for the next iteration. How does that sound?

Summary example: https://github.com/ava-labs/firewood/actions/runs/21334552166

scripts/bench-cchain-reexecution.sh

Elvis339 · 2026-01-27T17:53:19Z

Executing:

RUNNER=avalanche-avalanchego-runner-2ti \
FIREWOOD_REF=v0.0.18 \
CONFIG=firewood \
START_BLOCK=1 \
END_BLOCK=100 \
BLOCK_DIR_SRC=cchain-mainnet-blocks-200-ldb \
./scripts/bench-cchain-reexecution.sh trigger

Triggered reexecution benchmark in AvalancheGo https://github.com/ava-labs/avalanchego/actions/runs/21408080471

scripts/bench-cchain-reexecution.sh

.github/workflows/track-performance.yml

…ith err and intentional exit code status

RodrigoVillar

Approving on the condition that #1493 (comment) be addressed

Elvis339 · 2026-01-28T17:23:48Z

Approving on the condition that #1493 (comment) be addressed

Thank you, it was addressed:

809c8ea
firewood/scripts/bench-cchain-reexecution.sh

Line 14 in d5027de

# help Show this help message

RodrigoVillar · 2026-01-28T17:27:11Z

@Elvis339 I meant this:

firewood/scripts/bench-cchain-reexecution.sh

Line 333 in 06c4d0a

help Show this help

Elvis339 added 3 commits November 27, 2025 16:39

ci: track performance

1ea99bb

test(ci): add PR label trigger for testing

1ce1481

Elvis339 added the run-benchmark label Nov 27, 2025

temp fix ci

8794250

Elvis339 removed the run-benchmark label Nov 27, 2025

Elvis339 changed the title ~~Es/enable firewood dev workflow~~ ci: Track Firewood Performance via AvalancheGo Benchmarks Nov 27, 2025

Elvis339 changed the title ~~ci: Track Firewood Performance via AvalancheGo Benchmarks~~ ci(perf): Track Firewood Performance via AvalancheGo Benchmarks Nov 27, 2025

Elvis339 marked this pull request as ready for review November 27, 2025 15:04

Elvis339 requested review from aaronbuchwald, demosdemon and rkuris as code owners November 27, 2025 15:04

Elvis339 mentioned this pull request Nov 27, 2025

Track Firewood Performance via AvalancheGo Reexecution Benchmarks #1494

Closed

Elvis339 marked this pull request as draft November 27, 2025 15:11

Elvis339 self-assigned this Dec 1, 2025

Elvis339 added 4 commits December 2, 2025 22:13

ci: use switch CI token

4aa0e08

ci: lint

32fd06f

fix

7a75021

docs

e35fa5d

Elvis339 marked this pull request as ready for review December 2, 2025 20:32

Merge branch 'main' into es/enable-firewood-dev-workflow

83a113a

RodrigoVillar reviewed Dec 2, 2025

View reviewed changes

README.md Show resolved Hide resolved

RodrigoVillar reviewed Dec 4, 2025

View reviewed changes

.github/workflows/track-performance.yml Show resolved Hide resolved

Elvis339 added 3 commits December 4, 2025 20:54

ci: push performance to benchmark data

18f4035

ci(perf): add benchmark workflow with nix-based just commands

73fc781

Merge branch 'es/enable-firewood-dev-workflow' of https://github.com/…

a19f1e8

…ava-labs/firewood into es/enable-firewood-dev-workflow

Elvis339 requested a review from alarso16 as a code owner December 4, 2025 17:17

docs

668cef3

RodrigoVillar requested changes Dec 4, 2025

View reviewed changes

METRICS.md Outdated Show resolved Hide resolved

refactor(bench-cchain-reexecution): rename verify_run_inputs to run_i…

84b381a

…nputs_match reducing cognitive load

Elvis339 requested a review from rkuris January 26, 2026 14:58

Elvis339 marked this pull request as ready for review January 26, 2026 16:39

Elvis339 and others added 3 commits January 26, 2026 20:39

Merge branch 'main' into es/enable-firewood-dev-workflow

a8db04d

chore: exclude all scripts from license header check

18d4047

Merge branch 'es/enable-firewood-dev-workflow' of https://github.com/…

79a2def

…ava-labs/firewood into es/enable-firewood-dev-workflow

Elvis339 commented Jan 26, 2026

View reviewed changes

rkuris requested changes Jan 26, 2026

View reviewed changes

.github/workflows/track-performance.yml Show resolved Hide resolved

.github/workflows/track-performance.yml Show resolved Hide resolved

.github/workflows/track-performance.yml Show resolved Hide resolved

.github/workflows/track-performance.yml Show resolved Hide resolved

Elvis339 mentioned this pull request Jan 26, 2026

chore(track-performance): local iteration #1642

Open

chore: split PR - extract local tooling to separate PR

cb0f6be

Remove local developer tooling (justfile recipe, flake.nix, METRICS.md) to reduce PR scope. These will be submitted in a follow-up PR after the CI workflow changes are merged.

RodrigoVillar reviewed Jan 26, 2026

View reviewed changes

scripts/bench-cchain-reexecution.sh Show resolved Hide resolved

Elvis339 requested review from RodrigoVillar and rkuris January 26, 2026 18:51

Elvis339 commented Jan 26, 2026

View reviewed changes

scripts/bench-cchain-reexecution.sh Show resolved Hide resolved

Elvis339 removed the DO NOT MERGE This PR is not meant to be merged in its current state label Jan 27, 2026

chore(bench-cchain-reexecution): replace hardcoded AvalancheGo branch

5ef9313

RodrigoVillar requested changes Jan 27, 2026

View reviewed changes

rkuris approved these changes Jan 27, 2026

View reviewed changes

.github/workflows/track-performance.yml Show resolved Hide resolved

.github/workflows/track-performance.yml Show resolved Hide resolved

.github/workflows/track-performance.yml Show resolved Hide resolved

Elvis339 and others added 2 commits January 28, 2026 11:38

docs: clarify benchmark workflow comments and simplify env var docs

809c8ea

Merge branch 'main' into es/enable-firewood-dev-workflow

2236393

Elvis339 requested a review from RodrigoVillar January 28, 2026 10:40

Elvis339 mentioned this pull request Jan 28, 2026

ci: add scheduled benchmark runs (#1639) #1645

Open

3 tasks

chore(bench-cchain-reexecution): remove die function and repalce it w…

d5027de

…ith err and intentional exit code status

RodrigoVillar approved these changes Jan 28, 2026

View reviewed changes

Elvis339 merged commit 06c4d0a into main Jan 28, 2026
51 checks passed

Elvis339 deleted the es/enable-firewood-dev-workflow branch January 28, 2026 17:24

                     "**/tests/compile_*/**",
                     "justfile",
-                    "scripts/run-just.sh",
+                    "scripts/**",

ci(perf): Track Firewood Performance via AvalancheGo Benchmarks #1493

ci(perf): Track Firewood Performance via AvalancheGo Benchmarks #1493

Conversation

Elvis339 commented Nov 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

How

Changes

Usage

Related

Uh oh!

Elvis339 commented Nov 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Elvis339 Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

rkuris left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Elvis339 commented Jan 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Elvis339 commented Jan 27, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

RodrigoVillar left a comment

Choose a reason for hiding this comment

Uh oh!

Elvis339 commented Jan 28, 2026

Uh oh!

Uh oh!

RodrigoVillar commented Jan 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Elvis339 commented Nov 27, 2025 •

edited

Loading

Elvis339 commented Nov 27, 2025 •

edited

Loading

rkuris left a comment •

edited

Loading

Elvis339 commented Jan 26, 2026 •

edited

Loading