ci: add scheduled benchmark runs (#1639) #1645

Elvis339 · 2026-01-28T11:25:55Z

Why this should be merged

This patch is closing #1639 it adds automated benchmark triggers to catch performance regressions without manual intervention.

Daily (40M→41M): Quick 1M block test for fast regression detection
Weekly (50M→60M): Comprehensive 10M block stress test
Runner: avago-runner-i4i-4xlarge-local-ssd for consistent NVMe performance

Note: should be merged after #1493

How this works

JSON config (.github/benchmark-schedules.json): Defines benchmark parameters per schedule
resolve-config job: Uses actions/github-script to match cron → config
benchmark job: Matrix strategy triggers AvalancheGo's workflow, publishes to GitHub Pages

Manual dispatch still works via workflow inputs.

How this was tested

Verified S3 paths exist
Confirmed storage (~443 GB) fits i4i.4xlarge capacity (3.75 TB)
Manual workflow dispatch validation (pending)

RodrigoVillar

Could you please rebase this PR on top of #1642 (or whatever PR it's dependent on)? I'm not what's needed to be reviewed vs code that's being reviewed in a different PR.

RodrigoVillar · 2026-01-28T15:10:08Z

.github/workflows/track-performance.yml

+          summary-always: true
+          auto-push: true
+          fail-on-alert: true


q: what's the rationale for setting summary-always and fail-on-alert to true?

summary-always: true shows benchmark results in the job summary on every run with information such as versions of AvalancheGo, libevm, Firewood ...

fail-on-alert: true fails the workflow if a regression is detected (exceeds alert threshold). Ensures regressions don't go unnoticed

rkuris

Nit: Overall, this seems like a lot of somewhat-fragile code to parse json instead of just specifying the parameters in the normal github style.

I think we can save a ton of money by picking a much smaller instance size. I think i4i.xlarge will work at roughly 1/8 the cost.

rkuris · 2026-01-28T16:13:26Z

.github/benchmark-schedules.json

+      {
+        "name": "daily-40m-41m",
+        "cron": "0 5 * * *",
+        "runner": "avago-runner-i4i-4xlarge-local-ssd",


This instance type is likely wasting a ton of space and is double the cost of i4i-2xlarge and 4x the cost of i4i-xlarge. The latter has 937G of local disk. Let's lower the cost of this by using smaller instances.

Infra team opened Jira ticket for this, we can track it here: https://ava-labs.atlassian.net/browse/INFRA-6420

rkuris · 2026-01-28T16:17:38Z

.github/workflows/track-performance.yml

+        description: 'Firewood commit/branch/tag to test (leave empty to use the commit that triggered the workflow)'
+        default: ''
+      libevm:
+        description: 'libevm commit/branch/tag to test (leave empty to skip)'


What does 'skip' mean here? Does that mean use main/master?

rkuris · 2026-01-28T16:18:30Z

.github/workflows/track-performance.yml

+        default: ''
+      config:
+        description: 'Config (e.g., firewood, hashdb)'
+        default: ''


Should this be 'firewood'? What happens if none is specified in the json?

Why do we even need a default here? Same with many of the others. Is it required?

default: '' isn't strictly required omitting it has the same behavior (empty string when not provided). We use it for consistency with AvalancheGo's pattern.

Yes you're right in this context default value for config should be Firewood.

rkuris · 2026-01-28T16:23:20Z

.github/workflows/track-performance.yml

+            const fs = require('fs');
+            let matrix;
+
+            if (context.eventName === 'schedule') {


Is this true when fired from the scheduled cron job and false otherwise?

Yes, context.eventName returns the trigger type:

schedule when fired by cron

workflow_dispatch when triggered manually

push, pull_request, etc. for other triggers

Implements workflow to trigger C-Chain reexecution benchmarks in AvalancheGo and track Firewood performance over time. Supports task-based and custom parameter modes. Results stored in GitHub Pages via github-action-benchmark. # Conflicts: # .github/workflows/track-performance.yml

…stfile commands

- Refactor `bench-cchain` to trigger track-performance.yml instead of directly calling AvalancheGo's workflow - Add input validation and helpful error messages for test/custom params - Use commit SHA instead of branch name for reproducibility - Fix AvalancheGo workflow input: use `with-dependencies` format ("firewood=abc,libevm=xyz") instead of separate firewood-ref/libevm-ref - Remove status-cchain, list-cchain, help-cchain (use GitHub UI instead) - Remove debug logging from track-performance.yml

Remove local developer tooling (justfile recipe, flake.nix, METRICS.md) to reduce PR scope. These will be submitted in a follow-up PR after the CI workflow changes are merged.

Daily (40M→41M) and weekly (50M→60M) benchmarks with JSON-based config and matrix strategy for cleaner workflow management.

…coded branch name used for temp. testing

- Replace `avago-runner-i4i-4xlarge-local-ssd` with `avago-runner-i4i-2xlarge-local-ssd` for daily and weekly benchmarks. - Add new input options for `firewood` and `libevm` commits/branches/tags. - Extend runner choices with additional configurations in the workflow.

Elvis339 self-assigned this Jan 28, 2026

Elvis339 added the DO NOT MERGE This PR is not meant to be merged in its current state label Jan 28, 2026

Elvis339 requested review from demosdemon and rkuris as code owners January 28, 2026 11:25

Elvis339 added the performance label Jan 28, 2026

Elvis339 requested review from RodrigoVillar and aaronbuchwald as code owners January 28, 2026 11:25

Elvis339 added the c-chain label Jan 28, 2026

RodrigoVillar reviewed Jan 28, 2026

View reviewed changes

rkuris requested changes Jan 28, 2026

View reviewed changes

Elvis339 added 19 commits January 28, 2026 18:53

ci: track performance

d64e77d

docs

a55cb7d

ci(perf): add benchmark workflow with nix-based just commands

d82ed73

docs

8c42437

docs

04d6367

address PR

ac94798

lint: descriptive link text

126d564

docs

a261cdf

ci: update workflow for C-Chain reexecution benchmarks and improve ju…

3b95709

…stfile commands

docs

e7fec1b

debug

f56b9ac

temp

1406e14

docs

aa785b1

docs

119cc33

ci(gh-pages): temp. add workflow_dispatch to rebuild Pages

344bcf3

ci(gh-pages): remove temp. set workflow_dispatch

29ee34c

chore: split PR - extract local tooling to separate PR

1d648ea

Remove local developer tooling (justfile recipe, flake.nix, METRICS.md) to reduce PR scope. These will be submitted in a follow-up PR after the CI workflow changes are merged.

ci(track-performance): add scheduled C-Chain reexecution benchmarks

6ee9ccc

Daily (40M→41M) and weekly (50M→60M) benchmarks with JSON-based config and matrix strategy for cleaner workflow management.

Elvis339 force-pushed the es/scheduled-perf-tracking branch from 2af94f6 to 6ee9ccc Compare January 28, 2026 17:55

chore(bench-cchain-reexecution): use AVALANCHEGO_REF instead of hard-…

c5efed7

…coded branch name used for temp. testing

RodrigoVillar mentioned this pull request Jan 28, 2026

ci: add benchmark workflow #1643

Open

Elvis339 added 2 commits January 28, 2026 19:03

chore(bench-cchain-reexecution): clarify help text wording

5d39f47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci: add scheduled benchmark runs (#1639) #1645

ci: add scheduled benchmark runs (#1639) #1645

Elvis339 commented Jan 28, 2026 •

edited

Loading

Uh oh!

RodrigoVillar left a comment •

edited

Loading

Uh oh!

RodrigoVillar Jan 28, 2026

Uh oh!

Elvis339 Jan 28, 2026

Uh oh!

rkuris left a comment

Uh oh!

rkuris Jan 28, 2026

Uh oh!

Elvis339 Jan 28, 2026

Uh oh!

rkuris Jan 28, 2026

Uh oh!

rkuris Jan 28, 2026

Uh oh!

Elvis339 Jan 28, 2026

Uh oh!

rkuris Jan 28, 2026

Uh oh!

Elvis339 Jan 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ci: add scheduled benchmark runs (#1639) #1645

Are you sure you want to change the base?

ci: add scheduled benchmark runs (#1639) #1645

Conversation

Elvis339 commented Jan 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why this should be merged

How this works

How this was tested

Uh oh!

RodrigoVillar left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rkuris left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Elvis339 commented Jan 28, 2026 •

edited

Loading

RodrigoVillar left a comment •

edited

Loading