test(reexecute): add firewood chaos test [2/2] #4695

RodrigoVillar · 2025-12-15T18:39:33Z

Why this should be merged

As mentioned in ava-labs/firewood#1371, we need a crash test to make sure that Firewood is safe against application crashes. This PR fulfills this by adding the chaos executable, which does the following:

Starts the reexecution test as a child process which uses Firewood
After waiting for x amount of time, the executable kills the test
Starts the reexecution test again, picking up from where the child process crashed

The test is considered a success if the reexecution test is able to start up again.

Closes ava-labs/firewood#1588

How this works

Adds the tests/reexecute/chaos executable
Adds firewood chaos test workflow which is scheduled and can also be invoked via a manual dispatch
Extends reexecution script to add invocation of chaos test

How this was tested

CI

https://github.com/ava-labs/avalanchego/actions/runs/20284273056/job/58254137735?pr=4695

Need to be documented in RELEASES.md?

No

tests/reexecute/chaos/deps.go

RodrigoVillar · 2025-12-16T22:11:45Z

.github/workflows/firewood-chaos-test.yml

+  # XXX: remove this before merging
+  pull_request:


This is for testing the PR - will remove prior to merging.

Copilot

Pull request overview

This PR adds a chaos testing framework for Firewood to verify crash resilience during C-Chain block reexecution. The test simulates application crashes by forcefully killing the reexecution process, then verifies that Firewood can recover and resume from the persisted state.

Key Changes:

Implements a chaos test executable that spawns, kills, and restarts reexecution test processes
Adds GitHub Actions workflow for scheduled and manual chaos testing
Integrates chaos testing into the project's Taskfile build system

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/reexecute/chaos/main.go	Core chaos test implementation with process management and crash recovery logic
tests/reexecute/chaos/deps.go	VM initialization helpers for creating mainnet C-Chain VM instances
Taskfile.yml	Task definitions for running chaos tests with and without data copying
.github/workflows/chaos-test.yml	CI workflow for scheduled and manual chaos test execution
.github/workflows/chaos-test.json	Configuration matrix for chaos test parameters

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

tests/reexecute/chaos/main.go

ci: add chaos test job chore: nits chore: add nix installation step chore: configure AWS credentials chore: remove inputs chore: add perms chore: nit chore: nit chore: extend wait time chore: nits chore: nit chore: nit chore: Create shared `evm` module (#4690) chore(reexecute/c): remove go bench from benchmark (#4640) chore: nits fix: MAX_WAIT_TIME chore: nit chore: extend wait times chore: log errs chore: stdout tail chore: nit chore: nits chore: nits ci: improve workflow chore: nits chore: nits chore: nits chore: lint

maru-ava

Apart from the implementation considerations mentioned inline, I'm ok merging what you've proposed provided its scope remains limited. Fault injection will be a priority post-monorepo and tools like antithesis and chaos mesh provide a much richer set of capabilities that I would rather not see us attempt to duplicate.

maru-ava · 2025-12-17T20:06:05Z

Taskfile.yml

    cmds:
      - cmd: bash -x ./scripts/copy_dir.sh {{.SRC}} {{.DST}}

+  firewood-chaos-test:


As per the example of #4761, var indirection as proposed below is best avoided. Instead, please prefer passing args directly so that any changes to the inputs don't require task modification:

test-firewood-chaos: desc: ... cmd: go run ./tests/reexecute/chaos {{.CLI_ARGS}}

Followed the example of #4761 and moved the logic of test-firewood-chaos into it's own script: 00a4852

maru-ava · 2025-12-17T20:12:03Z

.github/workflows/chaos-test.json

+    "pull_request": {
+        "include": [
+            {
+                "start-block": "101",


Also as per the example of #4761, please ensure that configuration required to run a test locally is defined outside of github actions - ideally execution would be something like task run-test test-name. That could mean that only test, runner and timeout-minutes would be defined in this file, and given that both the runner and the timeout are static, the value of this file would seem questionable.

Also went ahead and removed the JSON file as suggested: 9b24d42

.github/workflows/firewood-chaos-test.yml

RodrigoVillar · 2025-12-18T19:48:47Z

Will wait for #4761 to be merged in before making any of the requested changes here.

scripts/tests.firewood_chaos.sh

.github/workflows/firewood-chaos-test.yml

scripts/benchmark_cchain_range.sh

RodrigoVillar self-assigned this Dec 15, 2025

RodrigoVillar added the reexecution label Dec 15, 2025

github-project-automation bot added this to avalanchego Dec 15, 2025

RodrigoVillar force-pushed the rodrigo/firewood-chaos-test branch from c68a289 to e1cf68f Compare December 16, 2025 22:05

RodrigoVillar commented Dec 16, 2025

View reviewed changes

RodrigoVillar requested a review from rkuris December 16, 2025 22:12

RodrigoVillar marked this pull request as ready for review December 16, 2025 22:23

RodrigoVillar requested a review from a team as a code owner December 16, 2025 22:23

Copilot AI review requested due to automatic review settings December 16, 2025 22:23

Copilot AI reviewed Dec 16, 2025

View reviewed changes

tests/reexecute/chaos/main.go Outdated Show resolved Hide resolved

tests/reexecute/chaos/main.go Show resolved Hide resolved

rkuris reviewed Dec 16, 2025

View reviewed changes

tests/reexecute/chaos/main.go Outdated Show resolved Hide resolved

tests/reexecute/chaos/main.go Outdated Show resolved Hide resolved

RodrigoVillar added 6 commits December 17, 2025 10:14

refactor(reexecute): export NewMainnetCChainVM()

8971646

chore: exponential => linear

5c7ec5c

chore: add timeout when waiting for killed process to terminate

9a893da

chore: ctx

4e1b5ac

chore: rebase nits

7b6d057

RodrigoVillar force-pushed the rodrigo/firewood-chaos-test branch from 27869b7 to 7b6d057 Compare December 17, 2025 15:19

RodrigoVillar requested a review from a team as a code owner December 17, 2025 15:19

RodrigoVillar changed the base branch from master to rodrigo/export-new-mainnet-c-chain-vm December 17, 2025 15:20

chore: remove unnecessary diff

c8e8582

RodrigoVillar changed the title ~~test(reexecute): add firewood chaos test~~ test(reexecute): add firewood chaos test [2/2] Dec 17, 2025

RodrigoVillar mentioned this pull request Dec 17, 2025

refactor(reexecute): export NewMainnetCChainVM() [1/2] #4758

Merged

chore: remove deps.go

506a2e3

rkuris approved these changes Dec 17, 2025

View reviewed changes

RodrigoVillar requested a review from maru-ava December 17, 2025 17:52

Base automatically changed from rodrigo/export-new-mainnet-c-chain-vm to master December 17, 2025 18:09

Merge branch 'master' into rodrigo/firewood-chaos-test

004bc43

maru-ava mentioned this pull request Dec 17, 2025

[ci] Migrate reexec tasks to a script to simplify maintenance #4761

Merged

1 task

maru-ava reviewed Dec 17, 2025

View reviewed changes

chore: nit

d989910

RodrigoVillar added 6 commits January 6, 2026 08:08

Merge branch 'master' into rodrigo/firewood-chaos-test

fd0d1e0

refactor!: replace task logic with script

00a4852

chore: add archival test

8b17166

fix: EOF

8a239e6

fix: config

c84015d

refactor: remove JSON

9b24d42

maru-ava reviewed Jan 6, 2026

View reviewed changes

scripts/tests.firewood_chaos.sh Outdated Show resolved Hide resolved

.github/workflows/firewood-chaos-test.yml Show resolved Hide resolved

RodrigoVillar added 5 commits January 6, 2026 10:39

fix: config (again)

f1c7cf0

chore: rename chaos test yml

ef82e6a

refactor!: merge chaos script into reexecution script"

c793199

Merge branch 'master' into rodrigo/firewood-chaos-test

439dd59

chore: nits

b472805

RodrigoVillar requested a review from maru-ava January 6, 2026 16:27

maru-ava reviewed Jan 6, 2026

View reviewed changes

RodrigoVillar added 2 commits January 6, 2026 12:12

refactor!: address PR comments

219b24a

chore: defined tests set chaos defaults

59581a8

maru-ava reviewed Jan 6, 2026

View reviewed changes

scripts/benchmark_cchain_range.sh Show resolved Hide resolved

scripts/benchmark_cchain_range.sh Show resolved Hide resolved

RodrigoVillar added 4 commits January 6, 2026 13:33

refactor: simplify chaos test validation conditional

c29a101

doc: make clear that chaos test works only with Firewood

ada0690

Merge branch 'master' into rodrigo/firewood-chaos-test

6464738

chore: license

4740b85

maru-ava approved these changes Jan 6, 2026

View reviewed changes

maru-ava added this pull request to the merge queue Jan 6, 2026

Merged via the queue into master with commit 5bb3329 Jan 6, 2026
56 checks passed

maru-ava deleted the rodrigo/firewood-chaos-test branch January 6, 2026 20:21

github-project-automation bot moved this to Done 🎉 in avalanchego Jan 6, 2026

RodrigoVillar mentioned this pull request Jan 6, 2026

chore(chaos): remove chaos test from PR CI #4817

Merged

test(reexecute): add firewood chaos test [2/2] #4695

test(reexecute): add firewood chaos test [2/2] #4695

Uh oh!

Conversation

RodrigoVillar commented Dec 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why this should be merged

How this works

How this was tested

Need to be documented in RELEASES.md?

Uh oh!

Uh oh!

RodrigoVillar Dec 16, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

maru-ava left a comment

Choose a reason for hiding this comment

Uh oh!

maru-ava Dec 17, 2025

Choose a reason for hiding this comment

Uh oh!

RodrigoVillar Jan 6, 2026

Choose a reason for hiding this comment

Uh oh!

maru-ava Dec 17, 2025

Choose a reason for hiding this comment

Uh oh!

RodrigoVillar Jan 6, 2026

Choose a reason for hiding this comment

Uh oh!

RodrigoVillar Jan 6, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

RodrigoVillar commented Dec 18, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

RodrigoVillar commented Dec 15, 2025 •

edited

Loading