feat: add timeouts to fuzz testing

[smartcontracts]

Adds --fuzz-timeout-secs to fuzz tests which will cause a property test to timeout after a certain number of seconds. Also adds --fuzz-allow-timeouts so that timeouts are optionally not considered to be failures.

First time writing Rust so I have no idea if this is the idiomatic way of doing things, happy to change anything based on suggestions! Would also be great to understand where the best place is to add tests for this sort of PR.

Keeping this as a draft for now. I want to add support for the same feature within invariant tests but I couldn't easily understand how FOUNDRY_INVARIANT_RUNS actually gets parsed from the environment. Apparently FOUNDRY_INVARIANT_RUNS can't be set in the CLI (there's no --invariant-runs flag). Any pointers here would be much appreciated.

This is sketch for #9393

[grandizzy]

Thank you! Supportive for such, let's have this

• as configs in foundry.toml

[fuzz]
timeout=60

which would be applied to all fuzz tests (I don't think the allow time is needed, just consider that if timeout is set then it implies we allow timeouts on test)

• inline, defined per test as:

/// forge-config: default.fuzz.timeout=60
function test_function_with_timeout(uint256 x) public {...}

• not sure if really needed to have them as global args but don't see any harm keeping them there too...

Same configs could be applied for invariants too, e.g.

[invariant]
timeout=60

and

/// forge-config: default.invariant.timeout=60

Lmk if you have issues implementing these, can provide more details / help with

[smartcontracts]

@grandizzy cool, will start making these changes! Regarding allowing timeouts, it might be worth considering that some people may want some way to detect that a fuzz run took too long and timed out. For now I would just allow timeouts by default and if someone asks for the feature we can add --reject-timeouts or whatever.

[smartcontracts]

Ok @grandizzy went ahead and made various changes. Please note the change to use Ok instead of returning an error. I think this is more accurate if we want to consider timeouts to not be failures by default. We could return an error if we want to add a --reject-timeouts flag in the future.

[grandizzy]

Ok @grandizzy went ahead and made various changes. Please note the change to use Ok instead of returning an error. I think this is more accurate if we want to consider timeouts to not be failures by default. We could return an error if we want to add a --reject-timeouts flag in the future.

the problem with Ok instead failing test is that for a config as

    /// forge-config: default.fuzz.runs = 4294967295
    /// forge-config: default.fuzz.timeout = 60
    function testFuzz_SetNumber(uint256 x) public {

the test still loops / consume cpu cycles for way more than 1 minute and don't think that's desired

[smartcontracts]

the test still loops / consume cpu cycles for way more than 1 minute and don't think that's desired

Ah yeah you're right, ok, will fix!

[smartcontracts]

@grandizzy I'm back to failing for timeouts, thanks for the tip there. I believe the PR works as-is, not exactly sure where I should be adding tests but happy to do so if you could point me to the test file I should modify!

[grandizzy]

@grandizzy I'm back to failing for timeouts, thanks for the tip there. I believe the PR works as-is, not exactly sure where I should be adding tests but happy to do so if you could point me to the test file I should modify!

Awesome, could you please make sure

cargo +nightly fmt -- --check
cargo +nightly clippy --all --all-targets --all-features -- -D warnings

are passing, CI is failing now. Re tests, let's add one similar with

foundry/crates/forge/tests/it/fuzz.rs

Lines 213 to 215 in 31dd1f7

	// tests that inline max-test-rejects config is properly applied
	forgetest_init!(test_inline_max_test_rejects, |prj, cmd| {
	prj.wipe_contracts();

that set inline config for a big number of runs and a timeout of 5 seconds and we assert test is passing. you can test it as

cargo test test_name_here

[grandizzy]

Last nit re invariant impl, let's fix clippy and format too

[grandizzy]

lgtm, thank you! I made couple of changes as

• move logic to interrupt invariant test in depth loop
• add and reuse start_timer fn and TEST_TIMEOUT constant
• add fuzz and invariant tests
• fix failing test
  Pending other reviews before merging

[smartcontracts]

@grandizzy sweet! Let me know if I can help in any way. Thanks for all your review here, very much appreciated 🙏

[mattsse]

I guess since this is opt-in, checking via elapsed for now is fine, and we can track improvements separately, but I'd suggest to wrap this if let Some into a helper type like Timeout(Option<Instant>) and an is_timed_out then this becomes easier to change

[grandizzy]

@mattsse I added FuzzTestTimer with da87095 please check

[DaniPopes]

lgtm

please re review