Default parameter option `evals_set` is not documented

[pjaap]

Hi there,

maybe I miss something completely, but I do not understand the meaning of samples and evals for the benchmarks.

If I run this code in REPL

using BenchmarkTools
BenchmarkTools.DEFAULT_PARAMETERS.samples = 1
BenchmarkTools.DEFAULT_PARAMETERS.evals = 1

foo_calls = 0

function foo()
  global foo_calls += 1
end

b = @benchmark foo()

@show foo_calls

b

it should, as I understood the documentation, evaluate foo once. However, it produces

foo_calls = 501494
BenchmarkTools.Trial: 1 sample with 993 evaluations.
 Single result which took 37.263 ns (0.00% GC) to evaluate,
 with a memory estimate of 16 bytes, over 1 allocations.

Why does it evaluate foo 993 times while it was called in reality 501494 times?

[pjaap]

Ok, I debugged the problem in the source code.
Control flow does not pass the hasevals(params) check since no explicit evals parameter is passed to the @benchmark call. This invokes a tune! that guesses the number of evals.

I can resolve my problem by passing evals=1 directly to the @benchmark call.

However, this makes the BenchmarkTools.DEFAULT_PARAMETERS.evals = 1 option obsolete and this should be documented somewhere.

[pjaap]

I found out from reading the source code that we can force the global default to be taken by adding BenchmarkTools.DEFAULT_PARAMETERS.evals_set=true. I.e., the following options will evaluate the function foo only once:

BenchmarkTools.DEFAULT_PARAMETERS.samples = 1
BenchmarkTools.DEFAULT_PARAMETERS.evals = 1
BenchmarkTools.DEFAULT_PARAMETERS.evals_set = true

The option BenchmarkTools.DEFAULT_PARAMETERS.evals_set is not documented.