Restructuring a benchmark group breaks loading tuning parameters and judging

[danielmatz]

I use PkgBenchmark.jl to run my benchmarks in CI. I recently decided to decompose one of my benchmarks into a few benchmarks. So, I originally had an entry in a BenchmarkGroup that was a single benchmark, but the new version instead has a nested BenchmarkGroup for that entry. This caused MethodErrors to be thrown in the BenchmarkTools.jl code, both when loading the tuning parameters and when judging.

ERROR: MethodError: no method matching loadparams!(::BenchmarkTools.Benchmark, ::BenchmarkGroup, ::Symbol, ::Symbol)
Closest candidates are:
  loadparams!(::BenchmarkTools.Benchmark, !Matched::BenchmarkTools.Parameters, ::Any...)
   @ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/execution.jl:25
  loadparams!(!Matched::BenchmarkGroup, ::BenchmarkGroup, ::Any...)
   @ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:172
  loadparams!(!Matched::BenchmarkTools.Parameters, !Matched::BenchmarkTools.Parameters, ::Any...)
   @ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/parameters.jl:96
Stacktrace:
 [1] loadparams!(::BenchmarkGroup, ::BenchmarkGroup, ::Symbol, ::Vararg{Symbol}) (repeats 3 times)
   @ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:174
 [2] __runbenchmark_local(file::String, output::String, tunefile::String, retune::Bool, runoptions::@NamedTuple{verbose::Bool})
   @ PkgBenchmark /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/PkgBenchmark/15w1x/src/runbenchmark.jl:279
 [3] (::PkgBenchmark.var"#30#31"{String, String, String, Bool, @NamedTuple{verbose::Bool}})()
   @ PkgBenchmark /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/PkgBenchmark/15w1x/src/runbenchmark.jl:264
 [4] with_logstate(f::Function, logstate::Any)
   @ Base.CoreLogging ./logging.jl:515
 [5] with_logger
   @ ./logging.jl:627 [inlined]
 [6] _runbenchmark_local(file::String, output::String, tunefile::String, retune::Bool, runoptions::@NamedTuple{verbose::Bool}, logger_factory_path::Tuple{String, String, Symbol})
   @ PkgBenchmark /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/PkgBenchmark/15w1x/src/runbenchmark.jl:263
 [7] top-level scope
   @ none:2

ERROR: LoadError: MethodError: no method matching judge(::BenchmarkTools.BenchmarkGroup, ::BenchmarkTools.TrialEstimate)
Closest candidates are:
  judge(::BenchmarkTools.BenchmarkGroup, !Matched::BenchmarkTools.BenchmarkGroup...; kwargs...)
   @ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:128
  judge(!Matched::BenchmarkTools.TrialEstimate, ::BenchmarkTools.TrialEstimate; kwargs...)
   @ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/trials.jl:231
Stacktrace:
  [1] (::BenchmarkTools.var"#25#26"{@Kwargs{}})(::BenchmarkTools.BenchmarkGroup, ::Vararg{Any})
    @ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:129
  [2] mapvals!(f::Function, dest::BenchmarkTools.BenchmarkGroup, src::BenchmarkTools.BenchmarkGroup, srcs::BenchmarkTools.BenchmarkGroup)
    @ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:88
  [3] mapvals
    @ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:96 [inlined]
  [4] #judge#24
    @ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:129 [inlined]
  [5] judge(group::BenchmarkTools.BenchmarkGroup, groups::BenchmarkTools.BenchmarkGroup)
    @ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:128
  [6] (::BenchmarkTools.var"#25#26"{@Kwargs{}})(::BenchmarkTools.BenchmarkGroup, ::Vararg{BenchmarkTools.BenchmarkGroup})
    @ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:129
  [7] mapvals!(f::Function, dest::BenchmarkTools.BenchmarkGroup, src::BenchmarkTools.BenchmarkGroup, srcs::BenchmarkTools.BenchmarkGroup)
    @ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:88
  [8] mapvals
    @ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:96 [inlined]
  [9] #judge#24
    @ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:129 [inlined]
 [10] judge(group::BenchmarkTools.BenchmarkGroup, groups::BenchmarkTools.BenchmarkGroup)
    @ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:128
 [11] (::BenchmarkTools.var"#25#26"{@Kwargs{}})(::BenchmarkTools.BenchmarkGroup, ::Vararg{BenchmarkTools.BenchmarkGroup})
    @ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:129
 [12] mapvals!(f::Function, dest::BenchmarkTools.BenchmarkGroup, src::BenchmarkTools.BenchmarkGroup, srcs::BenchmarkTools.BenchmarkGroup)
    @ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:88
 [13] mapvals
    @ /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:96 [inlined]
 [14] #judge#24
    @ /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:129 [inlined]
 [15] judge
    @ /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:128 [inlined]
 [16] judge(target::BenchmarkResults, baseline::BenchmarkResults, f::typeof(minimum); judgekwargs::Dict{Any, Any})
    @ PkgBenchmark /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/PkgBenchmark/15w1x/src/judge.jl:50
 [17] judge
    @ PkgBenchmark /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/PkgBenchmark/15w1x/src/judge.jl:49 [inlined]
 [18] judge(target::BenchmarkResults, baseline::BenchmarkResults)
    @ PkgBenchmark /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/PkgBenchmark/15w1x/src/judge.jl:49
 [19] top-level scope
    @ /nobackup2/gitlab_runner/zDyhf_w4/0/Genesis/Genesis.jl/benchmark/run_ci.jl:24

Could the logic be made more robust to handle a restructuring of benchmarks, similar to what happens if a benchmark is added or removed?