Mini sbibm

[manuelgloeckler]

What does this implement/fix? Explain your changes

This is a draft for some "benchmarking" capabilities integrated into sbi.

With pytest, we can roughly check that everything works by passing all tests. Some tests will ensure that the overall methodology works "sufficiently" well on simplified Gaussian analytic examples. Certain changes might still pass all tests but, in the end, negatively impact the performance/accuracy.

Specifically, when implementing new methods or, e.g., changing default parameters, it is important to check that what was implemented not just only passes the tests but that it works sufficiently well.

Does this close any currently open issues?

Prototype for #1325

Any relevant code examples, logs, error output, etc?

So it now should work that one simply has to use:

pytest --bm 

Which is a custom tag that will disable testing and instead switch to a "benchmark" mode, which will only run tests that are marked as such and will always pass. Instead, these tests cache a metric on how well an implemented method solved a specific task (currently some examples in "bm_test.py").

Once it finishes, instead of passed/failed, it will return a table with the metric (we still can kinda color some methods that are worse than expected).

Any other comments?

• What tasks to incldue - clearly they must be somewhat "fast" to solve.
• What methods to incldue (i.e. just standard methods with default parameters or more)

[manuelgloeckler]

Alright, on the current examples, the output looks like this:

Runtime linearly increases with a number of train simulations (currently 2k ~ 10 min on my laptop, with 1k, it was like 5 min). It would maybe also be nice to print runtimes on the right.

Overall runtime, of course, also depends on how many different methods should be included. I think some limited control over what is run would be nice i.e

pytest --bm # All base inference classes on defaults (similar to current behavior)
pytest --bm=NPE   # NPE with e.g. different density estimators
pytest --bm=SNPE # SNPE_ABC  2 round test
...

Either way, there needs to be a limit on what is run, and every configuration should finish in a reasonable amount of time.

[manuelgloeckler]

Alright, it is kinda now ready for review. The overall "framework" is done, one from now go into a few directions, i.e., depending on what scope we want this to have. Current interface is

pytest --bm # Runs all major methods on default
pytest --bm --bm-mode=npe # Runs all major npe methods with e.g. different density estimators ...
pytest --bm --bm-mode=nre # Runs all major nre methods with different classfiers ...
pytest --bm --bm-mode=snpe # Runs all sequential NPE methods 2 rounds
pytest --bm --bm-mode=snle/snre # As above
pytest --bm --bm-mode=fmpe # Runs fmpe with different nets
pytest --bm --bm-mode=npse # Runs NPSE with different nets and others.

Not sure how much "configurability" we want this to have.

Tests cannot fail based on performance currently, but you will get a report which shows relative performance of each method on each task i.e. like this:

[janfb]

Overall, this is really great to have - thanks a lot for pushing this!
Love the relative coloring of the results 🎉

Added a couple of comments and questions for clarification.

[janfb]

so sequential methods are evaluated only on observation idx=1? and amortized methods on all observations and then c2st scores are averaged?

[manuelgloeckler]

Yes, currently the amortized methods are also only evaluated on the first 3 observations.

More evaluation adds more runtime. Not sure if we should do a full evaluation.

[janfb]

so GPU is required for running the benchmarks, why?

[manuelgloeckler]

This was a bug. Move it out of the if. It just should skip all bm tests in normal testing mode

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 78.44%. Comparing base (d3f22b5) to head (8ff30c1).
Report is 12 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #1335       +/-   ##
===========================================
- Coverage   89.40%   78.44%   -10.97%     
===========================================
  Files         118      118               
  Lines        8715     8779       +64     
===========================================
- Hits         7792     6887      -905     
- Misses        923     1892      +969     

Flag	Coverage Δ
unittests	78.44% <ø> (-10.97%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

see 42 files with indirect coverage changes