Proposal - PyMC speed benchmark code

[drbenvincent]

It would be quite cool if someone could type pm.bench() and then a variety of PyMC models are run and the timings are returned back.

Why would this be good?

• It would allow speedups in PyMC/Aesara/Aeppl to be quantified and graphed over time
• Could compare across different hardware
• Could be used to identify areas for attention

[michaelosthege]

Can probably start from https://github.com/pymc-devs/pymc/tree/main/benchmarks

These benchmarks didn't get a lot of attention for a long time, but it would be great to get them updated.

A comparison of C/numba/JAX compiled models, and a comparison of PyMC vs. blackjax vs. aehmc NUTS would be great!

[ColCarroll]

Would it make more sense for this to be a function (maybe in pymc-experimental, or on its own?), rather than in the library namespace?

[aloctavodia]

I really like the idea, it would be nice if (eventually) we could output a report with a few figures and not just raw times. I agree with @ColCarroll it may be weird to have this on the library namespace, but having this in pymc-experimental is also weird. Maybe we can put this inside pm.util or this could be on its own repo/library pm-devs/pmbench?

[drbenvincent]

I like the idea of pm.util.bench().

[ricardoV94]

I think it makes sense as its own repo/package