Idris 2 automated benchmarks

Goal: To accurately measure the impact of changes on the Idris compiler against its exisiting ecosystem of libraries.

Here are all the kinds of benchmarks we want to run:

• benchmarks checking the speed of the compiler at compiling things
  • Parsing speed - How fast we parse
  • Typechecking speed - How fast we typecheck parsed trees
  • Codegen speed - How fast we process typechecked code
• benchmarks checking the speed of the programs generated by the compiler
  • With startup
  • Without startup

What we do right now

We pull all packages from the latest pack collection, install them and measure how long it takes to typecheck them. After that, we also run the tests on each of them to measure the performance of compiled code.

In the current state we only have a coarse view of performance where compiler stages are not separated and startup time is not removed from the measurement.

Left to do

• Obtain data for each compiler stages during typechecking
  • Not sure of the extra logging will degrate performance in a way that makes measurement meaningless
• Measure runtime performance without startup time
  • Requires new compiler features
• Run all the benchmarks on a dedicated server in a controlled environement
• Make the benchmarks part of CI

Existing benchmark suites

• idris2 self benchmark suite https://github.com/idris-lang/Idris2/tree/main/benchmark
• smallTT benchmarks https://github.com/AndrasKovacs/smalltt
• idris1 benchmarks https://github.com/ziman/idris-benchmarks (would require porting!)