Dotty benchmarks

[felixmulder]

• Compilation speed
• Run speed
• Community build benchmarks
• Collection Strawman
• Graphs, numbers, website, such wow
• Get a stable environment, ask Fabien for a machine (where nothing's allowed to run on it concurrently)
• scala/compiler-benchmark for comparison with Scala 2.x (Compile benchmark with Dotty, fixes #29. compiler-benchmark#31)

[smarter]

See also https://github.com/scala/compiler-benchmark/ and the dotty port by @jvican : https://github.com/jvican/compiler-benchmark

[biboudis]

@jvican's port seems neat!

I prefer JMH but I also agree with @DarkDimius points in favor of ScalaMeter which Dotty currently uses. A small WIP from me is to enable JMH on dotty. For Scala the results are browsable with grafana here which seems super nice.

So for the next steps I would propose:

1. to decide on the microbenchmarking tool that we will adopt (its going to stick with us for a long time hopefully)
2. set up a linux-based, commodity machine that we can set up clock frequency and all that
3. set up grafana-based reporting
4. decide the dependency scheme (git submodule, sonatype?)
5. decide the triggering strategy (e.g., in compiler-benchq for scala)
6. pull in existing dotty benchmarks (@felixmulder already separated the benchmarks from tests Split benchmarks from regular run tests #2220)
7. pull in existing scalac benchmarks (@jvican's fork)
8. pull in existing collection-strawman benchmarks (and all those we are going to develop according to Improve test infrastructure #54)
9. clean up dotty/strawman of independent attempts and have a central point of reporting for our projects

[jvican]

For Scala the results are browsable with graphana here which seems super nice.

You just blew up my mind, didn't know that. It's really great.

I prefer JMH but I also agree with @DarkDimius points in favor of ScalaMeter which Dotty currently uses.

What are the arguments? I missed that discussion.

[biboudis]

What are the arguments? I missed that discussion.

This discussion was comprehensive regarding this scala/collection-strawman#19 (comment)

[biboudis]

A unified dotc/scalac suite is developed on olafurpg/compiler-benchmark#1

[felixmulder]

@biboudis - update on this? Do things then close plz.

[biboudis]

We are waiting scala/compiler-benchmark#31 to be merged.

[olafurpg]

What about https://github.com/liufengyun/bench @liufengyun?

[biboudis]

It is good to have both and testing scalac and dotc side by side, on the same stable infrastructure will increase the confidence on either of these! Consequently, it would be good!

[olafurpg]

scala/compiler-benchmark#31 is now merged. The fancy website and wow numbers for open PRs seem to be solved by Fengyun's awesome bench project, I think this issue can be closed.