Benchmarking

[Mossaka]

This issue is served as a place for discussing various ways / ideas we can benchmark runwasi the wasm shims that was proposed by @ipuustin

• One idea is that we can write a simple wasm program (Fibonacci) and execute in runwasi and a native program executing in runc. This provides a base benchmark of comparising the performance of wasi program vs. native runc processes. It is not meant to benchmark the performance of WASI in general.
• Having the base benchmark set, we can observe the performance difference for each version increments. For example, we can observe how much speed increase / descrease for version 0.2 vs. 0.3
• Another idea of benchmarking is testing how "dense" of wasm pods can we go for a node. It is often advertised that wasm modules can increase CPU utilization and thus increasing the density of running pods per node. We can verify this point to push the containerd runtime to the extreme by running thousands of pods at the same time.

Feel free to add ideas and thoughts on this topic! Any suggestion is welcome 🙏

• feat(bench): Re-enable cargo bench #612
  • Support benchmarking charts on GitHub Pages #620
  • Consider ways to optimize the benchmakring performance #621
  • feat(bench): Re-enable cargo bench #612
  • Enable benchmarking for WasmEdge #626
• add comparison to the normal containers #613
• Benchmark the start up time for each shim #614
• try running hunderds of containers at once to see how "dense" containers on a node would perform #615
  • Add crate for stress testing the shims #734
• reach out to the CNCF containerd maintainers and ask what managed services can we use for monitoring
• look into grafana and prometheous for monitoring the CPU, memory, disk, etc. usage of runwasi #616
• Benchmark the end-to-end wasi-demo-app workloads benches: add a new wasi-demo-{app, oci} benchmark #764
• Benchmark the shim performance on wasi-demo-app workloads with mocked Containerd) #771

[Mossaka]

Close this one as #126 adds benchmarking support.

[Mossaka]

Wanted to reopen this issue because I think #126 does not fully address the scope of the issue mentioned above. A few things that will make runwasi benchmarking stories better

• add benchmarks: add initial benchmarking support. #126 to the CI
• run runc and crun in the CI to give comparison results
• dense test

[Mossaka]

Check out SpinKube perf test: https://github.com/fermyon/spinkube-performance

TODO: look into grafana and prometheous options