Evaluate Profile-Guided Optimization (PGO)

[zamazan4ik]

Hi!

Recently I checked Profile-Guided Optimization (PGO) improvements on multiple projects. The results are here. LLVM-related results are here.

Since PGO showed measurable improvements in compiler-like loads (CPython, Clang, Clangd, clang-format, GCC, Rustc, etc.) I think it could be useful to check PGO on OnnxRuntime as well.

We need to perform PGO benchmarks on OnnxRuntime. And if it shows improvements - add a note about possible improvements in OnnxRuntime performance with PGO. Providing an easier way (e.g. a build option) to build scripts with PGO can be useful for the end-users too. Maybe testing Post-Link Optimization techniques (like LLVM BOLT from Facebook) would be interesting too but I recommend starting from the usual PGO.