AutoTuner and CI sensitivity to small random changes in initial conditions

[oharboe]

My expectation is that if one adds AutoTuner results to git, then one should expect a flaky CI.

The problem is that AutoTuner results are right at the edge of the abyss: change any random initial condition by the tiniest amount and the flow will fail.

Specific case study: The-OpenROAD-Project/OpenROAD-flow-scripts#2735

Here we can see that PLACE_DENSITY 0.60 and CORE_UTILIZATION 40 for asap7/ethmac_lvt are right at the edge of grt failure.

The dip in DRC errors for grt at 0.63 is interesting: it is entirely conceivable that grt could work with some slightly different settings and that AutoTuner could find these conditions. That would result in a particularly nasty set of configuration parameters w.r.t. CI: I would expect any change at all to cause grt to fail.

This is also an example of how to use bazel-orfs to learn something. I wrote 140 lines of bazel + Python combined and I got the graph below. Each datapoint is ca. 1 hour to build, but the build for the 20 datapoints runs in parallel on my workstation, so I got the graph after lunch. Each of these datapoints is a build that can be accessed and examined through artifacts.

PLACE_DENSITY is an interesting challenge for AutoTuner. It is not obvious to me that a search algorithm can use the derivative of an objective function to search for the best PLACE_DENSITY here.

[oharboe]

I see that some AutoTuner tightening was rolled back in ORFS, so I guess that answers my question.

What AutoTuner outputs and what are good values for ORFS CI is not a trivialy copy and paste.