Should bf-diags option be built in all pre-commit CI tests?

[jafingerhut]

I am right now confirming whether the build succeeds when you enable bf-diags, but if it does, it appears that it cannot be enabled at the same time as both Tofino1 and Tofino2 architectures.

Do we think it is important to double the number of CI builds per supported OS, so that all three of Tofino1, Tofino2, and bf-diags builds are exercised on every commit?

Or perhaps bf-diags should be a CI test triggered by adding an optional label to a PR, so it is only run when we want to verify bf-diags still builds?

[vgurevich]

@jafingerhut -- I would say that it is important to ensure that bf-diags works, period.

Having said that, there is difference between CI/CD and regression testing, in a sense that usually people run a small subset of critical sanity tests before allowing the code to be checked in and then they run regressions separately.

I think that building bf-diags for one target is a good and important "sanity" check.

At the same time, we might need to think how to do regression testing on the project, especially since it takes quite a bit of resources. I would also ask Intel whether they can share their regression testing code/frameworks that are not a part of this repo.

[jafingerhut]

Ryan Goodfellow at Oxide Computer verbally mentioned the possibility of Oxide lending us some hardware resources for testing purposes of this code, in case that is any easier to achieve. I have no concrete ideas on how to mix the code of this repo and use the portions of it that work on a real hardware system (at least the bf-p4c compiler output, maybe bf_switch code, too) with the portions of the driver software required by a real board.