ci job control theory

[k-rister]

My only concern is that we are adding more code to control what runs and what doesn't. Ideally, CI jobs should be all controlled by configuration files with minimum code.

Originally posted by @rafaelfolco in #169 (review)

[k-rister]

This is an interesting topic to discuss about how this all works.

What we currently have is a series of GitHub actions and workflows that combine together to run a bunch of tests. The actions inspect the code to generate lists of "things" (userenvs, benchmarks, tools, endpoints, etc.) that combine to form a test using the matrix approach.

Without much intelligence this matrix can yield very large number of scenarios that need to be tested. The change that is being made in the referenced PR is about modifying one of the actions (get-userenvs) to have added intelligence so that it can make the list of userenvs it is generating more targeted so that the tests that end up being performed are more appropriate in scope to the actual PR that is being submitted -- ie. including testing that is needed and skipping testing that is not.

That logic / intelligence cannot be embedded in a config file per se. The action has to look at the changes made by the submitted PR and determine what it should be returning. In a lot of situations that is an extraordinarily complex task that is very difficult to codify (because it would take analyzing the code in the PR and understanding what the repercussions of the changs are) and so we don't even bother.

In the case of the referenced PR, all we need to do is look at what directories the proposed changes are being made in and from that we can decide if we can limit the scope of userenv testing. If we can we reduce the number of tests (and therefor time) that is required to get appropriate test coverage this makes the CI more resource efficient and helps us move faster.

So given that, how would we control/do all this with only configuration files? The workflows that call these actions have input parameters that are passed to the actions so some of this is config driven, but ultimately adding more code that "knows more" about what is happening is going to lead to more efficient testing.

[k-rister]

Some improvements related to this are in the pipeline: #172

[k-rister]

Also related:

• https://github.com/perftool-incubator/rickshaw/blob/master/util/generate-ci-jobs.py
• https://github.com/perftool-incubator/rickshaw/blob/master/util/crucible-ci.json
• https://github.com/perftool-incubator/rickshaw/blob/master/schema/crucible-ci.json