Skip to content

ci: run only the eval cases a PR's changes reach - #29

Merged
jimisola merged 1 commit into
mainfrom
ci/plugin-evals-changed-cases
Oct 12, 2026
Merged

jimisola merged 1 commit into
mainfrom
ci/plugin-evals-changed-cases

Conversation

@jimisola

@jimisola jimisola commented Oct 12, 2026 •

Copy link
Copy Markdown
Owner

What & Why

To cut CI eval cost on narrow PRs, a PR now runs only the cases its changes reach, not every case of every changed plugin:

  • a change under plugins/<p>/skills/<skill>/ runs every case in plugins/<p>/evals/<skill>/;
  • a change under plugins/<p>/evals/<skill>/<case>/ runs that case;
  • a change to the workflow itself, or a manual run, runs every case of the chosen plugins (as before).

The Pick plugins step becomes Pick cases and outputs case directories; the run loop iterates over them. Stacked on #28 (-j 3), which touches the same loop.

Effect on past PRs

PR Change Cases before Cases now
#23 merge-ready SKILL.md + graders 14 6
#24 one grader 14 1
#20 bambu-3mf script 5 3
#21 new cad-tools cases 5 5

Test Plan

@jimisola jimisola self-assigned this Oct 12, 2026
@jimisola
jimisola added this pull request to stack #30 October 12, 2026 17:37
A PR now runs the cases of each skill it changes plus each case it
changes, instead of every case of every changed plugin. A change to the
workflow, or a manual run, still runs every case of the chosen plugins.

Signed-off-by: Jimisola Laursen <jimisola@jimisola.com>
@jimisola
jimisola force-pushed the ci/plugin-evals-changed-cases branch from 1de253a to e54da92 Compare October 12, 2026 18:02
@jimisola
jimisola removed this pull request from stack #30 October 12, 2026 18:06
@jimisola
jimisola changed the base branch from ci/plugin-evals-concurrency to main October 12, 2026 18:06
@jimisola
jimisola merged commit 23fa794 into main Oct 12, 2026
@jimisola
jimisola deleted the ci/plugin-evals-changed-cases branch October 12, 2026 18:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant