Reduce scheduled evaluation from every 3h to daily, skip if no changes - #362
Conversation
The evaluation schedule was consuming ~/5 days in Copilot premium requests from scheduled runs alone (~/month pace). This change: 1. Reduces the cron schedule from every 3 hours (8 runs/day) to once daily at 08:00 UTC. This alone saves ~87% of scheduled evaluation cost. 2. Adds a check-for-changes guard that skips the scheduled run entirely if there have been no new commits to main since the last successful scheduled evaluation. This avoids wasting tokens re-evaluating identical code. The PR trigger and all PR evaluation behavior is unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Adjusts the evaluation workflow schedule and adds an early-exit guard to reduce unnecessary scheduled evaluation runs (and associated token usage) when the repository hasn’t changed since the last successful scheduled evaluation.
Changes:
- Reduced the scheduled evaluation cron from every 3 hours to once daily (08:00 UTC).
- Added a scheduled-run guard step that checks the last successful scheduled run’s HEAD SHA and skips checkout/discovery when unchanged.
- Gated checkout and discovery steps on the guard result for scheduled runs.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Add actions: read permission so the workflow run lookup succeeds - Use status=completed with jq conclusion filter instead of status=success - Make all gh api calls non-fatal (fallback to has_changes=true on error) - Make compare API call best-effort for logging only Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Updates the scheduled evaluation workflow to reduce routine token spend by running less frequently and skipping scheduled runs when the repo hasn’t changed since the last successful scheduled evaluation.
Changes:
- Reduced cron schedule from every 3 hours to once daily (08:00 UTC).
- Added a pre-check on scheduled runs that compares the current
github.shato the last successful scheduled evaluation’shead_shaand skips the heavy steps when unchanged. - Added
actions: readpermission to support querying workflow runs via the GitHub API.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
Ironically this kicked off a full evaluation of everything... |
- Move actions:read from workflow-level to discover job permissions for least-privilege (only the schedule guard needs it) - Increase per_page from 10 to 100 so the skip guard finds the last successful run even after long failure streaks Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Updates the main evaluation workflow to reduce scheduled benchmark token usage by running less frequently and short-circuiting when the repository HEAD hasn’t changed since the last evaluation.
Changes:
- Reduce scheduled evaluation cron from every 3 hours to once daily (08:00 UTC).
- Add a scheduled-run “skip if no new commits” guard using the GitHub API via
gh. - Gate checkout and discovery steps on the guard result to avoid running evaluation/publishing when unchanged.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Only skip when the most recent completed scheduled run was successful AND at the same SHA. If it failed/cancelled at the current SHA, proceed to retry. This prevents the guard from skipping indefinitely after a transient failure. Also simplifies to per_page=1 since we only inspect the latest run. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Updates the scheduled evaluation workflow to reduce token usage by running less frequently and skipping scheduled evaluations when the repository SHA hasn’t changed since the last successful scheduled run.
Changes:
- Reduced the scheduled cron trigger from every 3 hours to once daily at 08:00 UTC.
- Added a schedule-only “skip if no changes” guard that checks the most recent completed scheduled run’s SHA/conclusion via GitHub’s API and conditionally skips checkout/discovery.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Use // empty to guard against null when workflow_runs array is empty, preventing 'null null' from leaking into LAST_SHA/LAST_CONCLUSION. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Updates the main evaluation workflow schedule and adds a pre-check to avoid running scheduled evaluations when the repository HEAD hasn’t changed since the last successful scheduled run, reducing token usage on quiet days.
Changes:
- Reduce scheduled evaluation cadence from every 3 hours to once daily (08:00 UTC).
- Add a “skip if no changes” guard for scheduled runs by querying the latest completed scheduled run and comparing SHA + conclusion.
- Scope
actions: readpermissions to thediscoverjob to support the GitHub API query with least privilege.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
When a maintainer clicks 'Re-run jobs' on a scheduled run, the event is still schedule with the same SHA. Without this check, the guard would skip the rerun if the previous attempt succeeded. Now reruns always proceed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
This PR reduces token usage from scheduled evaluations by running them once daily and skipping the scheduled evaluation when the last completed scheduled run already succeeded on the same commit SHA. This keeps PR-triggered evaluations unchanged while reducing redundant scheduled runs on quiet days.
Changes:
- Reduce the
evaluation.ymlschedule from every 3 hours to once daily (08:00 UTC). - Add a scheduled-only “skip if no changes” guard that queries GitHub Actions API for the latest completed scheduled run and skips when it already succeeded at the current SHA.
- Scope
actions:read/contents:readpermissions to thediscoverjob for least-privilege API access.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
I think we should run once daily even if there are no changes. We're probably going to have quiet days or even weeks in this repo soon and it would be good to catch any ambient changes that break the infrastructure. |
|
@adityamandaleeka in such cases, ideally we'd do 1 iteration not 5, right? edit: but then it would mess up your dashboard. OK for now I'll just change the cron timing. |
|
If you're suggesting a job that runs once a day to make sure the machinery works with 1 run (maybe even on a subset of evals) that seems reasonable. And we can keep this one the way you have it. |
07ddb7d to
0e5e995
Compare
danmoseley
changed the title
|
Oh nuts I already changed it how you said. Let me change it back |
|
haha either way works. |
0e5e995 to
07ddb7d
Compare
danmoseley
changed the title
There was a problem hiding this comment.
Pull request overview
Adjusts the evaluation GitHub Actions workflow scheduling to reduce token usage from scheduled benchmark runs, and adds logic to avoid running full scheduled evaluations when nothing has changed.
Changes:
- Reduce the scheduled cron trigger from every 3 hours to once daily at 08:00 UTC.
- Add a scheduled-run “skip guard” that avoids checkout/discovery (and therefore evaluation) when the last completed scheduled run at the same SHA succeeded.
- Restrict the
discoverjob token permissions to read-only (actions: read,contents: read).
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
We can add that later when things get quiet. For now this will run every day and we don't need yet more runs to confuse... |
Skill Validation Results
[1] (Isolated) Quality unchanged but weighted score is -10.0% due to: tokens (11732 → 89265), tool calls (0 → 5), time (10.2s → 39.1s)
Model: claude-opus-4.6 | Judge: claude-opus-4.6 |
|
@danmoseley we should always run, regardless of if there are changes. Changing to daily makes sense now with the increased quota. |
(This keeps 5 iterations to get "stable" data. If we only wanted to check infrastructure health we would do 1 iteration. I'm assuming we're looking at https://dotnet.github.io/skills, where all this goes, and it's important to have stable measurements there.)
Reduce scheduled evaluation frequency to save tokens
Problem
Scheduled evaluation runs are the dominant consumer of AI tokens. At 8 runs/day
(every 3 hours), each evaluating all plugins and scenarios with 5 iterations,
the schedule generates a very large number of Copilot sessions per day.
Additionally, on quiet days with no commits, every scheduled run re-evaluates
identical code, producing no new signal while consuming the same tokens.
Changes
Cron reduced from every 3h to once daily (08:00 UTC). This saves ~87%
of scheduled evaluation token consumption while still providing daily
benchmark data on active days.
Skip-if-no-changes guard: Before checkout/discovery, scheduled runs
query the GitHub API for the most recent completed scheduled evaluation run.
The guard only skips when that run was successful and at the same SHA
as the current run. If the last run failed or was cancelled, the evaluation
proceeds (retry). This means on days with no new commits and a prior
successful run, the evaluation is skipped entirely (no new signal to capture).
Hardening:
actions: readscoped to thediscoverjob only (least-privilege)Impact