Require explicit /evaluate for all PRs, add message to that effect

[danmoseley]

Require an explicit /evaluate command for all PRs instead of auto-evaluating on every push. This significantly reduces token consumption by only running evaluation when a maintainer explicitly requests it. Note, on opening a PR, it will post a message educating/reminding about /evaluate.

Changes

evaluation.yml — now schedule-only:

• Remove pull_request trigger entirely
• Remove evaluation-status check job (no longer needed for PRs)
• Simplify discover job to schedule-only code path

evaluation-fork-pr.yml — now handles all PRs (not just forks):

• Remove fork-only filter so pull_request_target posts status for all PRs
• Remove same-repo rejection in gate job so /evaluate works on upstream PRs too
• Add one-time reminder comment when a PR is opened with skill/test/infra changes
• Rename job and concurrency groups to reflect broader scope

How it works

1. When any PR is opened/synchronized, pull_request_target runs discovery from the base branch and posts an evaluation-status commit status:
   • pending if the PR changes skills, tests, or evaluation infrastructure — prompts for /evaluate
   • success if no evaluatable changes are detected
2. A maintainer posts /evaluate to trigger evaluation
3. No evaluation runs automatically on push — for any PR, fork or upstream

Important note — CI will still be red if evaluation didn't run on the most recent commit. That does not change, so /evaluate must be requested on the final commit (at least). It will go green if evaluate ran and completed.

[Copilot]

Pull request overview

Updates the evaluation GitHub Actions workflows to stop auto-evaluating on PR pushes and instead require a maintainer-triggered /evaluate, while keeping scheduled evaluations for the canonical repo.

Changes:

• Removed the pull_request trigger from evaluation.yml, leaving scheduled-only evaluation runs.
• Shifted PR status posting + /evaluate gating to evaluation-fork-pr.yml for all PRs (not just forks).
• Added an “open-time” reminder comment for PRs that change skills/tests.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 6 comments.

File	Description
.github/workflows/evaluation.yml	Makes evaluation scheduled-only; simplifies discovery and reusable workflow inputs.
.github/workflows/evaluation-fork-pr.yml	Expands PR handling to all PRs: posts evaluation-status, gates evaluation behind /evaluate, and comments a one-time reminder on open for skill/test changes.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Shifts evaluation to an explicit, maintainer-triggered /evaluate flow for pull requests, while keeping scheduled evaluations for the canonical repo to reduce automatic token usage on PR pushes.

Changes:

• Removed pull_request triggering from evaluation.yml, leaving it schedule-only with simplified discovery and fixed parallelization settings.
• Expanded evaluation-fork-pr.yml to handle all PRs via pull_request_target + issue_comment (/evaluate) gating.
• Added a one-time reminder comment on PR open when skill/test changes are detected.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
.github/workflows/evaluation.yml	Removes PR execution path; keeps scheduled evaluation in dotnet/skills and simplifies discovery/matrix inputs.
.github/workflows/evaluation-fork-pr.yml	Generalizes the PR evaluation gating/status workflow and adds reminder comment behavior.

Comments suppressed due to low confidence (1)

.github/workflows/evaluation-fork-pr.yml:54

• This step uses actions/checkout@v6 (mutable tag) and also omits persist-credentials: false, while most workflows in this repo pin actions to a commit SHA and disable persisted git credentials. Please pin actions/checkout to a specific commit SHA (with a version comment) and consider setting persist-credentials: false for consistency and to reduce token exposure risk.

      - name: Checkout base branch
        uses: actions/checkout@v6
        with:
          ref: ${{ github.event.pull_request.base.sha }}
          fetch-depth: 0

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Updates the evaluation automation so PRs no longer auto-run evaluation on every push; instead maintainers must explicitly trigger it via /evaluate, while scheduled evaluations continue to run periodically on the canonical repo.

Changes:

• Remove pull_request trigger and PR-specific logic/check job from evaluation.yml, making it schedule-only.
• Expand evaluation-fork-pr.yml to handle status posting and /evaluate-triggered runs for all PRs (fork and upstream).
• Add an “opened PR” reminder comment when evaluatable changes are detected.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
.github/workflows/evaluation.yml	Makes evaluation workflow schedule-only and simplifies discovery/evaluation inputs accordingly.
.github/workflows/evaluation-fork-pr.yml	Uses pull_request_target to post evaluation-status for all PRs and gates actual evaluation behind /evaluate, with a one-time reminder comment.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[ViktorHofer]

Pull_request_target or should only be used in forks. Otherwise we can never make changes to workflows or the skill-validator source code and respecting that in the run.

We also need to keep the evaluation-status job as that one is used in the rule settings and se as required for a PR.

I can help addressing the above points on Monday.

[danmoseley]

I don't fully understand that so happy for you to fix on Monday as offered!

[ViktorHofer]

Superseded by bd965cb