Run smoke evaluations for Vally updates

[AbhitejJohn]

Summary

• Treat eng/evaluation-tools/** changes as evaluation infrastructure and select two random eligible plugins.
• Evaluate the exact PR package manifests while keeping workflow, adapter, and validator code on the trusted workflow commit.
• Distinguish a skipped evaluation from a completed evaluation in PR-triage messaging.
• Document the Vally dependency smoke path and its trust boundary.

Validation

• 12 targeted workflow regression tests passed.
• Shell syntax, eval-quality checks, and actionlint passed.
• Live Vally 0.13.0 smoke run: https://github.com/dotnet/skills/actions/runs/32174524180
  • selected dotnet-data and dotnet-maui
  • ran both plugins with claude-sonnet-4.6 and gpt-5.6-luna
  • all four matrix jobs completed successfully

Follow-up

Vally 0.13.0 declares npm >=11.11.1, while the current runner provides npm 10.9.8. The live run completed despite the EBADENGINE warning; npm toolchain alignment is intentionally separate from this automation and messaging change.

[Copilot]

Pull request overview

Updates the evaluation and triage infrastructure to treat eng/evaluation-tools/** (Vally dependency/tooling manifests) as evaluation-infrastructure changes, running a two-plugin smoke evaluation that installs the PR’s evaluation-tools manifests while keeping executable workflow/adapter/validator code pinned to the trusted workflow commit. Also refines PR-triage messaging to distinguish “no evaluation required” from “evaluation passed”.

Changes:

• Adds an evaluation_tools_ref signal from evaluation.yml to the reusable evaluation-run.yml so evaluation-tools PRs can install eng/evaluation-tools package manifests from the PR commit while keeping trusted tooling pinned.
• Improves PR triage messaging by reading the evaluation-status description (not just state) to differentiate skipped/no-op evaluations from completed ones.
• Documents the smoke-evaluation trust boundary and behavior in both the Vally adapter guide and legacy skill-validator docs, plus the triage workflow design doc.

Show a summary per file

File	Description
eng/vally-adapter/InvestigatingResults.md	Documents the evaluation-tools smoke path and trust boundary for Vally harness runs.
eng/skill-validator/src/docs/InvestigatingResults.md	Adds the same evaluation-tools smoke note for historical/legacy results documentation.
eng/evaluation/test_token_failover.py	Adds regression tests asserting the new evaluation_tools_ref wiring and triage “skipped vs passed” messaging behavior.
docs/design/pr-triage-workflows.md	Documents triage behavior and evaluation-tools smoke evaluation semantics.
.github/workflows/evaluation.yml	Treats eng/evaluation-tools/** as infra; emits evaluation_tools_ref when applicable and forwards it to the reusable workflow.
.github/workflows/evaluation-run.yml	Adds evaluation_tools_ref input, checks out PR manifests for eng/evaluation-tools, and installs manifests from the correct source.
.github/scripts/pr-triage-act.sh	Reads evaluation status state+description and adjusts maintainer ping text accordingly.

Review details

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

• Files reviewed: 7/7 changed files
• Comments generated: 0
• Review effort level: Lite

[y87feng]

Could we avoid installing PR-controlled manifests in a job that later runs the installed Copilot/Vally binaries with a real Copilot PAT? Previously these dependencies came from the trusted workflow commit; this change makes them PR-controlled, so a modified dependency could receive or exfiltrate the selected token when executed later.

Could we keep PAT-backed execution on trusted dependencies, or run this PR dependency smoke test without reusable secrets?

Suggested by Copilot.

[github-actions]

👋 @AbhitejJohn — this PR has 1 unresolved review thread(s). When you're ready, please address the feedback and push an update; the triage bot will pick up the next state automatically. (Add the no-stale label to silence further pings.)

[AbhitejJohn]

Thanks for catching this. We are closing this change because it would let a PR-selected Vally binary run with a reusable Copilot PAT. Although the path is restricted to maintainer-authorized same-repository evaluations, this expands the executable surface that can directly access the token.

We want to keep the evaluation infrastructure secure, so we will not add this automatic PR path. Vally updates can still be validated by pushing a same-repository branch and manually dispatching the evaluation workflow on that branch, with selected plugins, before merge.

(Copilot, closing on Abhitej's behalf.)