Skip to content

Add A/B experiment for rendering verifier prompt context - #58037

Merged
pelikhan merged 3 commits into
mainfrom
copilot/daily-rendering-scripts-verifier-remove-redundant
Sep 2, 2026
Merged

Add A/B experiment for rendering verifier prompt context#58037
pelikhan merged 3 commits into
mainfrom
copilot/daily-rendering-scripts-verifier-remove-redundant

Conversation

Copilot AI commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

daily-rendering-scripts-verifier has the fleet’s highest average AI-credit cost, partly due to four redundant closing guidelines. This change measures whether removing that duplicated context improves execution duration without reducing verification correctness.

  • Experiment
    • Adds balanced control/candidate variants via remove_redundant_context_v1.
    • Uses grader:execution-duration as the primary metric.
    • Guards correctness with eval:correct_verification_outcome >= 0.90.
  • Prompt mutation
    • control preserves the existing Guidelines section.
    • candidate retains only the unique safety rule.
  • Generated workflow
    • Recompiles the corresponding .lock.yml with experiment assignment, eval, and grader wiring.
experiments:
  remove_redundant_context_v1:
    variants: [control, candidate]
    metric: "grader:execution-duration"
    guardrail_metrics:
      - name: "eval:correct_verification_outcome"
        threshold: ">=0.90"

Copilot AI and others added 2 commits September 2, 2026 21:34
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot AI changed the title [WIP] Propose A/B experiment to remove redundant context in daily rendering scripts Add A/B experiment for rendering verifier prompt context Sep 2, 2026
Copilot AI requested a review from pelikhan September 2, 2026 21:43
@pelikhan
pelikhan marked this pull request as ready for review September 2, 2026 21:44
Copilot AI balanced review requested due to automatic review settings September 2, 2026 21:44
@pelikhan
pelikhan merged commit 9587a1c into main Sep 2, 2026
1 check passed
@pelikhan
pelikhan deleted the copilot/daily-rendering-scripts-verifier-remove-redundant branch September 2, 2026 21:44

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The control arm is never selected, the correctness guardrail is inadequate, and the router references a missing file.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds an A/B experiment to evaluate whether trimming redundant verifier prompt guidance reduces execution time without harming correctness.

Changes:

  • Adds control/candidate prompt variants with duration and correctness metrics.
  • Regenerates experiment, grader, and evaluation workflow wiring.
  • Adds compatibility guidance routing.
File summaries
File Description
.github/workflows/daily-rendering-scripts-verifier.md Defines the experiment and conditional prompt mutation.
.github/workflows/daily-rendering-scripts-verifier.lock.yml Compiles experiment, grader, evaluation, and state jobs.
.github/skills/agentic-workflows/SKILL.md Adds a reference to a nonexistent compatibility document.
Review details

Suppressed comments (1)

.github/skills/agentic-workflows/SKILL.md:27

  • The PR description and its validation explicitly scope this change to the verifier source and generated lock file, but this unrelated router update adds a third changed file. Remove it from this PR, or document why compatibility-advisory guidance is part of the experiment change and update the stated validation scope.
- `.github/aw/compat.md`
  • Files reviewed: 3/3 changed files
  • Comments generated: 3
  • Review effort level: Balanced


## Guidelines

{{#if experiments.remove_redundant_context_v1 == 'candidate' }}
- `.github/aw/charts-trending.md`
- `.github/aw/charts.md`
- `.github/aw/cli-commands.md`
- `.github/aw/compat.md`
Comment on lines +23 to +25
evals:
- id: correct_verification_outcome
question: Did the agent correctly complete the verification cycle — creating a pull request with concrete parser/render fixes when real issues were found, or confirming no issues were found (cache-memory update / noop) without fabricating problems?
@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

🎉 This pull request is included in a new release.

Release: v0.88.2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[harness-experiment-proposal] daily-rendering-scripts-verifier — context assembly/remove_redundant_context A/B harness experiment

3 participants