Skip to content

feat: add source-grounded Google Search evaluation - #2

Merged
tongchen92 merged 1 commit into
mainfrom
feat/source-grounded-google-search-evals
Aug 14, 2026
Merged

tongchen92 merged 1 commit into
mainfrom
feat/source-grounded-google-search-evals

Conversation

@tongchen92

Copy link
Copy Markdown
Contributor

Summary

MediaBuyerBench can now distinguish a plausible headline recommendation from a safe, auditable Google Search operating decision.

  • Replaces the saturated cross-platform smoke cases with five public synthetic Google Ads cases that test retrieval, calculation, diagnosis, recommendation, and safe refusal.
  • Adds deterministic assertions, a blind decision-quality rubric, panel aggregation, and reviewer-calibration reporting.
  • Adds a source-grounded two-pass expert-referee framework, public Google/Adalysis source pack, public demo suite, and repeatable candidate/referee runners.
  • Keeps customer-derived hard cases, references, and source annotations ignored locally; the public repository contains only synthetic cases and reusable tooling.

Validation

  • python3 -m unittest discover -s tests -v (48 passed)
  • python3 -m mediabuyerbench.cli run-samples --split public_lite (5/5 sample responses score 100)
  • python3 -m py_compile mediabuyerbench/*.py scripts/*.py
  • git diff --check

@tongchen92
tongchen92 merged commit 1d2265f into main Aug 14, 2026
2 checks passed
@tongchen92
tongchen92 deleted the feat/source-grounded-google-search-evals branch August 14, 2026 00:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant