Skip to content

feat: make benchmark scores decision-safe and repeatable - #3

Merged
tongchen92 merged 1 commit into
mainfrom
codex/decision-safety-benchmark
Sep 8, 2026
Merged

tongchen92 merged 1 commit into
mainfrom
codex/decision-safety-benchmark

Conversation

@tongchen92

Copy link
Copy Markdown
Contributor

Summary

MediaBuyerBench can now compare models on a narrow, auditable claim: whether they make source-grounded paid-media decisions safely under a recorded harness. Headline results fail closed when safety evidence, repeated-run coverage, or judge-cache identity is incomplete.

What changed

  • Added required and critical safety gates for facts, prerequisites, action scope, unsupported actions, and context-bound calculations.
  • Added a five-run suite contract with model-plus-runtime cohorts, exact case coverage, shared prompt/limit/retry metadata, and strict summary validation.
  • Added a blind three-family judge panel with opaque candidate labels and input-bound cache digests.
  • Added neutral reporting for Tasks Completed Safely and Responses With Serious Errors, including 95% Wilson confidence intervals.
  • Narrowed the documented claim: the current five-case suite validates the framework, not general media-buyer competence or a publishable leaderboard.

Validation

  • python3 -m unittest discover -s tests - 81 tests passed.
  • Python compilation completed for mediabuyerbench and scripts.
  • Every JSON case, suite, schema, rubric, and source pack parsed successfully.
  • All five public sample answers passed score --require-safety-pass.
  • Review findings fixed false passes caused by negation, wrong-metric numbers, missing safety records, incomplete run cohorts, and stale judge caches.

Code review completed: run 20260907-225515-4f421eaa. Four local reviewer lenses completed and an independent validator confirmed all six findings. The optional Claude cross-model review could not authenticate, so no cross-model review result was used.

Limitations

  • Automated cross-family agreement is a consistency check, not proof of real-world media-buying skill.
  • The five public cases are a framework validation set. Model-level claims should remain secondary until the suite has at least 20 source-closed cases.
  • Live external judge CLIs were not exercised in final validation; the runner aborts rather than emitting a summary when a judge fails.

Post-Deploy Monitoring & Validation

No additional operational monitoring required. This changes a local benchmark CLI and artifacts, not a production service.

  • Validation window: the first complete five-run comparison after merge.
  • Owner: benchmark maintainer.
  • Healthy signal: every cohort has five unique runs per case, summarize-panel succeeds, and all three judge families return complete judgments.
  • Failure signal: manifest rejection, judge subprocess failure, cache digest mismatch, or summary validation failure. Do not publish a chart from that run; fix the input or runner and rerun the full suite.
  • Rollback: revert this commit and retain the prior benchmark outputs as historical artifacts only.

New concepts

Fail-closed benchmark scoring

The headline metric is emitted only when the evidence contract is complete. A missing safety result, incomplete cohort, mismatched model/runtime, or stale cached judgment becomes an error instead of an optimistic default. Use this pattern for release gates and leaderboards; avoid it for exploratory notebooks where partial output is explicitly useful.

Wilson confidence intervals

Wilson intervals express uncertainty in pass and error rates without the unstable edge behavior of a simple normal interval near 0% or 100%. Here they keep a small benchmark from presenting a point estimate as precise. They do not remove dependence between repeated runs or compensate for too few distinct cases.


Compound Engineering

@tongchen92
tongchen92 merged commit 69ef6c2 into main Sep 8, 2026
2 checks passed
@tongchen92
tongchen92 deleted the codex/decision-safety-benchmark branch September 8, 2026 07:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant