RFC: AgentForensics collaboration — automated failure taxonomy and span-level trace auditing over BQAA traces

[haiyuan-eng-google]

Executive summary

AgentForensics proposes a diagnostic framework that inspects agent execution traces on existing benchmarks (KramaBench, LakeQA, Spider 2.0) to automatically identify and categorize the exact failure point in a run.

That is a close fit for this SDK, and the overlap is complementary rather than duplicative. BQAA already provides the substrate AgentForensics would otherwise have to build — trace capture into BigQuery, trace/DAG reconstruction, an LLM classification engine with strict label validation, and BigQuery-scale aggregation. What AgentForensics brings is the part the SDK does not have: a literature-grounded failure taxonomy, span-level failure localization, and a labeled corpus from public benchmarks to validate against.

The short version: AgentForensics supplies the taxonomy and the ground truth; BQAA supplies the storage, reconstruction and classification engine. Neither side has to build the other's half.

What the SDK already provides

Relevant primitives that exist today:

Capability	Module	Relevance to AgentForensics
Trace reconstruction + DAG visualization	trace/observability layer	Recovering the step sequence a failure sits in
Categorical evaluation into user-defined categories via AI.GENERATE, with strict category validation	categorical_evaluator.py	This is a taxonomy classifier already — it just needs a taxonomy
Facet extraction, aggregation, top-N distributions across sessions	insights.py	Per-benchmark failure-mode distributions
Trajectory matching (exact / in-order / any-order)	evaluators.py	Detecting where an actual trajectory diverges from a reference
Multi-trial evaluation with pass@k / pass^k	multi_trial.py	Separating deterministic failures from flaky ones
Agent Context Graph	context_graph.py	Decision lineage around the failure point
Eval suite lifecycle, graduation, saturation detection	eval_suite.py	Turning a labeled failure set into a durable regression suite

What is genuinely missing (the collaboration surface)

1. No canonical failure taxonomy. categorical_evaluator classifies into whatever categories you define; there is no shipped, defensible set of agent failure modes.
2. Classification is session-level, not span-level. AgentForensics' core claim is locating the exact failure point. That requires attributing a category to a span_id, not a session.
3. No public labeled trace corpus. Every evaluator in this SDK is currently validated against synthetic or internal data.
4. No benchmark harness. Nothing here runs a public benchmark end-to-end and lands the traces.

Proposed plan

Phase 0 — Instrument the benchmarks. Run the benchmark agents with BQAA telemetry enabled so traces land in BigQuery. Deliverable: a reproducible harness under examples/ that executes a benchmark suite and produces a trace dataset. Open question below on which ingestion path applies.

Phase 1 — Express the taxonomy as data, not code. AgentForensics contributes the failure taxonomy; we encode it as CategoricalMetricDefinition / CategoricalMetricCategory. This is runnable on day one with no engine changes, which makes it a cheap way to test whether the taxonomy actually discriminates before anyone writes new code.

Phase 2 — Span-level localization. Extend categorical classification from session to span/turn granularity, emitting (trace_id, span_id, failure_category, evidence, confidence). This is the one real engineering item and should reuse the turn-tagging and sub_trajectories work in #429 rather than inventing a parallel mechanism.

Phase 3 — Validation and golden set. Human-label a stratified sample; measure classifier agreement against it. The labeled set becomes a public regression fixture — which is independently valuable to this SDK and connects to #428 and #430.

Phase 4 — Reporting. Reuse the insights.py aggregation path for per-benchmark failure-mode distributions, plus a notebook and dashboard panel.

Phases 0–1 are deliberately sequenced to produce a falsifiable result early: if the taxonomy does not separate failure modes on real benchmark traces, that is worth discovering before building Phase 2.

What each side gets

• AgentForensics avoids building trace storage, reconstruction, an LLM classification harness and an aggregation layer, and gets BigQuery-scale analysis for free.
• BQAA gets a validated failure taxonomy, a public labeled trace corpus, and external validation of categorical_evaluator on non-synthetic data.

Open questions

1. Which framework do the benchmark reference agents use? If they are ADK-based, the ADK plugin path works directly. If not, ingestion needs an OpenTelemetry route, which changes Phase 0's scope materially. This is the biggest unknown.
2. Is the trace fidelity sufficient for localization? Span-level attribution needs tool-call and step boundaries in the emitted trace. Benchmark harnesses often log only final answers and intermediate SQL, which would make Phase 2 impossible without first improving instrumentation.
3. One taxonomy or three? Spider 2.0 (text-to-SQL) and data-to-insight pipeline benchmarks may not share failure modes. A single taxonomy is more useful; per-benchmark dialects may be more accurate. Worth resolving in Phase 1.
4. Can the resulting traces and labels be published? The corpus is a large part of the value, and that depends on each benchmark's licensing.
5. Ground truth for "the exact failure point" — do the benchmarks provide step-level references, or is the reference only a final answer? If the latter, localization is evaluated against human labels only, which caps the achievable rigor.

Related issues

• Promote correction/anti-parroting trace analysis to core (turn tagging + sub_trajectories) #429 — turn tagging and sub_trajectories (the natural substrate for span-level localization)
• Canonical evaluation rubrics in core — extracted from the current quality_report metrics (supersedes the #91 approach) #430 — canonical evaluation rubrics in core
• Promote golden Q&A matching to core (the producer for CategoricalContextSource.GOLDEN_EXPECTED_ANSWER) #428 — golden Q&A matching

---

Opening this for discussion rather than as a committed plan. Feedback on the phasing — particularly whether Phase 2 should precede a taxonomy validation result — is welcome.

[haiyuan-eng-google]

Related: #436 proposes synthetic evaluation dataset generation from real traces. The two connect directly — the failure taxonomy proposed here supplies the label space that #436 needs for rare-failure oversampling, and #436's labeled corpus would in turn give this issue's taxonomy classifier more data to validate against.

[caohy1988]

Execution & implementation plan

Turning the RFC into a committed, phased plan. First, a re-baseline: two of the RFC's assumptions closed since it was filed (Aug 18), and the plan below is grounded in the repo as of Aug 29.

What changed since the RFC

Event	Consequence
#431 merged (Aug 27) — evaluation_rubrics.py: config-as-data metric schema + build_metrics() interpreter	Phase 1 needs no new engine code. The taxonomy is a JSON config in an already-shipped schema; the interpreter, strict category validation, and the drift-guard test pattern all exist.
#444 merged (Aug 27) — EvalBench reader: EvalBenchRun.from_bigquery() → BQAA event rows	Open question 1 (ingestion path) is largely answered for EvalBench-hosted benchmarks. Phase 0 reduces to finishing the #97 writer phases.
#429 still open — turn tagging / sub_trajectories live in scripts/quality_report.py	Phase 2's preferred substrate needs a promotion-to-core PR first; a fallback that doesn't need it is specified below.
#316/#317 OTLP receiver — design docs only	Non-EvalBench, non-ADK benchmark agents have no ingestion path this quarter → wave 2, don't block on the receiver.
#436 (synthetic datasets) open	Consumes this taxonomy as its label space; no wave-1 dependency either way, but category names freeze once G1 passes.

M0 — Decisions + fidelity spike (weeks 1–2)

• D1 — Ingestion path per benchmark (KramaBench, LakeQA, Spider 2.0): runs under EvalBench → merged reader + Add EvalBench bridge: import EvalBench BigQuery runs into a BQAA-owned mirror table #97 writer; ADK-based agent → ADK plugin directly; neither → wave 2 (blocked on Claude Code → BigQuery: OTel-native OTLP receiver endpoint #316). Record assignments here.
• D2 — One taxonomy with dialects: single core taxonomy + optional per-benchmark extension categories, encoded in the feat(eval): canonical evaluation rubrics in core + rubric reference doc (closes #430) #431 config schema. G1 below resolves RFC question 3 with data, not debate.
• D3 — Licensing audit per benchmark; gates only the publication deliverable of Phase 3, never the engineering.
• S1 — Trace fidelity spike (answers RFC question 2): pull one real EvalBench job through EvalBenchRun.from_bigquery() → to_agent_event_rows(). Are tool-call boundaries present (stdout.tool_calls / accumulated_tools)? Does Client.get_session_trace() reconstruct? If traces carry only final SQL, Phase 2 is impossible for that benchmark — and we want to know in week 1, not week 6.

Phase 0 — Land benchmark traces = finish #97 (weeks 2–4)

Item	Work	Size
W0.1	Mirror-table writer: BQAA-owned table, delete-by-experiment_id-then-append idempotency, cross-project target. Fix the two open #444 review findings first (successful runs polluted with evalbench_error_fields: {returncode: 0}; duplicate scenario-id retries colliding on session/span ids) — both corrupt the corpus at scale.	1.5 wk
W0.2	evalbench-import CLI over the writer.	0.5 wk
W0.3	examples/ harness for one benchmark: AgentForensics side runs it under EvalBench; BQAA side imports. Reproducible from a clean project.	1 wk

Acceptance: ≥100 scenarios queryable in the mirror table; get_session_trace() reconstructs a scenario end-to-end; every event carries text_summary (the judge's 10-char visibility floor); re-import of the same job produces zero duplicates.

Phase 1 — Taxonomy as data, then prove it discriminates (weeks 2–5, parallel with Phase 0)

Item	Work	Size
W1.1	failure_taxonomy.py: builtin_failure_taxonomy() returning an AgentForensics-authored config in the #431 schema; build_metrics() is the interpreter, unchanged. Include the #431-pattern drift-guard test and pin the doc to the data (parity test or committed generator).	1 wk
W1.2	docs/failure_taxonomy.md, generated from the config.	0.5 wk
W1.3	Discrimination experiment: session-level evaluate_categorical() with the taxonomy over the Phase 0 corpus, run twice (stability), plus a 30-trace human spot-check.	1 wk

Gate G1 (end of week 5). Proceed to Phase 2 only if: no single category absorbs >60% of failed sessions; two runs agree on ≥80% of labels; the spot-check finds the modal label plausible. Otherwise iterate the taxonomy (data-only, cheap) or split per-benchmark dialects (D2 fallback). This is the falsifiable result the RFC promised before any real engineering.

Phase 2 — Span-level localization, the one real engineering item (weeks 5–9)

Item	Work	Size
W2.0	Promote #429 to core (turn tagger + sub_trajectories), following the #431/#432 extraction pattern and its review lessons (no monkeypatch-trap aliases in the script, lazy script imports, expect the __init__.py export-block conflict zone).	1 wk
W2.1	localize_failures() API: for each session the taxonomy marks failed, render the ordered event list with span ids and ask the judge for the first failing span_id + category + verbatim evidence + confidence. One LLM call per failed session — not one per span.	1.5 wk
W2.2	Strict validation mirroring the categorical evaluator's label validation: returned span_id must exist in the trace's span set; the evidence quote must be a substring of the rendered trace; violations retry once then reject. This keeps localization auditable.	0.5 wk
W2.3	Output contract (trace_id, span_id, failure_category, evidence, confidence) + aggregation view; tests on synthetic traces with planted failure spans.	1 wk

Fallback if #429 promotion slips: W2.1's windowed design reads the raw ordered event list and doesn't strictly require sub_trajectories; it loses the correction-segment prior but unblocks the phase. Decision point pinned at week 6 so the extraction can't become the critical path.

Phase 3 — Golden set and agreement (weeks 8–12)

• W3.1 — Flakiness triage first: multi_trial pass@k over sampled scenarios; label deterministic failures (pass@k = 0) first; flaky failures get their own stratum — mixing them in poisons the ground truth.
• W3.2 — Stratified sample, ~300 sessions (benchmark × predicted category × outcome). Labeling guide derived verbatim from the taxonomy's judge definitions; two labelers + adjudication.
• W3.3 — Metrics: session-level Cohen's κ (classifier vs adjudicated human); localization hit@1 and hit-within-±1-span. Where benchmarks provide only final-answer references (RFC question 5), report human-label agreement as the ceiling and say so.
• W3.4 — Durability: land labels as test fixtures; graduate stable ones through eval_suite into a regression suite; publish the corpus if D3 clears.

Gate G2 (end of week 12). κ ≥ 0.6 at session level to claim the classifier works; below that, the deliverable is the labeled corpus plus an honest error analysis, and Phase 4 ships with agreement caveats attached.

Phase 4 — Reporting (weeks 11–13)

Per-benchmark failure-mode distributions through the existing insights.py facet aggregation; a worked notebook under examples/; a dashboard panel. Dashboard lesson from #440: the shipping path is bindings/report_template.yaml — update it and the spec contract together or the feature never reaches users.

Risks

Risk	Mitigation
Benchmark traces carry only final answers → localization impossible	S1 spike answers this in week 1 per benchmark; Phase 2 descopes per-benchmark; session-level taxonomy still delivers value.
Taxonomy collapses into one bucket on real traffic	G1 catches it before Phase 2 spend; iterate data-only or split dialects.
Judge cost at benchmark scale	Classify only failed sessions (filter on scores/returncode first); localization is one call per failed session by design.
#429 extraction becomes critical path	Windowed fallback in W2.1; decision at week 6.
Licensing blocks corpus publication	D3 in week 1; worst case the corpus stays an internal fixture and the external-validation claim is softened; engineering unaffected.
Corpus corrupted by importer defects	The two open #444 findings fixed in W0.1 before bulk import.

Sequencing at a glance

Weeks	Track A (BQAA eng)	Track B (parallel)
1–2	M0: D1–D3, S1 fidelity spike	AF drafts taxonomy content (W1.1 input)
2–4	Phase 0: writer, CLI, harness	Phase 1: taxonomy encoding + docs
5	G1: discrimination gate — first falsifiable result
5–9	Phase 2: #429 promotion, localize_failures	AF runs remaining benchmarks through the Phase 0 harness
8–12	Phase 3: triage, labeling, agreement	Labeling by AF collaborators
11–13	Phase 4: insights, notebook, dashboard	G2: agreement gate

~13 weeks, 1–2 engineers plus AgentForensics collaborators; first falsifiable result at week 5. On the RFC's phasing question (should Phase 2 precede taxonomy validation): no — G1 stays in front of Phase 2, and the re-baseline makes that ordering nearly free since Phase 1 is now data-only.

[caohy1988]

Review of the execution plan

Full-context review: I checked the re-baseline against the repo as of today and read the plan against the RFC's five open questions. Overall this is strong — the gate structure is right, the S1 spike targets the exact risk that should be de-risked first, and W0.1 correctly fixes the corpus-corrupting #444 findings before bulk import. One factual error in the re-baseline table, plus several methodological gaps worth tightening before this becomes the committed plan.

Re-baseline verification

• ✅ feat(eval): canonical evaluation rubrics in core + rubric reference doc (closes #430) #431 merged Aug 27 (evaluation_rubrics.py, build_metrics(), drift-guard pattern) — the "Phase 1 is data-only" conclusion holds.
• ✅ feat(evalbench): add BigQuery run reader and trace mapping #444 merged Aug 27 (EvalBenchRun.from_bigquery() → event rows).
• ✅ Promote correction/anti-parroting trace analysis to core (turn tagging + sub_trajectories) #429 open; confirmed _tag_conversation_turns, sub_trajectories, correction_boundaries all live in scripts/quality_report.py (~lines 836–1308).
• ✅ RFC: Synthetic evaluation dataset generation from real agent traces #436 open.

Correction — the #316/#317 row is out of date. "#316/#317 OTLP receiver — design docs only" is no longer accurate: #317 closed COMPLETED on Jul 7 with a real-CLI-verified live deployment (codex-cli 0.142.5 against the #324 receiver, which also closed COMPLETED), and the receiver ships as released code — deploy/otlp_receiver/ plus the bqaa-otel bootstrap package with a digest-pinned public image. #316 is still open, but the receiver is deployed infrastructure, not docs. This matters for D1: the "neither EvalBench nor ADK → wave 2" branch is likely too pessimistic. Any OTel-capable harness (KramaBench's reference agents are Python) is a candidate for the receiver path, with the caveat that the agent_events_otlp projection is currently shaped around Claude Code/Codex producers — a generic benchmark producer would need projection work, which is smaller than "no ingestion path." Suggest re-scoping D1 to evaluate three routes, not two.

Methodological gaps

1. G1 measures stability, not validity — say so. "Two runs agree on ≥80%" only shows the judge is consistent; LLM judges can be stably wrong. The 30-trace spot-check with "modal label plausible" is a smoke test, not a validity measurement — at n=30 you cannot detect a systematic 10–15% mislabel rate. That's arguably fine (G2 is the real validity gate), but the comment calls G1 "the falsifiable result the RFC promised" — it only falsifies degenerate collapse. One sentence of honest framing here prevents over-claiming at week 5.

2. The >60% concentration threshold needs a base-rate anchor. On a benchmark whose failures genuinely concentrate (Spider 2.0 plausibly has a dominant wrong-SQL mode), a correct taxonomy could trip the gate. Make the check comparative, not absolute: concentration is a red flag only if the human spot-check base rate disagrees with the classifier's distribution.

3. W3.2 doesn't collect the ground truth W3.3 measures. W3.3 computes localization hit@1 and hit-within-±1-span, but the W3.2 labeling guide is described as session-level category labeling. Span-level ground truth — labelers reading ordered traces and marking the first-failing span — is a different, much more expensive annotation task, and it's currently implicit. Make it explicit, re-cost it (300 sessions × 2 labelers × span-level reading is plausibly more than the 4 weeks allotted), and measure inter-labeler agreement on span choice, not just category κ — otherwise hit@1 has an unaudited denominator.

4. W2.1's "one call per failed session" needs a length story. Spider 2.0-class traces can exceed the judge's context when rendered as a full ordered event list. The windowed design is mentioned only in the #429-slippage fallback; it should be part of W2.1 proper. Note the interaction with W2.2: evidence-substring validation against the rendered trace constrains windowing (evidence can only come from non-elided windows), which is fine but should be deliberate.

5. Self-reported confidence will be miscalibrated. The output contract includes a confidence field; LLM self-reports are known-poorly calibrated. Either validate it against Phase 3 agreement data before exposing it, or document it as informational-only.

6. W0.1 idempotency: delete-then-append is non-atomic. A failed or concurrent re-import between the delete and the append leaves the mirror table partially populated. Given the acceptance criterion is "re-import produces zero duplicates," consider staging-table-then-transaction or a partition-replace pattern instead.

7. W3.1 flakiness triage has an unbudgeted dependency. pass@k over sampled scenarios requires re-running benchmark agents k times through the Phase 0 harness — compute cost and wall-time aren't estimated anywhere, and k trials × 300-session strata is not free. A line-item (even "k=3, sampled 50 scenarios") would make G2's timeline more credible.

Smaller notes

• D1 says "record assignments here" but doesn't — the framework each benchmark's reference agent uses is presumably a day of investigation, not two weeks; resolving it in the comment would strengthen M0.
• Category-name freeze at G1 for RFC: Synthetic evaluation dataset generation from real agent traces #436's benefit is a good cross-issue callout; consider also freezing the insert_after/category-order semantics so RFC: Synthetic evaluation dataset generation from real agent traces #436's label space is byte-stable.
• Phase 4's feat(dashboard): add global filters for trace, agent, session, and span #440 lesson (update bindings/report_template.yaml and the contract together) is exactly right.

Verdict

The phasing answer to the RFC's closing question is correct and the re-baseline genuinely makes it cheap: G1 in front of Phase 2 costs nothing now that Phase 1 is data-only. Fix the #316/#317 row (it changes D1's answer, not just the table), and tighten items 1–3 before this is treated as committed — everything else is iteration-grade feedback.

Review by @caohy1988's assistant (Kimi Code CLI).

[caohy1988]

Fresh second-pass review of the execution plan

Verdict: the phase ordering is right, but I would not treat this as implementation-ready yet. Keeping taxonomy validation ahead of localization is the correct answer to the RFC's closing question. The M0 fidelity spike, explicit fallback for the correction-segment dependency, and a negative deliverable when agreement fails are all good choices. The load-bearing gates and a few re-baseline claims still need correction.

I rechecked current main, the referenced issues/PRs, and the earlier review comment. This independently confirms its concerns about the stale OTLP baseline, weak G1 validity signal, missing span-labeling protocol, long-trace handling, and flakiness cost. The following are the additional or sharper changes I think are required.

1. Correct the re-baseline and state the actual Phase 0 scope

• The two findings assigned to W0.1 are already fixed in merged feat(evalbench): add BigQuery run reader and trace mapping #444. Current main only records a non-zero return code as an error (implementation), and duplicate scenario IDs are rejected before synthetic IDs are produced (implementation, test). Remove both as open prerequisites and re-estimate W0.1.
• The #316/#317 — design docs only row is incorrect. #317 is closed as completed, with live Codex logs, metrics, and spans through the receiver; receiver deployment and producer code are present on main. D1 should test an OTLP-capable route before sending every non-EvalBench/non-ADK benchmark to wave 2.
• “Phase 0 = finish Add EvalBench bridge: import EvalBench BigQuery runs into a BQAA-owned mirror table #97” does not match #97's declared scope. The plan lists the event writer, import CLI, and one harness, but Add EvalBench bridge: import EvalBench BigQuery runs into a BQAA-owned mirror table #97 also owns the imported score table, evalbench-score, evaluation integration tests, and end-to-end documentation. Either call this the trace-ingestion subset of Add EvalBench bridge: import EvalBench BigQuery runs into a BQAA-owned mirror table #97 and track the remainder, or add those deliverables and effort.
• #432 merged just before this plan and now supplies golden expected-answer matching. Add it to the re-baseline and use it explicitly where Phase 3 can compare benchmark references.

Phase 0 also needs a canonical failed-session contract. Later phases say they classify only failed sessions using scores/returncode, but the acceptance criteria mention only event rows. A zero return code means the process completed; it does not mean the benchmark passed. Require a persisted or explicitly queried score/outcome source, a scenario-to-score join, per-benchmark scorer/threshold policy, and an acceptance test that selects both completed-but-low-scoring runs and process failures. Otherwise G1 has no reproducible denominator.

2. G1 cannot yet support the decision or freeze it triggers

As scheduled, W0.3 imports one benchmark before G1, while the sequencing table runs the remaining benchmarks during Phase 2. G1 is nevertheless expected to settle the core-vs-dialect decision and freeze category names. A one-benchmark corpus cannot justify a cross-benchmark taxonomy.

The criteria also measure balance and repeatability, not validity. A judge can be consistently wrong, assign a balanced set of wrong labels, and still pass all quantitative checks; “the modal label is plausible” does not test the other categories. The 100-scenario Phase 0 floor is not a failed-session floor, so the percentages may also be computed over a very small denominator.

Before Phase 2, G1 should require:

• a representative minimum of failed sessions from every wave-1 benchmark, with results reported per benchmark and pooled;
• a blinded, category-stratified human sample with two independent labelers, pre-adjudication human-human agreement, per-category confusion/coverage, and an explicit unknown/abstain path;
• a stated failed-session sample-size floor and uncertainty around the concentration/agreement estimates;
• a time-boxed failure branch: one taxonomy revision, then the dialect fallback, then descope localization if the second gate fails.

Treat the G1 taxonomy as a versioned provisional contract. A hard external freeze should wait for G2; otherwise #436 and regression fixtures can depend on labels that the later human study rejects.

3. Define what “first failing span” means and gate it separately

The EvalBench bridge produces deterministic synthetic span IDs from reconstructed events (current mapping). Passing get_session_trace() proves the reconstruction is queryable; it does not prove that the returned ID is an original execution span or that it identifies a causal failure. S1 should run per benchmark and compare source events with mirror rows for ordering, multiplicity, errors/results, and the boundary that a human would label. If original spans are unavailable, name the claim honestly as localization to an imported event boundary.

The API must also handle omission failures and ambiguous cascades. An expected action that never occurred has no valid failing span_id; root cause, first bad action, and first observable symptom can be different spans. Add unlocalized/missing_step and multi-span or ambiguity outcomes, define the target precisely, and make span_id optional when no defensible span exists. Self-reported confidence should remain informational until calibrated.

W3.2 must explicitly collect human span ground truth; category labels alone cannot produce W3.3's hit@1 denominator. Define adjacency for “±1 span” on an ordered/tree trace, blind labelers to model output, and report span-choice inter-labeler agreement. Add a deterministic baseline such as the first explicit error span.

Finally, G2 currently gates only session-level κ. Span localization is described as the plan's main engineering contribution, yet it can fail every localization metric while the project still passes G2. Add predeclared hit@1/near-hit thresholds and scope reporting back to session-level categories when a benchmark misses them. Also make the long-trace/window policy part of W2.1; the fallback calls the design “windowed,” but W2.1 currently specifies one full-trace call.

4. Fix the Phase 3 measurement design

Sampling by the classifier's predicted category is useful for diagnosis, but an unweighted κ on that sample does not estimate agreement on the natural benchmark distribution. Use two samples or appropriate weights: a frozen random holdout for the headline gate, plus the prediction-stratified sample for per-category error analysis and localization. Report human-human agreement before adjudication, classifier per-class precision/recall (or equivalent), confusion matrices, and confidence intervals alongside κ.

pass@k = 0 in a finite run means “no success observed,” not “deterministic failure.” Predefine k, seeds/retry conditions, and the maximum success probability the trial budget is intended to rule out.

5. Add the missing privacy and data-governance gate

D3 cannot be licensing-only. The plan sends rendered traces to a judge, copies them cross-project, exposes them to external labelers, stores verbatim evidence, lands fixtures in the repository, and may publish a corpus. #436 already states that traces contain customer content. Licensing does not answer privacy, confidentiality, consent, retention, or third-party processing questions.

Add an M0 data-governance decision covering the approved judge deployment and retention terms, field minimization/redaction before judge calls, secret/PII scanning, least-privilege dataset and labeler access, TTL/deletion, provenance, and release approval. Gate both real repository fixtures and public corpus artifacts on it; use synthetic or approved redacted fixtures before clearance.

6. Make reporting follow the validated result path

insights.py aggregates a fixed conversation-facet schema, while the SDK already has persisted categorical results and dedicated deduplicated dashboard views (current categorical view contract). Phase 4 should build on categorical-result persistence/views and add benchmark identity/grouping, unless a separate insights integration is an intentional new work item.

Ship a small session-level report/notebook immediately after a valid G1 so collaborators can test usefulness before the localization spend. Phase 4 implementation can begin before G2, but public/external reporting claims should not ship before G2. A failed G2 should block classifier/localization claims rather than merely attach a caveat to the same dashboard.

Bottom line

I agree with the core ordering: S1 first, then a real taxonomy-validity gate, then localization. I would approve the plan after four things are explicit: the corrected dependency/scope baseline and failure-outcome join; a multi-benchmark, human-grounded G1; a precise localization target with human span ground truth and its own gate; and a privacy/data-governance contract for judge, labeling, fixtures, and publication.

[caohy1988]

Execution & implementation plan — v2

Revised after the two reviews above. I re-verified every factual correction against current main before accepting it — all of them check out — and the methodological items are incorporated below. Thanks to both reviewers; this version supersedes my Aug 29 comment.

Changelog v1 → v2:

1. Re-baseline corrected: the OTLP receiver is deployed infrastructure (Codex → BigQuery: reuse OTel-native receiver + org-wide config templates #317 closed completed Jul 7, deploy/otlp_receiver/ on main), not design docs — D1 now evaluates three ingestion routes. The two feat(evalbench): add BigQuery run reader and trace mapping #444 findings are already fixed in the merged code (_failed_returncode, duplicate-scenario rejection) and are removed from W0.1. feat(eval): golden Q&A matching in core — the producer for GOLDEN_EXPECTED_ANSWER (closes #428) #432 merged Aug 29 — golden matching is now in core and Phase 3 uses it.
2. Phase 0 re-scoped to all of Add EvalBench bridge: import EvalBench BigQuery runs into a BQAA-owned mirror table #97, and the score path is now load-bearing: a new failed-session contract (W0.4) defines G1's denominator; returncode alone is explicitly not a pass/fail signal.
3. G1 redesigned: multi-benchmark corpus required, honest stability-vs-validity framing, a blinded two-labeler mini-study, comparative (not absolute) concentration check, and a versioned provisional taxonomy — the hard freeze moves to G2.
4. Phase 2 redefined: the localization target is named honestly (imported event boundary, since EvalBench-path span ids are synthetic), span_id becomes optional with unlocalized/missing_step outcomes, windowing is part of W2.1 proper, a deterministic baseline is added, and G2 gains localization-specific thresholds.
5. Phase 3 measurement fixed: frozen random holdout for the headline κ + a separate prediction-stratified diagnostic sample; explicit span-level ground-truth labeling task (blinded, span-choice agreement measured, re-costed); pass@k semantics and budget predeclared.
6. New D4 data-governance gate in M0 covering judge deployment/retention, redaction, labeler access, fixtures, and publication — licensing (D3) was never the whole question.
7. Phase 4 rebuilt on categorical-result persistence + categorical_views instead of insights.py facets, with an early session-level report after G1 and external claims gated on G2.
8. Horizon moves ~13 → ~14 weeks (G1 shifts to week 6 to cover all wave-1 benchmarks).

---

Re-baseline (verified against main @ 235cdd4, Aug 29)

Fact	Consequence
#431 merged — evaluation_rubrics.py config schema + build_metrics()	Phase 1 is data-only; interpreter, strict category validation, drift-guard pattern all shipped.
#444 merged — EvalBenchRun.from_bigquery() → event rows, with both review findings already fixed (non-zero-only returncode capture; duplicate scenario ids rejected with a ValueError)	Ingestion reader is corpus-safe as-is; W0.1 shrinks to the writer proper.
#432 merged Aug 29 — golden Q&A matching in core (match_golden_qa, DEFAULT_GOLDEN_THRESHOLD)	Phase 3 binds benchmark reference answers as judge context through the shipped producer instead of ad-hoc plumbing.
#317 closed completed (Jul 7); OTLP receiver deployed (deploy/otlp_receiver/, bqaa-otel bootstrap); #316 still open	Any OTel-capable benchmark harness is a candidate ingestion route today. Caveat: the agent_events projection is shaped around Claude Code/Codex producers, so a generic benchmark producer needs projection work — smaller than "no path," bigger than zero.
#429 open — turn tagging / sub_trajectories still script-side	Phase 2 prerequisite with a specified fallback (below).
#436 open — synthetic datasets RFC	Consumes this taxonomy as its label space. It gets a versioned provisional taxonomy at G1 and a hard freeze (names + category-order semantics) only at G2.

M0 — Decisions, governance, and the fidelity spike (weeks 1–2)

• D1 — Ingestion route per benchmark, now a three-way evaluation: (a) EvalBench-hosted → merged reader + Add EvalBench bridge: import EvalBench BigQuery runs into a BQAA-owned mirror table #97 writer; (b) ADK-based agent → ADK plugin directly; (c) OTel-capable harness → deployed OTLP receiver + a benchmark-producer projection (scoped in M0, not assumed). Only benchmarks fitting none of the three go to wave 2. The framework-identification work is ~a day per benchmark; assignments get recorded here as an M0 deliverable, not deferred.
• D2 — One taxonomy with dialects, encoded in the feat(eval): canonical evaluation rubrics in core + rubric reference doc (closes #430) #431 config schema; the core-vs-dialect question is resolved by G1's per-benchmark results, which is why G1 now requires multi-benchmark coverage.
• D3 — Licensing audit per benchmark (unchanged; gates publication only).
• D4 — Data governance (new): approved judge deployment and retention terms; field minimization/redaction before judge calls; secret/PII scanning on rendered traces; least-privilege dataset and labeler access; TTL/deletion and provenance for copied traces; release approval for repo fixtures and any public corpus. Until cleared, fixtures are synthetic or approved-redacted. This gates Phase 3's labeling and W3.4's publication, and it starts in week 1 because approvals have lead time.
• S1 — Trace fidelity spike, per wave-1 benchmark: pull a real job through the reader; compare source events against mirror rows for ordering, multiplicity, errors/results — and specifically whether the boundary a human would call "the failure point" survives the mapping. Where original execution spans don't exist, the plan's localization claim is named honestly: localization to an imported event boundary, not an original span.

Phase 0 — Land benchmark traces and outcomes = #97, in full (weeks 2–5)

Item	Work	Size
W0.1	Mirror-table event writer: BQAA-owned table, cross-project target, idempotent replacement. Replace delete-then-append with a staging-table or partition-replace pattern — delete-then-append is non-atomic and a failed re-import leaves a partial corpus.	1 wk
W0.2	evalbench-import + evalbench-score CLI; evalbench_scores_imported table (the #97 score half, previously glossed — now required, see W0.4).	1 wk
W0.3	examples/ harness for wave-1 benchmarks: AF side runs them under EvalBench; BQAA side imports. Reproducible from a clean project.	1 wk
W0.4	Failed-session contract (new): per-benchmark scorer/threshold policy; a scenario-to-score join view defining "failed session"; explicit rule that returncode == 0 means completed, not passed. Acceptance test selects both completed-but-low-scoring runs and process failures. This is G1's denominator — without it the gate has no reproducible input.	0.5 wk

Acceptance: every wave-1 benchmark has ≥100 scenarios and their scores queryable; get_session_trace() reconstructs; re-import of the same job is a no-op; the failed-session view returns a stable, documented set.

Phase 1 — Taxonomy as data (weeks 2–5) and Gate G1 (week 6)

W1.1–W1.3 as in v1 (config in the #431 schema, build_metrics() unchanged, doc pinned to data, discrimination run over the corpus). What changes is the gate:

Gate G1 (week 6) — stability and non-degeneracy, honestly framed. G1's quantitative checks (run-to-run agreement ≥80%, non-degenerate distribution) measure whether the judge is consistent and discriminating — a judge can be stably wrong, so G1 does not claim validity; G2 does. What G1 requires:

• a failed-session floor from every wave-1 benchmark (target ≥50 failed sessions each, ≥150 pooled; report per-benchmark and pooled, with uncertainty on the estimates);
• concentration judged comparatively: a dominant category is a red flag only where the human sample's base rate disagrees with the classifier's distribution — Spider-class benchmarks may legitimately concentrate;
• a blinded, category-stratified mini-study: ~60 traces, two independent labelers, pre-adjudication human-human agreement reported, per-category confusion/coverage, and an explicit abstain/unknown path;
• a time-boxed failure branch: one taxonomy revision → per-benchmark dialects (D2 fallback) → if a second gate attempt fails, descope localization and ship session-level classification as the wave-1 deliverable.

The G1 taxonomy is a versioned provisional contract (v0.x). #436 and any fixtures may build on it at "provisional" status; the hard freeze of category names and order semantics happens at G2.

Phase 2 — Localization to a failing event boundary (weeks 6–10)

Item	Work	Size
W2.0	Promote #429 to core (turn tagger + sub_trajectories), #431/#432 extraction pattern and review lessons. Fallback unchanged: the windowed design below reads raw ordered events and doesn't strictly need it; decision point week 7.	1 wk
W2.1	localize_failures(): for each failed session, render the ordered event list with event-boundary ids and ask the judge for the first failing boundary + category + verbatim evidence + confidence. Windowing is part of this item, not a fallback: long traces are classified over windows with a first-pass coarse pass to select the window; evidence-substring validation applies per rendered window (evidence can only come from non-elided content — deliberate, documented).	2 wk
W2.2	Strict validation: returned id must exist in the trace's boundary set; evidence must be a substring of the rendered window; violations retry once then reject. Output contract widened: span_id is optional; outcomes include localized, unlocalized, and missing_step (an expected action that never happened has no valid failing span — that's a finding, not an error); ambiguous cascades may carry an optional secondary span. The target is defined precisely as first observable failing event boundary — not root cause, which G2's human labels also target so the two stay comparable. confidence is documented as informational-only until calibrated against Phase 3 data.	1 wk
W2.3	Aggregation view + tests on synthetic traces with planted failures, planted omissions (missing_step), and over-length traces. Plus a deterministic baseline: first status=ERROR event boundary — the number localize_failures() must beat for the LLM step to be worth its cost.	1 wk

Phase 3 — Golden set and agreement (weeks 9–13)

• W3.1 — Flakiness triage, budgeted: pass@k with predeclared k=3 over a 50-scenario sample per benchmark (compute/wall-time line-item in the harness). pass@3 = 0 is reported as "no success observed in 3 trials," with the maximum success probability the budget rules out stated — never as "deterministic failure."
• W3.2 — Two samples, two tasks: (a) a frozen random holdout (~150 failed sessions, natural distribution) for the headline agreement number; (b) a prediction-stratified diagnostic sample (~150) for per-category confusion and error analysis — stratified κ doesn't estimate natural-distribution agreement, so it never headlines. Span-level ground truth is an explicit separate annotation task: labelers read ordered traces blinded to model output and mark the first failing boundary (or missing_step/unlocalizable); adjacency for ±1 is defined on the rendered event order; span-choice inter-labeler agreement is reported, not just category κ. Re-costed: two labelers + adjudication across both tasks is ~3 labeler-weeks — sized into the window, and the session count shrinks before the protocol does.
• W3.3 — Metrics: session-level Cohen's κ with CIs on the holdout (target ≥0.6), per-class precision/recall + confusion on the diagnostic sample, pre-adjudication human-human agreement as the ceiling, localization hit@1 / hit-within-±1 vs both human labels and the deterministic baseline. Where a benchmark provides reference answers, they're bound as judge context via the now-merged feat(eval): golden Q&A matching in core — the producer for GOLDEN_EXPECTED_ANSWER (closes #428) #432 match_golden_qa.
• W3.4 — Durability: labels land as fixtures (per D4: synthetic/redacted until cleared); stable ones graduate through eval_suite; corpus publication only after D3 and D4 clear.

Gate G2 (weeks 13–14) — the validity gate, now two-part. Session-level: holdout κ ≥ 0.6 to claim the classifier works; below that, the deliverable is the labeled corpus + error analysis, and no classifier claims ship. Localization (new): predeclared hit@1 must beat the deterministic first-error baseline by a stated margin, per benchmark; a benchmark that misses it gets session-level-only reporting — localization can't ride through G2 on κ alone when it's the plan's main engineering claim. G2 passing triggers the hard taxonomy freeze (names + order semantics) for #436 and dashboards.

Phase 4 — Reporting (weeks 12–14, external claims gated on G2)

Built on categorical-result persistence and categorical_views (the deduplicated dashboard view contract), extended with benchmark identity/grouping — not on insights.py's fixed conversation-facet schema; if an insights integration is wanted it's a separate, explicit item. A small session-level report/notebook ships immediately after G1 so collaborators can judge usefulness before the localization spend. Dashboard work updates bindings/report_template.yaml and the spec contract together (#440 lesson). External/public claims wait for G2; a failed G2 blocks claims rather than footnoting them.

Risks (delta from v1)

Risk	Mitigation
Synthetic event boundaries misrepresent the real failure point	S1 compares source↔mirror per benchmark in week 1; claim renamed to "imported event boundary"; G2's localization gate measures it against human labels.
G1 passes on a stably-wrong judge	G1 explicitly framed as stability-only; blinded mini-study catches gross invalidity; G2 is the validity gate.
Governance/privacy blocks labeling or publication late	D4 starts week 1 with the approvals on the critical path called out; fixtures synthetic/redacted until cleared.
Long traces exceed judge context	Windowing in W2.1 proper, tested in W2.3.
Span ground-truth labeling blows the Phase 3 budget	Costed up front (~3 labeler-weeks); session count is the shrink lever, protocol is not.
(Retired) #444 corpus-corruption findings	Fixed on main before this revision — verified, not assumed.

Sequencing v2

Weeks	Track A (BQAA eng)	Track B (parallel)
1–2	M0: D1–D4, S1 per benchmark	AF drafts taxonomy content; governance approvals start
2–5	Phase 0: writer, score path, harness, failed-session contract	Phase 1: taxonomy encoding + docs; AF runs all wave-1 benchmarks
6	G1: multi-benchmark stability gate + mini human study
6–10	Phase 2: #429 promotion, localize_failures, baseline	Session-level report ships right after G1
9–13	Phase 3: triage, holdout + diagnostic labeling, span ground truth	Labeling by AF collaborators (per D4 access)
12–14	Phase 4: views, notebook, dashboard	G2: validity + localization gate → taxonomy freeze

~14 weeks. The core ordering survives both reviews intact: S1 first, a real gate before localization, localization before the corpus claim — what changed is that each gate now measures what it claims to measure.

[caohy1988]

Review of plan v2 — plus what the MVP looks like

Verification of v2's claims (all spot-checked against main @ 235cdd4)

• ✅ The two feat(evalbench): add BigQuery run reader and trace mapping #444 findings are fixed in the merged code: _failed_returncode captures non-zero only (evalbench.py:629-633), and duplicate scenario ids raise ValueError naming the job, id, and both indexes (:154-160). Removing them from W0.1 is correct.
• ✅ feat(eval): golden Q&A matching in core — the producer for GOLDEN_EXPECTED_ANSWER (closes #428) #432 merged Aug 29; match_golden_qa / DEFAULT_GOLDEN_THRESHOLD are in core.
• ✅ Codex → BigQuery: reuse OTel-native receiver + org-wide config templates #317 closed completed, receiver is deployed code (deploy/otlp_receiver/).
• ✅ Add EvalBench bridge: import EvalBench BigQuery runs into a BQAA-owned mirror table #97 is open and its scope genuinely includes the score half and the mirror-table bridge — the Phase 0 re-scope is legitimate, not scope creep.
• ✅ categorical_views exists, so Phase 4's new foundation is real.

Every v1 review item is addressed — stability-vs-validity framing, comparative concentration check, explicit span ground truth, windowing in W2.1 proper, confidence as informational-only, atomic import, budgeted pass@k. This round's findings are all new and smaller.

Findings

1. MAJOR — D4's gate scope contradicts G1's mini-study. D4 is stated to gate "Phase 3's labeling and W3.4's publication," but G1 (week 6) now includes a blinded two-labeler study over 60 real benchmark traces. That is human labeling on real data — it needs the same clearance (labeler access, redaction, retention) the plan says has lead time. Either D4 must clear by week 5 (putting approvals on G1's critical path — call that out), or the G1 mini-study explicitly runs on synthetic/redacted traces, which weakens it. Pick one and write it down; as written, the plan's first human study is ungoverned.

2. MINOR — G1's per-benchmark floor is hostage to Track B, with no descope stated. ≥50 failed sessions per benchmark by week 6 requires AF to have run every wave-1 benchmark by ~week 5. If one benchmark's pass rate is high, 50 failures could mean 200+ scenario runs. State the rule now: does a benchmark that can't produce 50 failures get dropped from G1, or does the gate slip?

3. MINOR — the localization margin should be predeclared at Phase 2 start, not at G2. G2 says hit@1 must beat the deterministic baseline "by a stated margin," but if the margin is chosen at G2 it can be reverse-engineered from observed results. Predeclare it when W2.1 lands, before any localization numbers exist.

4. MINOR — κ gate: point estimate or lower bound? "κ with CIs (target ≥0.6)" is ambiguous — at n≈150 the CI half-width is roughly ±0.1 depending on prevalence. Clarify whether G2 requires the point estimate ≥0.6 or the CI lower bound ≥ some floor; the latter is much harder and the choice should be deliberate.

5. MINOR — don't use embedding matching to bind references that have known keys. Phase 3 binds benchmark reference answers via match_golden_qa, but benchmark scenarios have stable ids — a direct key join is exact, free, and can't silently drop references (recall from the #432 review: below-threshold matches and empty questions are silently omitted, and EMBEDDING_MODEL is read at SDK import time). Reserve match_golden_qa for genuinely fuzzy question text; bind by scenario id where ids exist.

6. NIT — Phase 2 sizing. W2.0(1) + W2.1(2) + W2.2(1) + W2.3(1) = 5 weeks in a weeks 6–10 window, with zero slack for the __init__.py export-conflict zone v1 itself flagged for the #429 extraction. Either the window is 6–11 or one item shrinks.

7. NIT — total labeler load. G1's mini-study (~60 traces × 2 labelers) plus Phase 3's ~300 sessions × 2 plus the span task: confirm the combined volume sits inside the D4 labeler-access approval and the ~3 labeler-weeks budget, or say which number moves.

What the MVP looks like

The plan already contains its MVP — it's the G1 failure-branch deliverable ("ship session-level classification") promoted to the default first milestone:

Scope: one benchmark, one route, session-level only.

• The benchmark S1 says has the best trace fidelity (most EvalBench-native), EvalBench route only — no OTLP projection work, no ADK plugin work.
• W0.1 writer + W0.2 import/score CLI + evalbench_scores_imported + W0.4 failed-session contract (all of it — the score half is what makes the MVP's denominator reproducible).
• Taxonomy v0.1 in the feat(eval): canonical evaluation rubrics in core + rubric reference doc (closes #430) #431 schema + generated doc (W1.1/W1.2).
• Discrimination run ×2 + the 60-trace blinded mini-study on the single benchmark (a G1-lite: stability ≥80%, comparative concentration, human base rate).
• The early session-level failure-distribution report/notebook — Phase 4's post-G1 piece, pulled into the MVP.

Explicitly out: localization (all of Phase 2), golden set/holdout/κ (Phase 3), additional benchmarks, dashboard, corpus publication.

Timeline: ~6 weeks, ending at a single-benchmark G1-style checkpoint.

Exit criteria: ≥100 scenarios + scores queryable; ≥50 failed sessions via the W0.4 view; judge stability ≥80%; distribution non-degenerate against the human base rate; and the report judged useful by AF collaborators. That artifact is the evidence for funding Phase 2 — and if G1-lite fails, you've spent 6 weeks learning the taxonomy doesn't discriminate, which was the RFC's entire reason for gating early.

Every MVP piece is reusable if the full plan proceeds; nothing is throwaway. The main thing the MVP doesn't de-risk is localization feasibility — but S1 (week 1) already bounds that risk before the MVP ends.

Review by @caohy1988's assistant (Kimi Code CLI).

[caohy1988]

Fresh review of plan v2 — recommended six-week MVP

I re-read the full v2 plan and re-checked its factual baseline against main at 235cdd4. The corrections from v1 hold: #317 is deployed code, the two #444 defects are fixed, #432 is merged, and #97 includes the score/outcome half. The core order is also right: trace fidelity -> taxonomy evidence -> localization -> corpus/reporting claims.

My verdict: v2 is materially stronger, but I would not start the full 14-week program unchanged. The remaining risks are not stale facts; they are execution-contract and false-green risks. I recommend a six-week, one-benchmark MVP before productionizing the three-route architecture.

Findings that still need changes in the full plan

1. The wave-1 ingestion contract contradicts the work plan. D1 permits EvalBench, the ADK plugin, and OTLP, while W0.3 requires every wave-1 benchmark to run under EvalBench. The ADK and OTLP score/failure paths have no Phase 0 work item or estimate. Route C is not only a projection: the receiver currently accepts only claude_code and codex as explicit source products (receiver contract), so a generic benchmark producer also needs provenance, identity/auth, score mapping, and fidelity tests. Fix: name the wave-1 roster and make every Phase 0 item route-specific, or keep non-EvalBench routes in wave 2.

2. The corpus contract does not yet support its gates. A floor of 100 scenarios does not guarantee 50 failures per benchmark, much less the later 150-session holdout plus 150-session diagnostic sample. Events and scores are also promoted independently, so a retry can expose new events with old scores and change the failed-session denominator. Fix: size imports from required failed-session counts and measured failure rate; publish events and scores as one immutable import version/manifest that the failed-session view reads.

3. D4 gates too late and covers too little. Real traces reach the Phase 1 judge and G1 human labelers before the plan says D4 is required. Governance must also cover derived data: judge justifications/raw responses, verbatim localization evidence, annotations, notebooks, views, and dashboards. The evaluator must fail closed instead of falling back to an unapproved endpoint or execution mode. Fix: require D4 clearance before the first real-trace mirror, judge call, or labeler access; otherwise use only synthetic or approved pre-redacted data. Record execution mode, model/region, redaction-policy version, retention, and deletion propagation.

4. G1 is still not a reproducible go/no-go gate. The category-stratified mini-study cannot estimate human base rates unless it is reweighted, yet that sample is used to judge category concentration. The plan reports human-human agreement but gives it no pass floor. Its taxonomy-revision branch also consumes time while Phase 2 starts in the same week. Fix: use a random per-benchmark sample for prevalence and keep prediction-stratified cases for diagnostics; predeclare human-human agreement, classifier-versus-human agreement, unknown coverage, and the revision/re-gate schedule.

5. The localization design can pass while the claim is false. A coarse selector can choose a later symptom window and never inspect the first failure. The first-ERROR baseline has no output for low-score failures with no error event. The required improvement is still an unspecified “stated margin,” and hit@1 has no coverage/abstention floor. The final holdout is not explicitly disjoint from traces used to revise the taxonomy or prompts. Fix: use chronological, coverage-preserving windows; make the baseline a total policy over the same denominator; count window-selection misses and abstentions as outcomes; seal a disjoint holdout before tuning; and predeclare a numeric improvement threshold before results are visible.

6. No product-value gate stops the localization spend. The collaborator notebook ships when Phase 2 starts, so negative feedback cannot prevent the build. The cheap deterministic baseline is also measured after the localizer is built. Fix: compute baseline headroom during G1 and make collaborator task success a week-4 go/no-go condition.

7. Two smaller contracts should be corrected. Use one outcome vocabulary (localized, unlocalized, missing_step) everywhere. If the claim is an imported event boundary, expose a boundary_id/target kind or explicitly document that span_id can be a synthetic imported-event identifier. Also, pass@3 is an observation, not useful flakiness triage: 0 successes in 3 trials still permits a large true success probability. Either use a sequential policy sized to the probability being ruled out or remove flakiness triage from the claim.

The MVP I would run

MVP hypothesis: For one scored benchmark that the existing EvalBench reader already supports, can BQAA produce a stable, human-usable failure category and—only if collaborators need it—event-boundary evidence that improves their next debugging action?

In scope

• One named benchmark, one known job, and the mature EvalBench route only.
• One immutable/versioned snapshot of events plus scores, sized to yield at least 50 failed sessions.
• D4 clearance for the exact pilot data, judge path, and labelers; otherwise synthetic/pre-redacted data only.
• Taxonomy v0.1, two independent judge runs, a session-level notebook/report, and a 40–60 trace blinded human study.
• The deterministic first-error baseline computed before localization work.
• If the week-4 gates pass, a notebook/internal event-boundary prototype on 30–50 sealed labels using ordered, overlapping windows.
• A week-6 stop/go memo with quality, workflow value, latency/cost, and error analysis.

Explicitly out of scope

• ADK-plugin and generic OTLP benchmark ingestion; multi-benchmark dialects.
• Completing Add EvalBench bridge: import EvalBench BigQuery runs into a BQAA-owned mirror table #97 in full, including both production CLIs and general multi-benchmark harness hardening. Keep only the reproducible snapshot/score join needed by this pilot and track the remaining Add EvalBench bridge: import EvalBench BigQuery runs into a BQAA-owned mirror table #97 work separately.
• Promote correction/anti-parroting trace analysis to core (turn tagging + sub_trajectories) #429 promotion and a public/core localize_failures() API.
• The 300-session corpus, pass@k program, public fixtures/publication, hard taxonomy freeze, categorical_views expansion, and dashboard bindings.

Six-week sequence and gates

Week	Deliverable	Exit criterion
1	Name the benchmark/job/owner; approve D4 boundary; freeze metric, cost, and latency budgets	No real trace leaves its approved boundary; otherwise switch to synthetic/pre-redacted pilot
1–2	Build one immutable event+score snapshot and failed-session view	At least 50 reproducible failures; rerun produces the same import version and denominator
2–3	Run taxonomy v0.1 twice and ship the session-level notebook	Run-to-run agreement at least 80%; unknown remains a scored outcome
3–4	Two blinded labelers on a random 40–60 trace sample; three collaborators each use the report on 10 investigations	Human-human κ at least 0.6, non-unknown coverage at least 80%, classifier-vs-adjudicated κ at least 0.6, and at least 70% of investigations produce an accepted diagnosis or next action
4	Decide whether localization is needed; measure the total first-error baseline on sealed boundary labels	Continue only if the session classifier passes, collaborators say boundary evidence is needed, and the baseline leaves useful headroom
4–5	Build the internal boundary prototype with chronological/overlapping windows	Every failed session gets localized, unlocalized, or missing_step; no conditioning metrics on only successful localizations
6	Score on the common sealed denominator and issue the stop/go memo	Coverage at least 70%; hit@1 at least 10 percentage points above the total baseline; report 95% CI and every supported trace-length/window stratum; meet the week-1 cost/latency budget

These are pilot investment thresholds, not publication claims; the sample is intentionally too small for the latter.

MVP outcomes

• Taxonomy fails after one revision: stop. Keep the labeled errors and analysis; do not build localization.
• Taxonomy and workflow-value gates pass, but collaborators do not need boundary evidence: the session-level notebook is the MVP. Skip localization.
• Session classification passes but localization misses its gate: ship only the session-level pilot; do not promote a public localization API.
• All MVP gates pass: fund the post-MVP work in this order: finish the durable Add EvalBench bridge: import EvalBench BigQuery runs into a BQAA-owned mirror table #97 import contract, add a second benchmark/route, run the disjoint multi-benchmark validity study, then promote APIs/views/dashboard and make external claims.

This keeps v2's correct ordering but moves the highest-cost generalization behind evidence that the workflow is useful. It also gives the team a real product at week 4 and a defensible investment decision at week 6, instead of waiting until week 14 to learn whether the core interaction is worth productionizing.

[caohy1988]

Execution & implementation plan — v3: MVP-first

Both v2 reviews accepted the phase ordering but converged on the same structural recommendation: don't start the 14-week program — run a six-week, one-benchmark MVP first, and put localization behind a demonstrated-need gate. I verified the one new factual claim (the OTLP receiver's KNOWN_PRODUCTS = ("claude_code", "codex") allowlist at producers/.../otlp/receiver.py:53 — route C is genuinely more than a projection) and accept all findings from both rounds. v3 restructures accordingly: Part I is the MVP; Part II is the post-MVP program, funded by MVP evidence.

Changelog v2 → v3:

1. MVP-first: one benchmark, EvalBench route only, session-level classification as the default product; localization becomes a conditional prototype behind a week-4 need + headroom gate.
2. D4 moved to the front of everything — clearance (or synthetic/pre-redacted data) is required before the first real-trace judge call or labeler access, not just before Phase 3; scope extended to derived data (judge justifications, verbatim evidence, annotations, notebooks); the evaluator fails closed on unapproved endpoints. This resolves the v2 contradiction where G1's mini-study was ungoverned.
3. Immutable import versions: events + scores publish atomically as one versioned snapshot with a manifest; the failed-session view reads a pinned version, so retries can't shift the denominator.
4. Route contract fixed: wave-1 = EvalBench-route benchmarks only. ADK-plugin and OTLP routes move to post-MVP wave 2 with named work items (receiver source-product allowlist + provenance/auth, score mapping, fidelity tests) instead of an unestimated "projection."
5. Gates made reproducible: random samples for prevalence, stratified samples only for diagnostics; predeclared floors including human-human agreement; κ semantics disambiguated (point estimate ≥0.6 and CI lower bound ≥0.45); localization margin and coverage floor predeclared before any localization numbers exist; sealed disjoint holdout before tuning.
6. Localization contract corrected: chronological coverage-preserving windows; a total deterministic baseline (an answer for every failed session, not just those with an ERROR event); boundary_id + target_kind: imported_event vocabulary; window-selection misses and abstentions counted as outcomes.
7. Value gate added: collaborators must use the session-level report on real investigations at week 4, and the baseline's headroom is measured before the localizer is built.
8. Smaller fixes: benchmark references bind by scenario id (exact join) with match_golden_qa reserved for fuzzy text; pass@k reframed (0/3 successes ≠ deterministic failure) and dropped from MVP claims; a labeler-load ledger sits inside the D4 approval; Phase-2's post-MVP window gains slack.

---

Part I — The MVP (weeks 1–6, one benchmark)

Hypothesis: for one scored benchmark the existing EvalBench reader already supports, BQAA can produce a stable, human-usable failure-category breakdown that improves collaborators' next debugging action — and, only if they need it, event-boundary evidence.

What the MVP looks like when it ships

1. One pinned benchmark job in a BQAA mirror dataset, imported as immutable snapshot v1 (events + scores together, one manifest), with a failed_sessions view whose definition is the W0.4 contract: per-benchmark scorer threshold, returncode == 0 means completed not passed.
2. failure_taxonomy v0.1 — a config in the feat(eval): canonical evaluation rubrics in core + rubric reference doc (closes #430) #431 schema (categories with judge definitions, an explicit unknown abstain category), interpreted by the shipped build_metrics(); a generated docs/failure_taxonomy.md pinned to the data.
3. A classification results table: (session_id, failure_category, justification, run_id) from two independent judge runs over the failed-session set.
4. A session-level failure-distribution report/notebook: per-category counts with exemplar traces and drill-down to get_session_trace() — the artifact collaborators actually use in investigations.
5. (Conditional, post week-4 gate) an internal boundary-evidence prototype: for a failed session, the first failing imported event boundary + verbatim evidence quote, with outcomes localized / unlocalized / missing_step.
6. A week-6 stop/go memo: quality numbers with CIs, workflow-value results, cost/latency against the week-1 budget, error analysis, and the funding recommendation for Part II.

Everything is reusable if Part II proceeds; nothing is throwaway.

Explicitly out of MVP scope

ADK-plugin and OTLP ingestion; additional benchmarks and dialects; completing #97 in full (both production CLIs and multi-benchmark hardening — only the snapshot/score join this pilot needs; the rest stays tracked on #97); #429 promotion and any public localize_failures() API; the 300-session corpus, pass@k program, public fixtures/publication, hard taxonomy freeze, categorical_views expansion, dashboard bindings.

Week-by-week with exit criteria

Week	Deliverable	Exit criterion
1	Name the benchmark, job, and owner (chosen by S1 trace fidelity); approve the D4 boundary for exactly this data, judge path, and labelers; freeze metric floors and cost/latency budgets; predeclare the localization margin (hit@1 ≥ total baseline + 10 pp, coverage ≥ 70%) before any numbers exist	No real trace leaves the approved boundary — otherwise the pilot switches to synthetic/pre-redacted data
1–2	Immutable event + score snapshot (one import version + manifest); failed_sessions view; snapshot sized from the measured failure rate to yield the floor	≥ 50 reproducible failed sessions; re-import reproduces the identical version and denominator
2–3	Taxonomy v0.1, two independent judge runs, session-level notebook ships	Run-to-run agreement ≥ 80%; unknown is a scored outcome; non-unknown coverage ≥ 80%
3–4	Blinded two-labeler study on a random 40–60-trace sample; three collaborators each use the report on 10 real investigations	Human-human κ ≥ 0.6; classifier-vs-adjudicated κ ≥ 0.6 (point estimates; CIs reported); ≥ 70% of investigations produce an accepted diagnosis or next action
4	Localization go/no-go: collaborators state whether boundary evidence would change their action; total first-error baseline measured on 30–50 sealed boundary labels	Build only if the classifier passed, the need is stated, and the baseline leaves headroom against the predeclared margin
4–5	Internal boundary prototype: chronological, coverage-preserving overlapping windows; boundary_id + target_kind: imported_event	Every failed session gets localized, unlocalized, or missing_step; no metric conditions on successful localizations only
6	Score on the common sealed denominator; stop/go memo	Coverage ≥ 70%; hit@1 ≥ baseline + 10 pp with 95% CI and per-trace-length stratum reported; week-1 cost/latency budget met

These are pilot investment thresholds, not publication claims — the samples are deliberately too small for the latter.

MVP outcome branches

• Taxonomy fails after one revision → stop; keep the labeled errors and analysis; no localization build.
• Classification + value gates pass, collaborators don't need boundary evidence → the session-level notebook is the product; skip localization, fund Part II's corpus/hardening track only.
• Classification passes, localization misses its gate → ship session-level only; no public localization API.
• All gates pass → fund Part II in order: durable Add EvalBench bridge: import EvalBench BigQuery runs into a BQAA-owned mirror table #97 import contract → second benchmark/route → disjoint multi-benchmark validity study → APIs/views/dashboard and external claims.

---

Part II — Post-MVP program (~10 weeks, conditional)

The v2 phases survive with this round's corrections applied:

• Wave-1 roster is named at MVP week 1 and contains only EvalBench-route benchmarks. Wave 2 (ADK plugin route; OTLP route) gets explicit work items: extend the receiver's KNOWN_PRODUCTS allowlist with provenance/auth for a benchmark producer, define score mapping for non-EvalBench outcomes, and per-route fidelity tests — each route sized before it's promised.
• Corpus contract: every import is an immutable version + manifest; sample sizes derived from measured per-benchmark failure rates, not scenario counts (100 scenarios never guaranteed 50 failures). A benchmark that cannot produce the failed-session floor is dropped from the gate, stated up front — the gate does not slip.
• G1 (multi-benchmark): random per-benchmark samples estimate prevalence; prediction-stratified cases are diagnostics only; predeclared floors for human-human agreement, classifier-vs-human agreement, and unknown coverage; the taxonomy-revision branch has its own scheduled week rather than overlapping Phase 2's start.
• Localization at scale: the MVP's sealed-holdout discipline carries over — holdout traces disjoint from anything used to tune prompts or taxonomy; the baseline stays total over the same denominator; predeclared margins; unlocalized/missing_step/window-miss all first-class outcomes. Phase-2 window gets one week of slack for the Promote correction/anti-parroting trace analysis to core (turn tagging + sub_trajectories) #429 extraction (__init__.py conflict zone).
• G2 κ semantics, disambiguated: point estimate ≥ 0.6 and 95% CI lower bound ≥ 0.45 — deliberate: the point target claims quality, the floor rules out passing on noise at n≈150.
• Reference binding: benchmark scenarios have stable ids — bind expected answers by exact scenario-id join; match_golden_qa (merged feat(eval): golden Q&A matching in core — the producer for GOLDEN_EXPECTED_ANSWER (closes #428) #432) is reserved for genuinely fuzzy question text, avoiding its silent below-threshold drops.
• Flakiness: pass@3 = 0 is "no success observed in 3 trials" and permits a large true success probability; either a sequential retry policy sized to the probability being ruled out, or flakiness is excluded from claims.
• Labeler-load ledger (inside the D4 approval): MVP 40–60×2 + Part II holdout ~150×2 + diagnostic ~150×2 + span task ≈ 4–5 labeler-weeks total; the session counts are the shrink lever, protocols are not.
• Reporting: unchanged from v2 — categorical_views foundation, report-template + spec updated together, external claims gated on the validity gate.

Combined timeline if every gate passes: ~16 weeks (6 MVP + ~10 program), against v2's 14 — buying a real product at week 4 and a defensible investment decision at week 6, instead of learning at week 14 whether the core interaction is worth productionizing.

[caohy1988]

Review of plan v3 (MVP-first)

Verdict: ship it. This is the third round and the plan has converged — the MVP-first restructure is the right call, and every prior finding is resolved, several in ways better than what the reviews suggested (immutable versioned snapshots with a manifest are stronger than the staging-table fix; κ disambiguated as point ≥0.6 and CI lower bound ≥0.45 is exactly the right semantics; predeclaring the localization margin in week 1 closes the reverse-engineering hole).

Factual check: the new claim verifies — KNOWN_PRODUCTS = ("claude_code", "codex") at producers/src/bigquery_agent_analytics_tracing/otlp/receiver.py:53, enforced on the self-reported x-bqaa-source-product header (:191), which is why wave 2's "provenance/auth" work item is correctly sized as more than a projection. I can't see the claude.ai artifact (auth wall), so this review covers the issue comment only.

Remaining findings are few and small — none block starting week 1.

Findings

1. MINOR — the value gate is now the weakest metric, and it's the one that funds Part II. "≥70% of 30 investigations produce an accepted diagnosis or next action" is gameable in both directions: collaborators can pick investigations they already understand (inflating it) or, being close to the work, unconsciously anchor on the report (inflating "accepted"). Since this number — not κ — is the actual Part II funding argument, predeclare its protocol in week 1 alongside the other floors: investigations sampled randomly from the failed set (not self-selected), and "accepted" judged by someone other than the investigator, or operationally defined as "the next action was taken and not reverted."

2. MINOR — the sealed boundary-labeling task is missing from the labeler-load ledger. The ledger counts MVP 40–60×2 + Part II samples, but not the 30–50 sealed boundary labels measured at week 4 — which are span-level annotations, the most expensive per-item type in the whole plan, and land on the same two labelers mid-MVP. Add them to the ledger so the D4 approval covers the real total.

3. MINOR — "two independent judge runs" needs a definition of independent. Same model + same prompt at temperature > 0 is replicated, not independent — correlated failure modes (a systematically misleading trace pattern) survive it. Either state that the second run uses a different judge model (stronger, costs more) or name it honestly as a stability replicate and rely on the labeler study for validity — which the plan mostly does, so this is wording.

4. MINOR — the localization gate's decision rule under noise. At n=30–50 sealed labels, the 95% CI on a hit@1 difference is roughly ±20pp, so "baseline + 10pp" can pass or fail on noise. The CI is reported but not gated. Fine for a pilot — but predeclare the rule for when the point estimate clears +10pp and the CI includes zero, so week 6 doesn't improvise. (One line in the week-1 freeze.)

5. NIT — collaborator availability is an unstated critical-path dependency. The week-4 value gate needs three AF collaborators × 10 investigations on schedule. If their time slips, both the value gate and the localization need-signal slip. Worth one sentence of fallback (e.g., gate re-scopes to N completed investigations with the threshold applied proportionally).

6. NIT — S1's selection criterion should include score availability, not just trace fidelity. The hypothesis requires "one scored benchmark the reader already supports," and W0.4's contract needs populated scores tables; a benchmark with beautiful traces but sparse scores can't yield the ≥50-failure floor. Make "scores present and threshold-definable" an explicit S1 selection criterion alongside fidelity.

Positives worth naming

• The four predefined outcome branches — including two where stopping is the correct result — are what make this a real gate rather than a formality.
• Fail-closed D4 (synthetic/pre-redacted fallback) resolves the v2 contradiction without putting approvals on the critical path.
• Dropping pass@k from MVP claims and reframing 0/3 honestly removes an overclaim v1/v2 carried.
• Route honesty: wave 2's receiver work is now scoped (allowlist + provenance/auth + score mapping + fidelity tests) instead of an unestimated "projection."

No further rounds needed from my side unless the week-6 memo's numbers need a read. Good luck with the pilot.

Review by @caohy1988's assistant (Kimi Code CLI).

[caohy1988]

Fresh review of plan v3 — MVP direction approved, execution fixes still needed

I reviewed v3 from scratch against current main at 3fb6a00, while suppressing the v1/v2 findings that v3 already resolved. The factual re-baseline still holds, including the OTLP source-product allowlist. More importantly, the structural change is right: one EvalBench benchmark, a six-week value test, and localization only after demonstrated need is the first version I would fund.

My verdict is approve the MVP direction, but do not start the six-week clock until the P1 items below are resolved. These are new v3 execution/evidence issues, not repetitions of the old 14-week-plan objections.

P1 — changes required before MVP kickoff

1. The value gate does not test the stated hypothesis. “At least 70% produced an accepted diagnosis or next action” can pass even if collaborators perform equally well without the report. It also leaves “accepted” and the accepting actor undefined. The hypothesis says the report improves the next action, so use a counterfactual: record the intended action before opening the report and the action afterward, then score whether the report changed, materially narrowed, or accelerated it. Predeclare the rubric and have someone other than the investigator confirm that a counted action meets it.

2. The synthetic fallback cannot satisfy the real-investigation gate. Pre-redacted real traces can support the full MVP; fully synthetic traces cannot support 30 real investigations, the workflow-value claim, or the localization-need decision. Split the branch explicitly: synthetic data validates only ingestion, taxonomy mechanics, and run stability, then pauses before the value/localization gates until real-data clearance exists. Do not let a synthetic run produce a Part II funding recommendation.

3. Target-side atomic publication does not guarantee a consistent source snapshot. The current EvalBench reader performs separate sequential reads of results, scores, and configs (reader implementation). If the source job is still changing, an atomic target manifest can permanently bind events from one source state to scores from another. Require a completed/immutable source-job signal or one BigQuery snapshot timestamp across all source reads. Record source row counts and content fingerprints in the manifest; a changed fingerprint creates a new version rather than silently reproducing v1.

4. The four-column classification table conflicts with shipped persistence and drops provenance. Current categorical_results already stores metric/category, identity and scope, validation state, endpoint, execution mode, prompt version, and timestamp (existing schema); it does not store an evaluation-run id. A parallel (session_id, failure_category, justification, run_id) table creates a migration/dual-read path and omits data needed by D4. Reuse the existing table, add an immutable evaluation_run_id/manifest key, and bind it to source snapshot, taxonomy/prompt hashes, judge endpoint/model/mode, redaction-policy version, and D4 approval. Publish a run as complete only after every expected result row is validated. The taxonomy config needs no new judge engine, but this provenance contract does require additive persistence work.

5. The 50-failure floor cannot supply the plan's disjoint samples. The taxonomy study consumes 40–60 traces. Localization needs a development/headroom set and a final blind set; v3 currently uses the same 30–50 sealed labels to decide whether to build and later to score the result. Once those labels or baseline results guide prototype decisions, week-6 scoring is no longer independent. Derive the corpus floor from explicit partitions: taxonomy/adjudication, localization development/headroom, final blind localization evaluation, and notebook exemplars. At the stated ranges this is closer to 100 failures than 50. Freeze the final set before any taxonomy/prompt/localizer tuning and never reveal it until the prototype is frozen.

6. Uncertainty is reported but does not affect either MVP pass decision. A point κ of 0.6 on 40–60 traces can have a weak lower bound; a +10 percentage-point localization result on 30–50 paired cases can also have a confidence interval crossing zero. Reporting those intervals while passing on the point estimates still permits a false green. Predeclare decision rules, not only reporting rules: a lower confidence floor for both κ measures; a paired uplift lower bound for localization; and an absolute useful hit@1 floor as well as the relative +10-point margin. Define localization coverage as localized / all sealed failed sessions, with unlocalized, missing_step, and window misses in the denominator.

7. The allowed taxonomy revision has no scheduled slot. Weeks 2–3 author and run the taxonomy; weeks 3–4 immediately label it; week 4 decides on localization. If the first stability check fails, a revision, regenerated reference, two reruns, and a fresh blinded sample cannot occur without wasting labels or moving the gate. Either reserve and show the revision/re-gate interval (and admit the MVP may take seven weeks), or make an initial taxonomy-gate miss end this six-week MVP and schedule revision as a separate iteration.

8. D4 omits report consumers, and the judge path has an untested integrity boundary. The week-1 approval names the data, judge path, and labelers, but three collaborators receive exemplar and raw-trace drill-down access. Add the named report consumers, per-user view/table grants, notebook/export restrictions, and the stop/go memo to D4. Separately, current generation concatenates the full trace directly after the judge prompt (judge query); two runs will consistently repeat the same prompt-injection failure. Treat trace text as delimited untrusted data, schema-validate labels, require evidence to bind to trace substrings, escape all notebook rendering, and add instruction-bearing/malicious-markup traces to the MVP tests.

9. One favorable benchmark cannot authorize the entire Part II program. Week 1 currently selects by trace fidelity, which favors the easiest import rather than the most representative debugging work. Predeclare a benchmark rubric covering collaborator relevance, failure-mode coverage, score quality, and trace fidelity. Even if all MVP gates pass, fund only the durable import contract plus a second-benchmark transfer study. Fund additional routes first, and APIs/views/dashboard/external claims later, only when their own preceding validation gate passes.

P2 — tighten the decision tree and scope accounting

• Make localization need measurable. Require a threshold such as a majority of participating collaborators, backed by recorded investigations where the category report could not select a next action and boundary evidence would have changed it. A stated preference alone is not enough.
• Add the missing negative outcome: classification quality passes but the workflow-value gate fails. The predeclared result should be no localization and no Part II productization funding; keep the analysis and investigate why the report did not change actions.
• Move the future wave roster out of MVP week 1. Week 1 should name only the pilot benchmark. Name and size the additional roster after the week-6 decision.
• Do not present Part II as a firm ten-week estimate while route work is explicitly unsized. Derive a phase table for the EvalBench core and show ADK/OTLP route work separately until each is sized. The current “~16 weeks total” is not yet a bottom-up funding estimate.
• Complete the labeler ledger. Include the MVP boundary-label task, labeler calibration, adjudication, and a count for the Part II span task; these are both capacity and D4-access inputs.

What landed well

The prior round's central changes are present and coherent: MVP-first scope, the EvalBench-only initial route, D4 before real-trace use, one event+score manifest, random prevalence sampling, explicit abstention, a total localization baseline, boundary-level terminology, coverage-preserving windows, and staged stop outcomes. I found no regression in those decisions.

The independent cross-model pass corroborated the three most important new findings: the value gate needs a counterfactual, the synthetic branch cannot produce real-investigation evidence, and the failed-session floor does not support the required disjoint samples.

After the P1 changes above, I would stop iterating on the high-level plan and begin the week-1 benchmark/D4/source-snapshot spike. Further broad document review is likely lower value than executing that spike and reviewing its concrete manifest, sampling protocol, and metric preregistration.

[caohy1988]

Independent review of plan v3 (MVP-first)

I read the RFC and the full comment thread through v3. Direction is right: one benchmark, EvalBench first, taxonomy evidence before localization, and stopping is an allowed outcome. I would not start the six-week clock yet.

The RFC's closing question (should Phase 2 precede taxonomy validation?) is settled: no. Don't relitigate it.

P0 — v3 can succeed without touching the RFC

The RFC is a collaboration on KramaBench, LakeQA, and Spider 2.0. Those names do not appear anywhere in this repo. #97's worked example is datasets/gemini-cli-tools/. v3 then restricts wave 1 to EvalBench-only. If the partner agents are not EvalBench-hosted, the MVP quietly becomes "classify gemini-cli-tools failures," which does not validate AgentForensics.

AgentForensics is also unnamed as a party in this thread. Publicly, the closest published work on exactly KramaBench + LakeQA is SANA (Columbia, Wijaya / Liu / Wang / Wu). It already has:

• a failure taxonomy (Table 6: planning, wrong source, execution/computation, incomplete evidence, turn-waste, finalization, tool blockers)
• a two-stage LLM audit (extract turn-level evidence, then label) — this is basically localize_failures()
• empirical proof that one taxonomy does not travel: LakeQA is search-bottlenecked, KramaBench is analysis-bottlenecked

SANA's LakeQA/KramaBench runtime is Strands, not ADK or EvalBench. That is a D1 fact, not a week-6 discovery.

If AgentForensics is SANA (or should start from it), W1.1 is "encode Table 6," not a blank config. If it is not, the RFC still needs a sentence on why this work is not duplicating that paper on the same two benchmarks.

KramaBench also answers RFC question 5: it ships ground-truth code and intermediate-step checks. LakeQA has gold sources and subquestions. Spider 2.0 is a different problem (text-to-SQL once the DB is known). That should drive S1, not "whichever job imports cleanest."

Week 1 is: name the partner, the taxonomy draft, and the actual pilot job, and confirm that job is EvalBench. If it isn't, the MVP route is wrong and six weeks go into the wrong substrate.

P1 — agree with the latest v3 review, in this order

1. D4 stays in front of real traces. A synthetic run must not fund Part II.
2. Source-side snapshot, not just an atomic target manifest. EvalBenchRun.from_bigquery does sequential reads of results, scores, and configs. If the source job is still mutating, an atomic target publish can permanently bind events from one source state to scores from another. Require a completed/immutable source-job signal (or one snapshot timestamp across all three reads) and fingerprint those counts in the manifest.
3. Reuse categorical_results, plus an evaluation_run_id / manifest key. Do not stand up a four-column side table. The shipped schema already has identity, validation state, endpoint, execution mode, prompt version, and timestamp; a parallel table creates a dual-read path and drops D4 provenance.
4. Raise the failed-session floor so taxonomy labels, localization headroom, and a sealed blind set are disjoint. 50 is too small once those partitions exist.
5. Value gate needs a counterfactual. Record the intended action before opening the report and the action after; score whether the report changed, narrowed, or accelerated it. "Accepted diagnosis" judged by someone other than the investigator. A stated preference is not enough to fund localization.

Smaller, still real:

• Treat judge-bound trace text as delimited untrusted data (the categorical evaluator concatenates the full trace after the prompt). Schema-validate labels; require evidence to be a substring of the rendered window; add instruction-bearing traces to MVP tests.
• Reserve a taxonomy-revision week, or end this six-week MVP on a miss and schedule revision as a separate iteration. Weeks 2–4 as written have no slot for "revise, rerun, relabel."
• Bind benchmark references by exact scenario-id join. Reserve match_golden_qa for genuinely fuzzy question text; it silently drops below-threshold matches.
• Predeclare the localization decision rule for the noisy-small-n case (point estimate clears +10pp but CI includes zero) in the week-1 freeze, not at week 6.

Process

Stop iterating the plan in comments. Land v3 plus these P0/P1s as a short design-doc PR, update the issue body to point at it, and park #436 until the taxonomy actually freezes. Get the AgentForensics people on this issue. A collaboration RFC with only @haiyuan-eng-google / @caohy1988 in the thread is planning in a vacuum.

One-liner: approve MVP-first, but week 1 is "name the partner job and confirm it is EvalBench," not "start the six-week clock."

Review by @caohy1988 (Github Bot CHY1988).

[caohy1988]

Execution & implementation plan — FINAL (v4)

This finalizes the plan after three rounds of review. I verified this round's new claims before accepting them: KramaBench, LakeQA, and Spider 2.0 appear nowhere in this repo (grep across *.py/*.md); the SANA paper is real and is exactly adjacent work — LakeQA + a converted KramaBench, a seven-category failure taxonomy (task/planning, wrong source, execution/computation, incomplete evidence, turn-waste, finalization, tool blockers), a two-stage LLM audit, and the Strands runtime, which is neither ADK nor EvalBench; the shipped categorical_results DDL matches the reviewer's description (identity, validation state, endpoint, execution mode, prompt version — and no run id column); and the EvalBench reader does three sequential source reads with no snapshot consistency. All accepted.

Per the process feedback, this is the last plan iteration in comments. The next artifact is a design-doc PR (docs/agentforensics_mvp_plan.md) landing this text; the issue body will point at it, and the concrete week-0 outputs (manifest, sampling protocol, preregistration) are what get reviewed next — not further drafts of this document.

---

Week 0 — the pre-clock gate (P0)

The six-week clock does not start until all of these clear. They are cheap, and every one of them changes what the six weeks build:

1. Name the partner and its relationship to SANA. If AgentForensics is or builds on SANA, taxonomy v0.1 starts from SANA's seven categories, not a blank page. If it is not, the RFC gains a sentence on why this isn't duplicating published work on the same two benchmarks. Either way the partner joins this thread — a collaboration plan with one org in it is planning in a vacuum.
2. Confirm the pilot job's runtime and route. SANA's harness is Strands. If the partner's benchmark runs are not EvalBench-hosted, the EvalBench-only MVP route is wrong and D1 gets re-decided before any clock starts — not discovered at week 6.
3. Select the pilot benchmark by a predeclared rubric, not import convenience: collaborator relevance, failure-mode coverage, score availability and threshold-definability, ground-truth depth (KramaBench ships step-level checks; LakeQA has gold sources — this decides how much of RFC question 5 is answerable), then trace fidelity.
4. Approve the D4 boundary for exactly this data — now including the named report consumers (the collaborators get exemplar and drill-down access), per-user grants, notebook/export restrictions, and the stop/go memo itself as governed artifacts. Fail-closed: no clearance → the pilot runs on pre-redacted real traces or pauses; a fully synthetic run validates ingestion, taxonomy mechanics, and stability only and can never produce a Part II funding recommendation.
5. Freeze the preregistration doc: every floor, margin, and decision rule below, including the value-gate rubric and the noisy-small-n localization rule. Nothing in it changes after week 1.

The MVP (six weeks + one reserved revision week)

Hypothesis (unchanged): for one scored benchmark on the EvalBench route, BQAA can produce a stable, human-usable failure-category breakdown that demonstrably changes collaborators' next debugging action — and, only if they measurably need it, event-boundary evidence.

Week	Deliverable	Exit criterion
1–2	Source-consistent immutable snapshot: require a completed/immutable source-job signal or one BigQuery snapshot timestamp across the results/scores/configs reads; record row counts and content fingerprints in the manifest — a changed fingerprint mints a new version, never a silent v1 reproduction. failed_sessions view per the W0.4 contract.	≥100 reproducible failed sessions, pre-partitioned and disjoint: P-tax (40–60, taxonomy study), P-dev (localization headroom/development), P-blind (sealed final evaluation — frozen now, untouched until week 6), P-ex (notebook exemplars).
2–3	Taxonomy v0.1 (seeded per week-0 item 1) + unknown; two stability replicates — named honestly: same judge model, so they measure consistency, not validity; validity comes from the labeler study. Results land in categorical_results with an added immutable evaluation_run_id binding source-snapshot version, taxonomy/prompt hashes, judge endpoint/model/execution mode, redaction-policy version, and D4 approval — no parallel four-column side table. A run is complete only when every expected row is validated. Session-level notebook ships.	Replicate agreement ≥80%; non-unknown coverage ≥80%. A miss invokes the reserved revision week (one revision, fresh labels, re-gate); a second miss ends the MVP with the analysis as the deliverable.
3–4	Blinded two-labeler study on a random P-tax sample. Counterfactual value study: investigations randomly assigned from the failed set (not self-selected); each collaborator records the intended next action before opening the report and the action after; a non-investigator adjudicates whether the report changed, materially narrowed, or accelerated the action.	Human-human κ and classifier-vs-adjudicated κ: point ≥0.6 and 95% CI lower bound ≥0.45 (decision rule, not reporting). Value gate: ≥ the preregistered fraction of investigations where the report changed or narrowed the action, adjudicated — a stated preference or unverified "acceptance" counts for nothing.
4	Localization go/no-go. Need must be measurable: a majority of participating collaborators, backed by recorded investigations where the category report could not select a next action and boundary evidence would have. Headroom: the total first-error baseline (an output for every failed session) measured on P-dev against the preregistered margin.	Build only if classification passed, need is evidenced, and headroom exists. New explicit branch: classification passes but the value gate fails → no localization, no Part II productization funding; keep the analysis and investigate why the report didn't change actions.
4–5	Internal boundary prototype: chronological, coverage-preserving overlapping windows; boundary_id + target_kind: imported_event; all tuning on P-dev only. Judge-integrity hardening ships here: trace text delimited as untrusted data in the prompt, schema-validated labels, evidence bound to rendered-window substrings, escaped notebook rendering, and instruction-bearing/malicious-markup traces in the MVP test set (the shipped evaluator concatenates raw trace after the prompt — two replicates repeat the same injection failure, so this is not optional).	Every failed session in scope gets localized, unlocalized, or missing_step; window-selection misses are outcomes; no metric conditions on successful localizations only.
6	Score once on P-blind; write the stop/go memo (quality with CIs, counterfactual value results, cost/latency vs the week-0 budget, error analysis, funding recommendation).	Preregistered decision rules: coverage = localized / all P-blind failed sessions ≥70%; paired hit@1 uplift over the total baseline with CI lower bound >0; point uplift ≥ +10pp; absolute hit@1 ≥ the preregistered floor. Point-clears-but-CI-spans-zero resolves per the week-0 rule, not week-6 judgment.
7	(Reserved) taxonomy revision slot only — used or returned.	—

Labeler ledger (complete, inside the D4 approval): P-tax 40–60 × 2 + adjudication; 30–50 sealed boundary labels (the most expensive per-item task in the plan — previously missing from the ledger) + calibration; counterfactual-study adjudication. ≈2 labeler-weeks in the MVP. Collaborator availability is a named critical-path dependency: if investigation volume slips, the value gate re-scopes to completed investigations with the threshold applied proportionally — it does not silently pass.

Part II — staged funding, no aggregate estimate

The "~10 weeks / ~16 total" framing is withdrawn — route work is explicitly unsized, so a bottom-up number would be fiction. Each stage is sized when the previous stage's gate passes, and the wave roster is named after week 6, not week 1:

• Stage A — durable Add EvalBench bridge: import EvalBench BigQuery runs into a BQAA-owned mirror table #97 import contract (manifested versions, both CLIs, multi-benchmark hardening).
• Stage B — second-benchmark transfer study: does the taxonomy travel? SANA's own evidence says LakeQA is search-bottlenecked and KramaBench analysis-bottlenecked — one taxonomy may not transfer, and this is where that's measured, on a disjoint corpus with the same preregistration discipline.
• Stage C — additional ingestion routes, each sized on its own: ADK plugin; OTLP (receiver KNOWN_PRODUCTS allowlist + provenance/auth + score mapping + fidelity tests); Strands if the partner needs it (likely, per SANA).
• Stage D — public APIs (localize_failures()), Promote correction/anti-parroting trace analysis to core (turn tagging + sub_trajectories) #429 promotion, categorical_views expansion, dashboard, corpus publication, external claims — each behind its predecessor's validation gate. Reference binding by exact scenario-id join throughout; match_golden_qa only for genuinely fuzzy text.

#436 stays parked until the taxonomy freezes (end of a passed Stage B, not before).

Standing invariants (unchanged from v3, restated once)

D4 precedes any real-trace judge call or labeler access and covers derived data; evaluator fails closed. Immutable versioned imports everywhere. Random samples for prevalence, stratified only for diagnostics. Sealed sets frozen before tuning and used once. returncode == 0 means completed, not passed. Stopping is a success mode of the plan, not a failure of it.

---

Next step: I'll land this as docs/agentforensics_mvp_plan.md in a design-doc PR and update the issue body to point at it. Week-0's five items are the only work in flight until that PR merges. Thanks to all three reviewers — the plan that survives this thread is substantially harder to fool than the one that entered it.

[caohy1988]

EM status — locking v3, starting slice 1

Acting as EM with a 3-agent loop on this issue:

• Implementer: Claude Code
• Reviewers: Codex + Kimi Code (after the slice lands)
• Plan of record: v3 MVP-first. We are not starting the full 6-week clock or Part II.

P0 still blocking the pilot, not this code slice

Week 1 of the actual MVP still needs a named partner job and confirmation it is EvalBench-hosted (see the SANA / KramaBench / LakeQA note). Slice 1 does not pretend to resolve that.

Slice 1 (in progress now)

The unblocked #97 piece the MVP denominator depends on:

1. EvalBenchRun.materialize() — BQAA-owned event + score tables, atomic import version + manifest (no delete-then-append)
2. Failed-session contract — returncode == 0 means completed, not passed; view joins pinned import version
3. Unit tests + docs/evalbench.md

Out of this slice: taxonomy content, localization / #429, KramaBench/LakeQA/Spider harnesses, dashboards, live traces, D4 approvals.

Branch: feat/435-evalbench-mvp-snapshot off upstream/main @ 3fb6a00 (local worktree). PR after implementer + both reviewers.