Persist span-level G1 labels on the native snapshot (queryable localization)

[caohy1988]

Parent: #435 (AgentForensics RFC — automated failure taxonomy and span-level trace auditing). This is the next engineering slice after span-level G1 localization #466 (PR #467 library + PR #468 presenter e2e), and the native-writer analog of #463: publish already-landed localization as pinned snapshot rows.

Sequencing

• docs(evalbench): recordable MVP e2e demo of import → failed-sessions → score (#435, #97) #455 is merged. docs(435): record merged slices 451–455 and snapshot changelog #456–docs(evalbench): consolidate native import and span-G1 walkthroughs #468 are open Week-0 freeze / native-writer / span-G1 evidence, agent-review-complete (or stacked on it). They must not be merged as part of this slice, and this slice must not wait for org APPROVE on them.
• The future PR must be stacked on feat(evalbench): span-level G1 taxonomy on span_id (#466) #467 head a7fd389 — not on docs(evalbench): consolidate native import and span-G1 walkthroughs #468's demo commits, not on feat(evalbench): native agent_events freeze e2e (#463) #465, not on feat(evalbench): Week 0 freeze e2e team demo on the widget-stock session (#435) #462, and not on main. GitHub showing the feat(evalbench): mechanical failure-taxonomy scaffold from failed-session flags (#435) #458–feat(evalbench): span-level G1 taxonomy on span_id (#466) #467 file set against main is expected stacking, not a finding.
• G1 v0.1.0 was frozen in feat(evalbench): freeze Week 0 partner, D4, and G1 taxonomy v0.1 (#435) #461 (failure_taxonomy.py, g1_frozen: true).
• The native agent_events writer landed in feat(evalbench): native agent_events snapshot writer (#463) #464 (head f5c6109): NativeAgentEventsRun publishes events / scores / manifest / failed_sessions from production ADK agent_events with no EvalBench source tables in the path.
• The span-level G1 library landed in feat(evalbench): span-level G1 taxonomy on span_id (#466) #467 (head a7fd389, dual Codex+Kimi LGTM, P0=0 P1=0, CI green): span_taxonomy.py with label_native_run / label_failed_session_spans / SpanFailureLabel.as_tuple().
• The span-G1 presenter e2e landed in docs(evalbench): consolidate native import and span-G1 walkthroughs #468 (head 0778125, dual LGTM, P0=0 P1=0, CI green) — docs/script/tests only; no src/, no BigQuery write. docs(evalbench): consolidate native import and span-G1 walkthroughs #468 stays presenter-only.

The gap

Phase 2 localization currently exists as a library, not as snapshot data. #467 already emits the RFC shape (trace_id, span_id, failure_category, evidence, confidence) for the widget-stock silence session — inspectable in Python tests and a --fixture presenter, but not part of the BQAA-owned snapshot contract. Session-level G1 is queryable via failed_sessions; span labels are not.

Do not reimplement span_taxonomy or invent a new taxonomy or a second localizer. The missing piece is publication: NativeAgentEventsRun.materialize (or a thin sibling that inherits it) should write a BQAA-owned span-labels table keyed by the frozen pin (job_id, import_version), joinable to failed_sessions via the frozen eval_id rule.

Session-level failed_sessions + G1 remains the denominator. Span rows localize; they never classify or replace it.

Goal

On a native snapshot of the widget-stock session 7e352c34 (full id 7e352c34-4c1c-4395-acd5-fb3c8f215346), the published span-labels rows make the AGENT_STARTING silence inspectable as data:

• real span_id b7ad6b7169203331, target_kind="gap_after_span";
• the frozen categories task/planning, finalization, tool blockers;
• evidence that no subsequent TOOL_STARTING / check_inventory / AGENT_COMPLETED occurred.

Source of truth is production ADK agent_events (native path). The EvalBench adapter (#97, evalbench-import) stays optional; this slice must not require EvalBench source configs / results / scores.

Reuse label_native_run with the frozen EvalScorePolicy({"goal_completion": 1.0}) — per the #468 P1 finding, the default empty policy drops task/planning.

Acceptance (widget-stock, fixture-testable, no live BigQuery)

• Session-level G1 for 7e352c34 still yields task/planning, finalization, tool blockers — unchanged.
• Published span-label rows for that session are exactly the three library rows (same span/trace ids, categories, gap_after_span, evidence strings) and carry (job_id, import_version) with job_id=mvp-e2e-real-traces.
• Identity: eval_id = session_id[:8] when unique in the snapshot / join set; full session_id on first-8 collision. Span rows must stay joinable to the same eval_id that failed_sessions / G1 already uses.
• No synthetic span identifiers. Rows without a real span_id fail closed.
• Frozen G1 names only (v0.1.0). No SANA fork, no new category strings. If a test proves a span cannot be labeled with the frozen three, preserve the failing case and defer the taxonomy change to a separate issue.
• Offline unit tests only: no live BigQuery, no --synth, no live/LLM judge, no production agent_events writes.
• Prefer inheriting EvalBenchRun.materialize / NativeAgentEventsRun rather than a parallel publisher. A thin CLI extension of evalbench-native-import is allowed; do not invent a new command family.

Guardrails

• Clock OFF. This issue does not start the six-week clock, seal preregistration, or kick the Week 1 snapshot job.
• D4 fail-closed: Hai-Yuan Cao only.
• Never write production agent_events.
• Do not delete or rewrite evalbench-import (Add EvalBench bridge: import EvalBench BigQuery runs into a BQAA-owned mirror table #97).
• Do not merge docs(435): record merged slices 451–455 and snapshot changelog #456–docs(evalbench): consolidate native import and span-G1 walkthroughs #468.
• Do not start from main; stack on feat(evalbench): span-level G1 taxonomy on span_id (#466) #467 head a7fd389.
• Do not restore turn_index — it is omitted on purpose in feat(evalbench): span-level G1 taxonomy on span_id (#466) #467 until Promote correction/anti-parroting trace analysis to core (turn tagging + sub_trajectories) #429 exposes an importable full-conversation mapping. Do not fork Promote correction/anti-parroting trace analysis to core (turn tagging + sub_trajectories) #429's coordinate.

Out of scope

• Merging docs(435): record merged slices 451–455 and snapshot changelog #456–docs(evalbench): consolidate native import and span-G1 walkthroughs #468
• Preregistration seal / clock start / Week 1 job
• Dropping the EvalBench adapter (Add EvalBench bridge: import EvalBench BigQuery runs into a BQAA-owned mirror table #97)
• SANA fork / new taxonomy names
• Live judge as denominator
• Public labeled corpus / Phase 3 golden set
• Benchmark harness (Phase 0)
• github.io / Pages demo updates
• Re-adding turn_index (Promote correction/anti-parroting trace analysis to core (turn tagging + sub_trajectories) #429)

Related

• RFC: AgentForensics collaboration — automated failure taxonomy and span-level trace auditing over BQAA traces #435 (parent RFC)
• Span-level G1 taxonomy on span_id (AgentForensics RFC gap) #466 (span-level G1 library issue)
• Native agent_events snapshot writer for AgentForensics (EvalBench-adapter exit ramp) #463 (native agent_events writer)
• Promote correction/anti-parroting trace analysis to core (turn tagging + sub_trajectories) #429 (turn tagging / sub_trajectories; do not fork it)
• PRs feat(evalbench): freeze Week 0 partner, D4, and G1 taxonomy v0.1 (#435) #461, feat(evalbench): native agent_events snapshot writer (#463) #464, feat(evalbench): span-level G1 taxonomy on span_id (#466) #467, docs(evalbench): consolidate native import and span-G1 walkthroughs #468
• Demo (native path, EvalBench-format snapshot; not updated by this issue): https://caohy1988.github.io/evalbench/

[caohy1988]

Reviewer: Kimi
Review-round: 1

Verdict: ISSUE_OK — P0: 0, P1: 0, P2: 2 (informational).

Customer-object check — PASS

The customer-facing object is exactly right: published span-label rows on the native snapshot for widget-stock session 7e352c34-4c1c-4395-acd5-fb3c8f215346, with real span_id b7ad6b7169203331, target_kind="gap_after_span", keyed by (job_id, import_version) and joinable to failed_sessions via the frozen eval_id rule. The Goal section leads with "the published span-labels rows make the AGENT_STARTING silence inspectable as data" — queryable localization, not a library re-run and not an EvalBench scenario.

Denominator preserved: "Session-level failed_sessions + G1 remains the denominator. Span rows localize; they never classify or replace it." Acceptance line 1 re-asserts session-level G1 unchanged. This is the correct framing — no span-labels-as-denominator drift.

Frozen taxonomy honored: the issue pins task/planning, finalization, tool blockers (v0.1.0, g1_frozen: true), explicitly forbids new category strings / SANA fork, and adds the fail-closed escape hatch ("preserve the failing case and defer the taxonomy change to a separate issue") rather than inviting taxonomy invention. Spot-checked against docs/week0_g1_taxonomy.md in the freeze worktree — the names and the session's frozen triple ("task/planning", "finalization", "tool blockers") match.

EvalBench-in-customer-sentences check — PASS

• Title: "Persist span-level G1 labels on the native snapshot (queryable localization)" — native snapshot and queryable localization lead; EvalBench absent.
• Goal: leads with the session and the span rows. EvalBench appears only as the negative constraint ("must not require EvalBench source configs / results / scores") and the explicit "The EvalBench adapter (Add EvalBench bridge: import EvalBench BigQuery runs into a BQAA-owned mirror table #97, evalbench-import) stays optional."
• Acceptance punchline: session G1 unchanged + the three published span rows with real ids. No "import an EvalBench job" framing anywhere a PM/demo reader would land.

A PM reading title → goal → acceptance would conclude the product is "query the span that went silent on the native snapshot," not "run EvalBench."

Guardrail coverage — all present

Clock OFF stated; D4 fail-closed Hai-Yuan Cao only; never write production agent_events; do not delete/rewrite evalbench-import (#97); do not merge #456–#468 (and "must not wait for org APPROVE" on them); stack on #467 head a7fd389 with the expected-stacking note; do not reimplement span_taxonomy or invent a second localizer; turn_index stays omitted with the #429 do-not-fork note. The EvalScorePolicy({"goal_completion": 1.0}) reuse note correctly carries forward the #468 P1 finding so task/planning isn't silently dropped.

Findings

No P0s. No P1s.

P2-1 (informational): The future span-labels table is described as "a BQAA-owned span-labels table" without a candidate name or dataset placement. Fine to defer to the PR, but one line naming intent (e.g., sibling of the native writer's manifest/scores tables) would make the eventual snapshot-contract doc diff smaller.

P2-2 (informational): The Related section's demo link is labeled "EvalBench-format snapshot; not updated by this issue." Correct and honest — but when a follow-up issue does refresh the demo to the native path, the customer sentence there should mirror this issue's framing (queryable span localization on 7e352c34), so the public artifact doesn't lag the positioning set here.

What can wait

• Table naming / dataset placement for the span-labels table (P2-1).
• Demo-site wording refresh (P2-2).
• Any taxonomy gap discovered by tests — the issue already routes this to a separate issue, fail-closed.

Ship the issue as written.

[caohy1988]

Reviewer: Codex
Review-round: 1

Verdict: ISSUE_OK

P0 count: 0
P1 count: 0
P2 count: 0

Findings against the ISSUE TEXT

None.

The issue body closes the required P0/P1 contracts:

• Publication and identity are explicit: the new rows are BQAA-owned, keyed by (job_id, import_version), and "joinable to failed_sessions via the frozen eval_id rule"; Acceptance fixes eval_id = session_id[:8] when unique and the full session ID on collision, and requires reuse of the same eval_id as failed_sessions / G1.
• Localization does not fork the landed library: "Do not reimplement span_taxonomy" and "Reuse label_native_run with the frozen EvalScorePolicy({\"goal_completion\": 1.0})."
• The published localization keeps real span identity and fails closed: the widget-stock target is b7ad6b7169203331 with target_kind="gap_after_span"; synthetic span identifiers are forbidden, and rows without a real span_id fail closed.
• The denominator stays session-level: "Session-level failed_sessions + G1 remains the denominator. Span rows localize; they never classify or replace it." A live/LLM judge is also expressly excluded.
• The widget-stock punchline is testable as published data: the issue requires the silence after AGENT_STARTING, including the absence of subsequent TOOL_STARTING, check_inventory, and AGENT_COMPLETED, while preserving exactly task/planning, finalization, and tool blockers.
• Source and compatibility boundaries are explicit: native production ADK agent_events is the source of truth; production agent_events is never a write target; the EvalBench adapter remains optional and must not be deleted or rewritten.
• Scope governance is closed: D4 is Hai-Yuan Cao only; the clock, preregistration seal, and Week 1 job remain off; docs(435): record merged slices 451–455 and snapshot changelog #456–docs(evalbench): consolidate native import and span-G1 walkthroughs #468 are not merged; no SANA fork or new G1 strings are allowed; and turn_index is not restored.
• The future PR base is pinned unambiguously to feat(evalbench): span-level G1 taxonomy on span_id (#466) #467 HEAD a7fd389, excluding docs(evalbench): consolidate native import and span-G1 walkthroughs #468 demo commits, feat(evalbench): native agent_events freeze e2e (#463) #465, feat(evalbench): Week 0 freeze e2e team demo on the widget-stock session (#435) #462, and GitHub main. The preferred implementation path is the existing native materialization/inheritance path with at most a thin extension of evalbench-native-import, not a new command family.

What can wait

Physical table naming/DDL details, the exact internal method split within the required native materialization path, and broader fixture coverage beyond the frozen acceptance examples can be resolved and reviewed in the future PR. They are not issue-text blockers as long as the contracts above and offline-only verification remain intact.

[caohy1988]

Implementation PR is up: #470 — unique commit 81fa2be stacked on #467 head a7fd389 (does not include #468/#465/#462; do not merge #456–#468). Opt-in span_labels_table on NativeAgentEventsRun.materialize + evalbench-native-import --span-labels-table, reusing label_native_run with the frozen EvalScorePolicy({"goal_completion": 1.0}) fallback. Widget-stock 7e352c34 publishes exactly the three library rows on real span b7ad6b7169203331 (gap_after_span); session-level failed_sessions + G1 unchanged; no synthetic span ids; adapter #97 stays; clock OFF. Offline suite: 4257 passed.