Feature: Diff Mode
replay diff session-a.json session-b.json
Compare two eval runs or transcripts side-by-side. Show where tool call sequences diverge, which turns differ, and highlight the critical decision point where one model succeeded and another failed.
Motivation
When analyzing Arena progression results across models (e.g., GPT-5.1-Codex vs Sonnet 4 on the same eval prompt), the #1 manual task is figuring out what the weaker model did differently. Currently this means opening two JSON files and reading them in parallel.
Suggested Output
- Turn-by-turn alignment showing matching vs divergent turns
- Tool call diff (which tools were called, in what order)
- Summary: "Diverged at turn 5: Model A called maestro_subscriptions, Model B tried gh CLI"
Use Cases
- Cross-model eval comparison in Arena progression pipeline
- Before/after comparison when a skill prompt changes
- Regression hunting: "what changed between v1 and v2 of this skill?"
Related
Feature: Diff Mode
replay diff session-a.json session-b.jsonCompare two eval runs or transcripts side-by-side. Show where tool call sequences diverge, which turns differ, and highlight the critical decision point where one model succeeded and another failed.
Motivation
When analyzing Arena progression results across models (e.g., GPT-5.1-Codex vs Sonnet 4 on the same eval prompt), the #1 manual task is figuring out what the weaker model did differently. Currently this means opening two JSON files and reading them in parallel.
Suggested Output
Use Cases
Related
--summarymode (issue Add machine-readable output modes for agent consumption (--json, --summary) #5)