Repository navigation
feat: eval framework for benchmarking debate harness - #5
Merged
Merged
Conversation
Add programmatic eval pipeline to benchmark the debate engine against datasets with known answers (GSM8K, GPQA Diamond, MATH Level 5). Components: - Engine Run() — synchronous driver that loops through the action-dispatch cycle without TUI dependency (internal/engine/run.go) - CachingAPI — content-addressable file-based cache wrapping OpenRouterAPI, making re-runs cheap (~$1-3 vs ~$7 full) (internal/openrouter/cache.go) - Eval CLI — Cobra entrypoint with parallelism, cost limits, and caching (.eval/cmd/eval/) - Dataset curation scripts — one-time HuggingFace pulls for GSM8K (50), GPQA Diamond (50), MATH Level 5 (40) (.eval/scripts/) - Scoring script — numeric/letter/sympy matching (.eval/scoring/) - Justfile recipes — eval, eval-nocache, score, eval-curate
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the Use the checkbox below for a quick retry:
Note 🎁 Summarized by CodeRabbit FreeYour organization is on the Free plan. CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please upgrade your subscription to CodeRabbit Pro by visiting https://app.coderabbit.ai/login. Comment |
Base automatically changed from
claude/refactor-architecture-layers-U8mFG
to
main
February 12, 2026 06:03
Use cheap mode automatically for benchmarks to reduce costs, and switch dataset curation scripts to public HF mirrors (no auth token required).
Regex extraction operated on Round 5 vote responses (YES/NO), not the actual computed answers. Now ExtractAnswer reads Round 0 research responses and calls a flash model with JSON schema to extract the answer. Regex is kept as fallback. Verified on gsm8k Q0: 125 ✓
Major refactoring to reorganize debate engine architecture: - Remove old internal/debate and internal/engine packages - Add new debate/ and internal/prompt/ packages for cleaner separation - Extract provider logic to internal/openrouter/provider.go - Add session writer utility for output management Eval framework updates: - Replace GSM8K dataset with MMLU Pro (50 samples) - Update extraction and scoring logic - Improve test coverage and runner implementation TUI improvements: - Simplify model.go by consolidating message handling - Update view rendering for better clarity - Refresh golden snapshots for sim tests
Rewrote all three docs files to be scannable narratives focused on understanding the system, not implementation details. flow.md now walks through every LLM interaction pattern chronologically.
Move eval command from .eval/cmd/eval to cmd/eval to follow Go convention. Replace three separate benchmark datasets (GPQA, MATH, MMLU-Pro) with a single curated warroom.json (15 hard questions). Simplify extraction to type-based routing (multiple-choice vs open-ended) instead of benchmark-specific prompts.
Run all 6 model calls concurrently in rounds that only depend on prior-round data (Research, Challenge, Convergence, Synthesis). Rounds with same-round dependencies (Pitch, Vote) remain sequential. Adds BatchResult type, parallel dispatch in both sync runner and TUI, Parallel flag on RoundConfig, and mutex protection for sim test RNG.
diyor28
marked this pull request as ready for review
February 13, 2026 22:09
diyor28
pushed a commit
that referenced
this pull request
Feb 15, 2026
Core resilience: - #1: Add React ErrorBoundary wrapping the app — render errors now show a styled fallback with reload button instead of a white screen - #2: Wire onUpdateDownloaded from preload into useDebate state so the renderer can display an update-available banner - #4: Remove dead submitTopic from useDebate hook and DebatePage - #5: Add backend crash detection — main process forwards Go exit event via IPC, preload exposes onBackendExited, useDebate sets connected=false and shows error banner when backend dies - #8: Fix route param changes being silently ignored — remove stale connectedRef guard; useEffect now disconnects previous session and reconnects on topic/resume change - #11: Fix O(n²) activity scan per stream tick — walk backwards and break on first match instead of .map + .slice().some() per element - #19: Add beforeunload handler during active debates to prevent accidental window close / data loss Accessibility: - #6: Add ARIA attributes across all components — role="alert" on ErrorBanner, role="status" + aria-label on GlowDot, role="complementary" on ModelRoster, role="progressbar" on progress bars, aria-label on textarea and dialog, aria-checked on QuestionStep options - #12: Add focus management to AskUserOverlay — saves previous focus, auto-focuses first interactive element on mount, restores focus on unmount, implements keyboard focus trap (Tab/Shift+Tab) - #14: Surface JSON parse errors in AskUserOverlay — shows visible error state with dismiss button instead of silently fabricating a question - #15: Fix partial answer submission — check raw answers array for all-empty instead of checking formatted string against "(no answer)" UX polish: - #3: Delete dead LoadingState component (zero imports) - #7: Lower noise-overlay z-index from 50 to 1 to avoid stacking conflict with titlebar drag region - #9: Add "Open Folder" button to SessionListPage using the existing but previously unused openSessionsFolder preload API - #10: Extract renderTranscriptEvent and roleBadge as pure functions outside DebateDashboard component to avoid recreation on every render - #13: Replace hardcoded Tailwind colors in StatusBadge with theme tokens (text-success, text-error, text-warning) - #16: Add window.confirm() before shutdown to prevent accidental cancellation of in-progress debates - #18: Add refresh() to useSessionList hook, wire refresh button and visibilitychange auto-refresh to SessionListPage - #21: Remove defaultOpen from transcript Collapsibles in SessionDetailPage — long transcripts now load collapsed https://claude.ai/code/session_01QPxpz4R7j4MFwGAqLEkwpb
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Engine.Run()— synchronous programmatic driver that loops through the action-dispatch cycle without TUI dependencyCachingAPI— content-addressable file-based cache wrappingOpenRouterAPI, making eval re-runs cheap (~$1–3 vs ~$7 full).eval/cmd/eval/) with parallelism, cost limits, caching, and answer extractioneval,eval-nocache,score,eval-curateTest plan
go test ./internal/engine/...— 5 Run() tests passgo test ./internal/openrouter/...— 6 cache tests passgo test ./.eval/cmd/eval/...— 20+ extraction tests passgo test ./...— all existing tests still passjust lint— 0 issuespython .eval/scripts/curate_gsm8k.py→ produces dataset JSONjust eval --benchmark gsm8k --question-ids 0→ runs single questionjust score .eval/results/<run>.json→ scores results