Skip to content

feat: eval framework, session persistence overhaul, and web UI - #6

Open
diyor28 wants to merge 24 commits into
mainfrom
feat/eval-framework
Open

diyor28 wants to merge 24 commits into
mainfrom
feat/eval-framework

Conversation

@diyor28

@diyor28 diyor28 commented Feb 14, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Eval framework: LLM-as-judge scoring, baseline comparison, parallel execution, voting system for benchmarking the debate harness against public datasets (MMLU Pro, GPQA)
  • Session persistence overhaul: Single session.json per session with slug-based IDs (k7m2x9pa-should-we-adopt-rust) replacing timestamp dirs with 10+ scattered files. Preset-aware resume detection, thread-safe writer, legacy read fallback
  • Web UI: Real-time dashboard with WebSocket streaming, client-side routing (wouter), Vite build pipeline, ask-user overlay, completion summary, shared components
  • Infrastructure: Parallel LLM calls within debate rounds, cassette recorder enhancements, openrouter cache layer, debate.State mutex, simulation fault improvements

Test plan

  • go test ./... — all packages pass
  • golangci-lint run ./... — 0 issues
  • npx tsc --noEmit — TypeScript clean
  • Simulation tests pass (including golden snapshots)
  • Session resume with non-uniform round sizes (round 4 = 2 models) tested
  • Manual: run a debate via TUI, verify session.json created with correct structure
  • Manual: web UI renders session list and detail pages correctly
  • Manual: eval framework go run ./cmd/eval runs end-to-end

Summary by CodeRabbit

  • New Features

    • Launched web-based UI accessible via browser at localhost:8080
    • Added Electron desktop app with auto-updater and system tray integration
    • Introduced single-room chat debate format replacing multi-round staged structure
    • Added new decision evaluation benchmark with 20 scenarios and trap analysis
    • Implemented new tools: transcript search, calculator, voting readiness
  • Chores

    • Refactored session storage to transcript-first model
    • Added JSON-RPC stdio server for machine/Electron communication
    • Updated evaluation framework with multi-model judge scoring and bias mitigation metrics

- Replace duplicate Simpson's paradox (#4) with sunk cost fallacy trap
- Remove answer-revealing constraints from #0, #1, #2 (model must derive insights)
- Rebalance dominated options in #13 (cloud costs) and #14 (prioritization)
- Add ethical dimension to #6 (government censorship client)
- Add data ambiguity to #10 (contested Phase 1 biomarker subgroup)
- Add elapsed timer, round progress (models done/total), discovery turn
  counter, per-model costs, and tool-use indicators to the web UI
- Fix CostTracker pricing: add ModelIDLookup so TotalCost() resolves
  short keys (e.g. "gpt") to model IDs for correct pricing lookups
- Wire configPricingAdapter in all entrypoints (web, TUI, eval runner)
- Add StreamType to stream events to distinguish content vs tool calls
- Regenerate golden snapshots to reflect corrected cost values
The web handler dispatches actions in goroutines that mutate State
(Activities, LogLines, SessionDir) while the event loop concurrently
reads those fields. Add an RWMutex to State with locking in the
write methods (Log, AddActivity, AddModelActivity, AddSystemActivity,
SetError) and in ExecutePersist, updateActivityExcerpt, and sendState.

Also log a warning (once per model) when PricingProvider falls back
to default $2/$10 pricing for an unknown model ID.
forwardStreamEvents runs as a detached goroutine that sends to the
caller's eventCh. After close(internalCh), buffered items may still
be draining to eventCh. If the caller returns and ws.go closes
eventCh, the forwarder panics with "send on closed channel".

Wait for the forwarder to finish via a done channel before returning
from CallWithToolsStreaming and CallDiscoveryStreaming.
Replace conditional rendering in App.tsx with wouter routes (/, /debate,
/sessions, /sessions/:id). LandingScreen now uses navigate() instead of
callback props. Add SPA-aware static file handler that falls back to
index.html for unmatched paths in prod mode.
- Fix loadFromSessionData using hardcoded 6-model count for all rounds;
  round 4 (synthesis) has only 2 models, causing incorrect resume detection.
  Now uses preset-aware per-round speaking order.
- Remove dead code: ResumableSessionWriter, duplicate SessionLoader interface,
  orphaned WriteDiscovery method, hand-rolled joinStrings (use strings.Join).
- Fix WriteKnowledgeBase silently swallowing JSON unmarshal errors.
- Fix Writer.Data() returning raw pointer under mutex; now returns value copy.
- Add TestLoadState_NonUniformRounds covering the resume bug.
…ements

Session persistence: single session.json format with slug-based IDs,
updated debate runner, TUI model, web handlers, and frontend pages.

Eval framework: add LLM judge, baseline comparison, parallel execution,
voting system; remove old extract/curate scripts.

Web UI: add client components (ask-user overlay, completion summary,
shared Badge/GlowDot/Panel), Vite config, SPA routing, dashboard updates.

Infrastructure: cassette recorder enhancements, openrouter cache layer,
sim fault improvements, golden snapshot updates.
@coderabbitai

coderabbitai Bot commented Feb 14, 2026 •

Copy link
Copy Markdown

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

Walkthrough

This PR comprehensively redesigns the system from a multi-round debate framework to a single-room collaborative chat model, introduces web and Electron UI support, restructures the evaluation framework from War Room to Decision scenarios, and refactors session and prompt infrastructure to support transcript-first state management.

Changes

Cohort / File(s) Summary
Evaluation Framework Overhaul
.eval/datasets/decisions.json, .eval/datasets/warroom.json, .eval/eval_framework.md, .eval/requirements.txt, .eval/scoring/score.py, .eval/scripts/curate_warroom.py, .eval/LEARNINGS.md
Replaces War Room 15-question benchmark with a new 20-scenario Decision benchmark across 5 categories. Removes dependencies, eliminates per-question scoring logic, and introduces comprehensive metrics (trap accuracy, cost efficiency, inter-judge kappa) and multi-mode evaluation (debate, single, voting). Adds learnings document on calibration and hardness approaches.
Baseline & Voting Runners
cmd/eval/baseline.go, cmd/eval/voting.go, cmd/eval/runner.go
Introduces SingleModelRunner and VotingRunner for independent model analyses with synthesis. Refactors RunAll to RunAllScenarios, replaces question-centric logic with scenario processing, and consolidates debate output into buildDebateOutput.
Evaluation Dataset & Judging
cmd/eval/dataset.go, cmd/eval/judge.go, cmd/eval/result.go, cmd/eval/eval_test.go
Replaces DatasetQuestion with DecisionScenario model including Trap metadata. Removes extract.go entirely. Introduces comprehensive Judge type with multi-model scoring, inter-judge kappa, trap evaluation, and aggregate EvalMetrics. Adds extensive unit tests.
Evaluation CLI & Main
cmd/eval/main.go, cmd/eval/parallel.go
Adds mode validation (debate, single, voting, all), conditional judge execution via --no-judge flag, scenario-based workflow replacing question-based, and parallelized judging. Introduces parallel runner for concurrency control.
Chat Room Engine
debate/steps/chat_room.go
New 568-line single-room debate orchestration with finite-state workflow for turns, tool execution (ask_user, calculate, transcript_search, start_vote), user interaction, and hidden-ballot voting. Replaces multi-round debate pipeline.
Debate Core API Updates
debate/action.go, debate/prompt.go, debate/provider.go, debate/calc_eval.go
Adds CallModelTurnAction and ForceVoteResult, extends PromptBuilder with RoomSystemPrompt/RoomUserPrompt/VotePrompt/VoteOptions, adds ToolCalls and ToolCallID to Message, and introduces arithmetic expression evaluator (EvaluateExpression).
Session Management Refactor
internal/session/data.go, internal/session/session.go, internal/session/writer.go, internal/session/session_test.go, debate/steps/loadsession.go
Introduces transcript-first SessionData model with ChatData payload. Replaces Dir-based writes with mutex-protected in-memory state flushed to session.json. Adds DefaultLoader.LoadChat and structured data loading pipeline. Updates LoadSessionAction to use ChatSessionSnapshot.
Web Server & Infrastructure
internal/web/server.go, internal/web/handler.go, internal/web/dispatch.go, internal/web/protocol.go, internal/web/ws.go, internal/web/embed.go, internal/web/embed_dev.go, internal/web/vite.go
Complete HTTP+WebSocket+static server with REST APIs for config/sessions, WebSocket debate engine, streaming event handling, and dual-mode embedded/Vite dev support. Implements DispatchAction for action routing and StateSnapshot protocol.
JSON-RPC Server for Electron
internal/jsonrpc/protocol.go, internal/jsonrpc/server.go, internal/jsonrpc/server_test.go, internal/jsonrpc/session.go, cmd/warroom/stdio.go
New JSON-RPC 2.0 stdio server for Electron IPC. Implements protocol types, request dispatch, session management, and comprehensive unit tests for debate methods (initialize, submitTopic, answerUser, forceVote) and config/session queries.
Electron Desktop Application
electron/forge.config.ts, electron/package.json, electron/webpack.*.ts, electron/src/main/go-bridge.ts, electron/src/main/index.ts, electron/src/main/ipc.ts, electron/src/main/menu.ts, electron/src/main/tray.ts, electron/src/main/window.ts, electron/src/main/updater.ts, electron/src/preload/index.ts
Full Electron desktop app with Forge configuration, main process lifecycle management, Go backend bridge over JSON-RPC, IPC handlers, native menu/tray integration, auto-updater, and security-isolated preload.
Web React Frontend
web/src/main.tsx, web/src/App.tsx, web/index.html, web/package.json, web/vite.config.ts, web/tsconfig.json, web/src/index.css, web/src/lib/models.ts, web/src/types/protocol.ts, web/src/hooks/useDebate.ts, web/src/hooks/useElapsedTimer.ts, web/src/pages/*.tsx, web/src/components/**/*.tsx
React SPA with Vite, Tailwind CSS, and wouter routing. Includes debate lifecycle management via WebSocket hook, multi-page layout (landing, debate, sessions), real-time dashboard with activity feed, model roster, phase tracking, and interactive ask-user overlays.
Electron Renderer Frontend
electron/src/renderer/src/main.tsx, electron/src/renderer/src/App.tsx, electron/src/renderer/src/index.css, electron/src/renderer/src/lib/models.ts, electron/src/renderer/src/types/protocol.ts, electron/src/renderer/src/hooks/useDebate.ts, electron/src/renderer/src/hooks/useSessions.ts, electron/src/renderer/src/hooks/useElapsedTimer.ts, electron/src/renderer/src/hooks/useNotifications.ts, electron/src/renderer/src/pages/*.tsx, electron/src/renderer/src/layouts/PageShell.tsx, electron/src/renderer/src/components/**/*.tsx
Electron renderer using same React architecture as web frontend but with electronAPI IPC bridge instead of WebSocket. Includes session loading via useSessionDetail/useSessionList, debate UI, and notification integration.
Prompt & Tool Framework
internal/prompt/default.go, internal/prompt/preset.go, internal/prompt/pipeline.go, internal/prompt/preset/*.go, internal/openrouter/tools.go, internal/prompts/prompts.ts
Adds RoomSystemPrompt/RoomUserPrompt/VotePrompt methods to prompt builders. Replaces warroom_search with transcript_search, removes discovery phase logic, introduces facilitate role terminology. Adds calculate and start_vote tools. Simplifies pipeline to single chat-room step.
TUI & Model Updates
internal/tui/model.go, internal/tui/view.go, internal/tui/layout.go, internal/tui/model_test.go
Removes NoDiscovery flag, wires CallModelTurnAction handling, adds PhaseChatRoom support, refactors view rendering from round-based to speaker/turn-based. Replaces renderRoundPipeline with simpler chat-room headers and voting indicators.
Simulation & Testing Updates
internal/sim/properties.go, internal/sim/view_test.go, internal/sim/sim_test.go, internal/sim/faults.go, internal/sim/scripted_api.go, internal/sim/testdata/snapshots/*.golden, debate/runner.go, debate/session.go, debate/state.go, debate/cost.go, internal/cassette/*.go
Replaces ValidPhases (discovery→chat_room), removes round-based checks, adds CallWithResponseFormat mocking/caching, updates snapshot golden files with new cost/duration values. Adds extensive state concurrency primitives (Mu, TranscriptEvent, VoteBallot, AnchorNotes).
Build & Configuration
Justfile, .gitignore, go.mod, cmd/warroom/main.go, CLAUDE.md, README.md, .github/workflows/electron-release.yml
Adds web/electron development and packaging recipes, Gorilla WebSocket dependency, GitHub Actions CI for Electron cross-platform release builds. Updates CLI with --web/--port flags and warroom stdio subcommand. Reframes product terminology and documentation.

Estimated code review effort

🎯 5 (Critical) | ⏱️ ~120 minutes


🐰 Hops excitedly
From rounds of debate to a cozy chat room we go,
With TypeScript dancers and Go's steady flow,
Desktop and web apps in Electron's glow,
Decision scenarios steal the old show! ✨


Note

🎁 Summarized by CodeRabbit Free

Your organization is on the Free plan. CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please upgrade your subscription to CodeRabbit Pro by visiting https://app.coderabbit.ai/login.

Comment @coderabbitai help to get the list of available commands and usage tips.

diyor28 and others added 17 commits February 14, 2026 20:15
Add native Electron desktop app that communicates with the Go backend
via JSON-RPC 2.0 over stdin/stdout, replacing the need for HTTP/WS.

Go side:
- New `internal/jsonrpc/` package: protocol types, stdio server, session
  management with proper goroutine lifecycle and race-free design
- New `warroom stdio` subcommand for Electron IPC
- Export SnapshotFromState, DispatchAction, TruncateExcerpt,
  SessionsBasePath from internal/web/ for shared use

Electron side:
- Forge + Vite build with React 19 renderer
- go-bridge.ts spawns Go binary, manages JSON-RPC over stdio
- Full React UI migrated from web/src/ with IPC-based useDebate hook
- Auto-updates via electron-updater, system tray, native menu
- Window state persistence, macOS/Windows platform support

Build pipeline:
- Justfile recipes for dev, cross-compile, package, and publish
- GitHub Actions workflow for cross-platform release builds
Replaced all 5 trap scenarios with domain-specific traps (network
effects misidentification, cloud TCO bias, pharma evergreening,
hub-and-spoke economics, AV safety statistics). Added LEARNINGS.md
documenting why traps are still too easy and research into harder
eval approaches (key insights checklists, adversarial filtering,
external knowledge gates).
Break up monolithic components and extract reusable primitives:
- Move shared/ to ui/, add Collapsible, ErrorBanner, LoadingState, StatusBadge
- Extract PageShell layout (dot-grid, noise, titlebar padding)
- Split AskUserOverlay into QuestionStep + formatAnswers
- Split DebateDashboard into IntelFeed + InterviewPlaceholder
- Extract useNotifications and useSessions hooks from useDebate/pages
- CompletionSummary and IntelFeed now use Collapsible
- Session pages use PageShell, StatusBadge, useSessions
- Replace Vite configs with Webpack for Electron main/renderer processes
- Switch to pnpm workspace for Electron dependency management
- Refactor preset prompts and system prompts for single-room chat model
- Update SessionDetailPage UI in both web and Electron renderers
- Simplify TUI layout and view (remove unused rendering code)
- Clean up docs and README
Closes multiple Electron UI gaps vs backend capabilities:

Backend (Go):
- Wire sessionId param through JSON-RPC initialize for session resume
- Pass ResumeSession to pipeline builder so LoadSessionStep runs

Frontend:
- Settings UI: preset selector, cheap mode toggle, API key warning,
  multiline topic input on landing screen
- Session resume: RESUME button on session list/detail pages, passes
  sessionId through to Go initialize
- Shutdown/cancel: CANCEL button in header calls shutdown RPC, navigates
  home
- Error recovery: ErrorBanner now has DISMISS and GO HOME actions
- Transcript export: EXPORT MD button copies markdown to clipboard (with
  download fallback), available during debate and on session detail
- Transcript rendering: markdown headings/bold/italic/code, visual
  differentiation for tool events (dimmer, mono) and vote events
  (accent border)
- Per-model cost: ModelCard shows individual model cost, ModelRoster
  passes modelCosts from snapshot
- Voting UX: progress bar in roster footer, per-agent confidence bar
  with ballot text, VOTED badge, sorted ballots in CompletionSummary
  with confidence bars and reasoning

https://claude.ai/code/session_01QPxpz4R7j4MFwGAqLEkwpb
- Fix data corruption risk: replace effectiveTopic=sessionID hack with
  a bool hasTopic parameter on eventLoop, so a failed resume never
  leaves a session ID in the topic field
- Fix path traversal: validate sessionId in initialize handler using
  the same check as getSession (reject /, \, .., .)
- Fix silent failure: config fetch error now sets configError state and
  defaults hasAPIKey to false (fail-safe), shows backend connection
  error banner
- Extract shared lib/export.ts with transcriptToMarkdown and
  copyOrDownload — removes duplicate code from DebateDashboard and
  SessionDetailPage
- Extract shared BallotCard component from CompletionSummary into
  ui/BallotCard.tsx, reused by SessionDetailPage
- Fix shutdown not resetting state: shutdown callback now resets to
  INITIAL_STATE after cleanup
- Fix setTimeout memory leak: export flash timers now use refs with
  cleanup on unmount
- Consolidate TranscriptEvent types: useSessions.ts now imports from
  types/protocol.ts instead of defining its own incomplete copy
- Fix inconsistent styling: text-red-400 → text-error in session pages

https://claude.ai/code/session_01QPxpz4R7j4MFwGAqLEkwpb
Core resilience:
- #1: Add React ErrorBoundary wrapping the app — render errors now show
  a styled fallback with reload button instead of a white screen
- #2: Wire onUpdateDownloaded from preload into useDebate state so the
  renderer can display an update-available banner
- #4: Remove dead submitTopic from useDebate hook and DebatePage
- #5: Add backend crash detection — main process forwards Go exit event
  via IPC, preload exposes onBackendExited, useDebate sets connected=false
  and shows error banner when backend dies
- #8: Fix route param changes being silently ignored — remove stale
  connectedRef guard; useEffect now disconnects previous session and
  reconnects on topic/resume change
- #11: Fix O(n²) activity scan per stream tick — walk backwards and
  break on first match instead of .map + .slice().some() per element
- #19: Add beforeunload handler during active debates to prevent
  accidental window close / data loss

Accessibility:
- #6: Add ARIA attributes across all components — role="alert" on
  ErrorBanner, role="status" + aria-label on GlowDot, role="complementary"
  on ModelRoster, role="progressbar" on progress bars, aria-label on
  textarea and dialog, aria-checked on QuestionStep options
- #12: Add focus management to AskUserOverlay — saves previous focus,
  auto-focuses first interactive element on mount, restores focus on
  unmount, implements keyboard focus trap (Tab/Shift+Tab)
- #14: Surface JSON parse errors in AskUserOverlay — shows visible error
  state with dismiss button instead of silently fabricating a question
- #15: Fix partial answer submission — check raw answers array for
  all-empty instead of checking formatted string against "(no answer)"

UX polish:
- #3: Delete dead LoadingState component (zero imports)
- #7: Lower noise-overlay z-index from 50 to 1 to avoid stacking
  conflict with titlebar drag region
- #9: Add "Open Folder" button to SessionListPage using the existing
  but previously unused openSessionsFolder preload API
- #10: Extract renderTranscriptEvent and roleBadge as pure functions
  outside DebateDashboard component to avoid recreation on every render
- #13: Replace hardcoded Tailwind colors in StatusBadge with theme
  tokens (text-success, text-error, text-warning)
- #16: Add window.confirm() before shutdown to prevent accidental
  cancellation of in-progress debates
- #18: Add refresh() to useSessionList hook, wire refresh button and
  visibilitychange auto-refresh to SessionListPage
- #21: Remove defaultOpen from transcript Collapsibles in
  SessionDetailPage — long transcripts now load collapsed

https://claude.ai/code/session_01QPxpz4R7j4MFwGAqLEkwpb
Display the 5 most recent sessions inline on the home screen so users
can resume in-progress sessions or reuse a previous topic without
navigating to the full sessions page.

- RESUME button on in-progress sessions navigates to debate with resume
- REUSE button on completed sessions copies the topic into the textarea
- Page switches from vertically-centered to top-aligned when sessions exist
- Clean first-run experience: section hidden when no sessions exist

https://claude.ai/code/session_01QPxpz4R7j4MFwGAqLEkwpb
Add async-loaded session list below topic input. Tab toggles focus
between text input and session list. In-progress sessions can be
resumed; completed sessions have their topic reused.
Both call sites in pipeline.go used the deprecated config.HasNativeSearch
wrapper. Replace with an inline adapter that calls the canonical
HasNativeSearchByID, satisfying staticcheck SA1019.
- Change AgentSet.HasNativeSearch field from func(string) bool to
  func(string) (bool, error) so config.HasNativeSearchByID can be
  assigned directly without a wrapper
- Update IsNativeSearch to handle errors explicitly (unknown model → false)
- Wire config.HasNativeSearchByID directly in both pipeline.go call sites
- Remove the deprecated config.HasNativeSearch shim; it no longer has callers
- Migrate TestHasNativeSearch → TestHasNativeSearchByID, testing error
  cases for unknown model IDs rather than silently returning false
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants