TODO.md

TODO

• Monitoring loop — background scheduler for online evals + outbound alerting on score/cost/latency/error thresholds (src/lib/alerts/kinds.ts, src/features/inbox/server.ts); also unblocks async runDataset
• Async runDataset — today it's one long-held POST (src/features/evaluation/server/datasets.ts): examples fire sequentially, then a ~6s sleep + 4 retry rounds resolve trace ids inline, so a big dataset risks proxy/request timeouts with only a spinner for feedback. Move to a job + status polling. Two couplings the rework must preserve: (1) the trace-linkage tail also snapshots toolCallsJson, which auto-judge ($datasetId.tsx onRunSuccess → judgeDatasetRun) reads — detaching it would make auto-judged runs grade with no tool calls; (2) can't just parallelize the example loop — concurrent inserts into sqlite dev.db risk lock errors
• Judge calibration — LLM-vs-human alignment score + tuning loop (src/lib/eval/evaluation.ts)
• Version snapshots — store judge prompt/model + example set per version so old versions can be viewed and re-run
• Compare two runs side-by-side — plans/compare-traces.md
• Historic data across agent versions (compare runs over time)
• Experiment tags (UI only) — "env" → "Tag" filter (tag.tags), chips on /traces + /sessions
• MCP global tools view — render the existing aggregateTools/UniqueTool[] as a once-per-tool list (which server(s) provide each), instead of only feeding duplicate/conflict badges in the by-server Tools tab
• Anthropic token counts estimated (o200k BPE); exact needs count_tokens API (src/features/inspect/logic/tokens.ts)
• RAG sidecars (embedding/retrieval as documents-with-scores) + multimodal render (image/audio message parts) — deferred until a producer emits those span shapes; classify/UI only, token-leak + base64-inflation already prevented
• Agent generative UI — render tool results as bespoke cards (trace-summary card, tool-error table, token bar / mini-waterfall) instead of prose, à la estate-agent listing cards; add outputSchema to src/features/agent/server/tools.ts to type them. Tools already return compact summaries; this is the render layer + per-tool opt-in