feat(eval): add agent-based iterative format optimization framework and history - #7
Closed
gspencergoog wants to merge 27 commits into
Closed
feat(eval): add agent-based iterative format optimization framework and history#7gspencergoog wants to merge 27 commits into
gspencergoog wants to merge 27 commits into
Conversation
gspencergoog
force-pushed
the
iterative_optimization
branch
from
July 23, 2026 02:54
a4ee7c1 to
85872d2
Compare
…mp loading and sys.path resilience
…on, path resolution, and recursion safety
…deduplication, and safe dict access
…ck, archiver log fallback, and sync race conditions
…eline dict normalization, and sys.path
…chiver metrics fallback, and transport format support
…ions, scores/usage guards, and mtime sorting
…rds, None coalescing, and glob de-duplication
…tering, S_opt zero fallbacks, and PID log dir
…solution, results.json archiving, and run_meta metric extraction
… samples dict normalization, and mtime eval file sorting
…ialized guard, module level sync import, and failure explanation list scores
…tion, non-none metric resolution, utf-8 subprocess, and baseexception cleanup
…guards, reporter metric accuracy fallback, and worktree history path filtering
…y dict guard, judging_val N/A check, and archiver results.json handling
…a2ui-agent-sdk in workspace
…r iterative optimization scripts
…-format-optimizer skill - Define inference-format-optimizer skill under eval/skills/inference-format-optimizer/ with SKILL.md, references/, templates/, scripts/, and unit tests (59/59 passing). - Update archiver.py, sync_history.py, and optimize_format.py to support workspace-relative history storage (eval/history/) and configurable --history-dir option. - Consolidate all historical run logs into format subdirectories (eval/history/atom/ and eval/history/express/) and update master index generators. - Remove legacy eval/iterative/ directory.
…aluation history in prettier
…ocess timeout fixes
gspencergoog
force-pushed
the
iterative_optimization
branch
from
July 24, 2026 17:41
16167d3 to
ab69d28
Compare
Owner
Author
|
Closing in favor of upstream PR a2ui-project#2092 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the modular
eval/iterativeagent-based format optimization framework and evaluation diagnostic utilities. This framework automates prompt engineering and compiler optimizations for A2UI inference formats (such as Atom) via autonomous subagent evaluation loops, performance benchmarking, history archiving, and comparative reporting.Changes
eval/iterative/optimize_format.pyinto dedicated submodules (utils/archiver.py,utils/format_tools.py,utils/reporter.py,utils/runner.py).eval/iterative/compare_results.py: Provides Markdown delta comparisons across evaluation runs and baseline benchmarks (computing median schema accuracy, quality score, latency, and reasoning token efficiency).eval/iterative/sync_history.py: Automatically discovers and syncs iteration runs across parallel worktrees into main history with run ID collision re-indexing.eval/tests/(test_compare_results.py,test_sync_history.py,test_utils.py,test_optimize_format.py), achieving 93% line coverage acrosseval/iterative/.eval/iterative/history/) detailing prompt and compiler optimization experiments for Atom format.Impact & Risks
eval/and developer tooling.atom_formatbranch.Testing
eval/iterative/:./scripts/fix_format.sh(Pyink).