feat(eval): add agent-based iterative format optimization framework and history - #2092
Open
gspencergoog wants to merge 22 commits into
Open
feat(eval): add agent-based iterative format optimization framework and history#2092gspencergoog wants to merge 22 commits into
gspencergoog wants to merge 22 commits into
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
gspencergoog
force-pushed
the
iterative_optimization
branch
from
July 24, 2026 18:01
3ef9c90 to
d949c9a
Compare
…lines with metadata
…-format-optimizer skill
…rmat-optimizer skill
…pdate sync_history exception handling
…eeping history index summaries and sync_history tool
… enrich running conditions
…history_summary.md
…and optimize_format
- Added 37 new unit test cases across test_compare_results.py, test_optimize_format.py, test_sync_history.py, and test_utils.py. - Total test count expanded to 93 tests (100% passing). - Increased statement line coverage across all 7 optimizer script modules from 69.7% to 90.2%.
- Apply Prettier formatting across JSON and Markdown files under eval/iterative_format_optimizer to pass enforce-formatting check. - Fix relative markdown links to history and history_summary.md in inference_format_iteration.md to resolve lychee link-check failures. - Update path references and fix duplicate text in agent_instructions.md.
gspencergoog
enabled auto-merge (squash)
July 30, 2026 02:49
…roject.toml - Add missing type annotations and fix duplicate method names in optimizer tests - Add optimizer tests to mypy exclude list in pyproject.toml
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
…use Manager context manager - Use asyncio.to_thread for parse_with_hard_kill_timeout to prevent blocking the asyncio event loop during evaluations - Use multiprocessing.Manager as a context manager to avoid leaking manager processes and IPC file descriptors
gspencergoog
force-pushed
the
iterative_optimization
branch
from
July 30, 2026 03:21
d887888 to
1d409a4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Introduces the
inference-format-optimizeragent skill and evaluation suite undereval/skills/inference-format-optimizer/. This skill provides a modular, reproducible framework for autonomous subagents and human contributors to benchmark, optimize, and iterate on A2UI LLM inference formats (such as Atom, Express, and Elemental).Changes
eval/skills/inference-format-optimizer/):SKILL.md: Main entrypoint cheat-sheet and workflow instructions.scripts/optimize_format.py: CLI orchestrator for running validation subsets or full evaluation suites across formats.scripts/compare_results.py: Computes 1:1 metric deltas, accuracy, latency, token efficiency, and compositescripts/sync_history.py: Merges iteration run archives across parallel Git worktrees into a unified history with collision-free run ID re-indexing.scripts/utils/: Modularized helper utilities (archiver.py,format_tools.py,reporter.py,runner.py).run_meta.json):model(e.g.google/gemini-3.5-flash) andthinking_budgetattributes.ModelandBudgetcolumns across master and format-specifichistory_summary.mdindexes.references/):tests/):compare_results.py,optimize_format.py,sync_history.py, andutilshelper modules (59 tests passed).eval/history/):patch.diff,report.md,run_meta.json) and master index (history_summary.md) detailing past optimization experiments for the Atom format.transport,express,elemental, andatomformats.Impact & Risks
eval/and developer tooling without changing core runtime packages.Testing
uv run --all-packages pytest eval/skills/inference-format-optimizer/tests/(59 passed)../scripts/fix_format.sh --check(passed).