feat(eval): standardize dataset schema, support messages format, and add multi-turn benchmarks#2078
feat(eval): standardize dataset schema, support messages format, and add multi-turn benchmarks#2078jacobsimionato wants to merge 14 commits into
Conversation
… 24-turn triage and 32-turn rebooking scenarios
…schema rules and encrypted commit workflow
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request expands the A2UI evaluation framework to support modular datasets, standard LLM chat completion message formats, custom catalogs, and merged domain system prompts. It migrates legacy monolithic dataset files into named YAML files under eval/datasets/ and updates the loader, solvers, scorers, and reporting tools accordingly. The reviewer feedback highlights several areas for improvement: uncommenting and correcting the .gitattributes pattern to ensure dataset files are encrypted by transcrypt, catching specific exceptions instead of using broad except Exception blocks, refactoring the migration script to use pathlib for consistency, and restoring several function docstrings that were removed.
…overy without hardcoded paths
…ing, and docstrings
…ttributes tracking
| "properties": { | ||
| "role": { | ||
| "type": "string", | ||
| "enum": ["user", "assistant", "system", "tool"], |
There was a problem hiding this comment.
I think the tool role should be called tool_response, since it doesn't seem intended to indicate tool calls, but rather just responses.
I find it a little asymmetric that there's an array of tool calls per message, but that the responses come back one at a time, but I suppose that makes sense given that they're async.
It's a little weird that you can have a tool role message with embedded tool calls. I mean, I guess? A tool could just immediately fork other tools without asking the LLM?
Overall, I think the tool call/response design could be cleaner. Maybe the tool_calls array should not exist, and you can just unroll it into messages with tool_call and tool_response roles? Probably the problem is just that messages need to be polymorphic so they match their roles.
There was a problem hiding this comment.
Thanks for the detailed review and thoughtful feedback, Greg!
To share the overall context: these choices were made primarily to maintain 1:1 consistency with Inspect AI’s internal framework, even if they aren't the exact standalone choices we would make in isolation.
Because Inspect AI serves as our evaluation harness, aligning our dataset schemas and YAML message formats directly with Inspect AI's internal data model (inspect_ai.model.ChatMessageUser, ChatMessageAssistant, ChatMessageTool, ChatMessageSystem) allows us to load datasets into Sample.input seamlessly without custom transformation scripts or translation layers. Inspect AI's provider adapters (such as google-genai for Gemini) then convert these standard structures directly into provider-native API calls (like Gemini's FunctionCall and FunctionResponse parts).
Addressing your points specifically:
role: "tool"Naming: We keptrole: "tool"for direct consistency withinspect_ai.model.ChatMessageTool(role="tool")and standard OpenAI/Anthropic/Gemini completion specs.- Asymmetry & Parallel Tool Calls: The array of tool calls on assistant turns vs individual tool response turns mirrors standard parallel tool calling across LLM APIs—where an assistant turn requests multiple tools at once (
tool_calls: [call_1, call_2]), and each tool returns its output in a separaterole: "tool"turn with atool_call_id. - Unrolling & Provider Adapters: Unrolling
tool_callsinto custom roles liketool_callwould break Inspect AI’s model adapters, which automatically mapChatMessageAssistant(tool_calls=[...])andChatMessageTool(...)into native API structures (like Gemini'sFunctionCallandFunctionResponseparts). - Schema Polymorphism: Based on your feedback, we have updated
dataset_schema.jsonso that message items are defined using a role-polymorphiconeOfschema (user,assistant,tool,system). Under the updated schema:user&systemmessages permit onlyroleandcontent.assistantmessages permitcontentand optionaltool_calls.toolmessages permitcontent,tool_call_id, and optionalfunction, but strictly forbidtool_calls.
There was a problem hiding this comment.
Ahh, OK. I can see wanting to fit into Inspect AI.
|
I think the PR description got a little messed up. |
2 participants
Description of Changes
This PR upgrades the A2UI evaluation framework to align with modern LLM standards and Inspect AI chat paradigms:
Dataset Schema Standardization (dataset_schema.json):
Modular Dataset Architecture:
Loader, Solvers & CLI Enhancements:
╭──────────────────────────────────────────────────────────────────────────────╮
│a2ui_v0_9_1_eval (55 samples): google/gemini-3.5-flash │
╰──────────────────────────────────────────────────────────────────────────────╯
sample_shuffle: 20260723, max_tasks: 10, grading_model: google/gemini-3.5-flash,
strategy: direct, dataset: (samples)
total time: 0:02:03
google/gemini-3.5-flash 999,674 tokens [I: 267,009, CW: 0, CR: 427,984, O:
88,060, R: 216,621]
a2ui_scorer measured_model_graded_qa
accuracy 0.964 accuracy 0.900
Log:
logs/20260723/2026-07-23T03-14-15-00-00_a2ui-v0-9-1-eval_aJLQc3GrCf9SNdqKrFUZe2.
eval
╭──────────────────────────────────────────────────────────────────────────────╮
│a2ui_v0_9_1_eval (55 samples): google/gemini-3.5-flash │
╰──────────────────────────────────────────────────────────────────────────────╯
sample_shuffle: 20260723, max_tasks: 10, grading_model: google/gemini-3.5-flash,
strategy: subagent_tool, dataset: (samples)
total time: 0:02:08
google/gemini-3.5-flash 1,091,535 tokens [I: 333,374, CW: 0, CR: 436,412, O:
108,794, R: 212,955]
a2ui_scorer measured_model_graded_qa
accuracy 0.964 accuracy 0.918
Log:
logs/20260723/2026-07-23T03-14-15-00-00_a2ui-v0-9-1-eval_GsSYibLu8eUnwxg4yFD2t7.
eval
Completed all tasks in 'logs/20260723' successfully
Evaluations complete. Logs saved to: /Users/jsimionato/development/a2ui_repos/datapoint/a2ui/eval/logs
Running evals with seed: 20260723 and max samples: 100
Executing: uv run python main.py --model google/gemini-3.5-flash --sample-shuffle 20260723 --log-dir logs/20260723 --max-retries 10 --grading-model google/gemini-3.5-flash --limit 100
=======================================================
Determining pass percentage from log file: 2026-07-23T03-14-15-00-00_a2ui-v0-9-1-eval_aJLQc3GrCf9SNdqKrFUZe2.eval
=== Evaluation Results Summary ===
--- Dataset: core_v0_9_1 ---
animalKingdomExplorer | Algorithmic: PASS | Judging: C | Inference Time: 66.22s
calendarEventCreator | Algorithmic: PASS | Judging: C | Inference Time: 33.78s
chatRoom | Algorithmic: PASS | Judging: C | Inference Time: 28.08s
checkoutPage | Algorithmic: PASS | Judging: C | Inference Time: 18.51s
cinemaSeatSelection | Algorithmic: PASS | Judging: C | Inference Time: 31.07s
clientSideValidation | Algorithmic: PASS | Judging: C | Inference Time: 22.42s
contactCard | Algorithmic: PASS | Judging: C | Inference Time: 16.37s
contextAwareUI | Algorithmic: PASS | Judging: C | Inference Time: 30.69s
courseSyllabus | Algorithmic: PASS | Judging: C | Inference Time: 22.90s
customYamlKeysTest | Algorithmic: PASS | Judging: C | Inference Time: 39.71s
dashboard | Algorithmic: PASS | Judging: C | Inference Time: 23.27s
deleteSurface | Algorithmic: PASS | Judging: C | Inference Time: 14.74s
dogBreedGenerator | Algorithmic: PASS | Judging: C | Inference Time: 32.83s
eCommerceProductPage | Algorithmic: PASS | Judging: C | Inference Time: 36.41s
fileBrowser | Algorithmic: PASS | Judging: P | Inference Time: 20.02s
[Judging Failure Reason (Grade P)]:
To determine if the submission meets the criterion, let's analyze it step-by-step:
fitnessTracker | Algorithmic: PASS | Judging: C | Inference Time: 28.11s
flashcardApp | Algorithmic: PASS | Judging: C | Inference Time: 18.47s
flightBooker | Algorithmic: PASS | Judging: C | Inference Time: 19.48s
hotelSearchResults | Algorithmic: PASS | Judging: P | Inference Time: 21.38s
[Judging Failure Reason (Grade P)]:
To determine whether the submission meets the criterion, we will break down the evaluation step by step:
interactiveDashboard | Algorithmic: PASS | Judging: C | Inference Time: 27.87s
jobApplication | Algorithmic: PASS | Judging: C | Inference Time: 21.47s
kanbanBoard | Algorithmic: PASS | Judging: C | Inference Time: 20.39s
loginForm | Algorithmic: PASS | Judging: C | Inference Time: 18.60s
mcpAppProxy | Algorithmic: PASS | Judging: C | Inference Time: 2.14s
musicPlayer | Algorithmic: PASS | Judging: C | Inference Time: 15.38s
nestedDataBinding | Algorithmic: PASS | Judging: C | Inference Time: 29.18s
nestedLayoutRecursive | Algorithmic: PASS | Judging: C | Inference Time: 17.65s
newsAggregator | Algorithmic: PASS | Judging: C | Inference Time: 24.66s
notificationCenter | Algorithmic: PASS | Judging: P | Inference Time: 19.88s
[Judging Failure Reason (Grade P)]:
To determine if the submission meets the criterion, let's break down the requirements step-by-step:
openUrlAction | Algorithmic: PASS | Judging: C | Inference Time: 18.76s
photoEditor | Algorithmic: PASS | Judging: C | Inference Time: 13.03s
podcastEpisode | Algorithmic: PASS | Judging: P | Inference Time: 29.59s
[Judging Failure Reason (Grade P)]:
To determine the grade for the submission, we evaluate it against each element of the criterion:
productGallery | Algorithmic: PASS | Judging: C | Inference Time: 31.13s
productGalleryData | Algorithmic: PASS | Judging: C | Inference Time: 30.41s
profileEditor | Algorithmic: PASS | Judging: P | Inference Time: 22.94s
[Judging Failure Reason (Grade P)]:
To evaluate the submission against the given criterion, let's break down the requirements step-by-step:
recipeCard | Algorithmic: PASS | Judging: C | Inference Time: 27.85s
restaurantFinder | Algorithmic: PASS | Judging: I | Inference Time: 72.36s
[Judging Failure Reason (Grade I)]:
To assess the submission based on the provided criterion, let's check each requirement:
restaurantMenu | Algorithmic: PASS | Judging: C | Inference Time: 27.45s
rizzCharts | Algorithmic: FAIL | Judging: I | Inference Time: 24.13s
[Algorithmic Failure Reason]:
A2UI tags '' and '' not found in response.
[Judging Failure Reason (Grade I)]:
Based on the evaluation of the task and the criterion, here is the step-by-step reasoning:
settingsPage | Algorithmic: PASS | Judging: C | Inference Time: 37.87s
simpleCalculator | Algorithmic: PASS | Judging: C | Inference Time: 28.61s
smartHome | Algorithmic: PASS | Judging: P | Inference Time: 22.20s
[Judging Failure Reason (Grade P)]:
An assessment of the submission against the provided criterion:
socialMediaPost | Algorithmic: PASS | Judging: C | Inference Time: 18.41s
standardFunctions | Algorithmic: PASS | Judging: C | Inference Time: 20.94s
stockWatchlist | Algorithmic: PASS | Judging: P | Inference Time: 17.63s
[Judging Failure Reason (Grade P)]:
To determine if the submission meets the criterion, let us analyze it step by step:
surveyForm | Algorithmic: PASS | Judging: C | Inference Time: 17.97s
travelItinerary | Algorithmic: PASS | Judging: C | Inference Time: 38.75s
triviaQuiz | Algorithmic: PASS | Judging: C | Inference Time: 15.79s
updateDataModel | Algorithmic: PASS | Judging: C | Inference Time: 24.96s
videoCallInterface | Algorithmic: PASS | Judging: C | Inference Time: 22.23s
weatherForecast | Algorithmic: PASS | Judging: C | Inference Time: 32.30s
--- Dataset: multi_turn_conversation_dataset ---
banking_dispute_resolution | Algorithmic: PASS | Judging: C | Inference Time: 17.99s
corporate_travel_multi_city_rebooking | Algorithmic: PASS | Judging: C | Inference Time: 30.12s
flight_booking_flow | Algorithmic: FAIL | Judging: C | Inference Time: 18.91s
[Algorithmic Failure Reason]:
Validation failed: {'version': '0.9.1', 'createSurface': {'surfaceId': 'main', 'catalogId': 'https://a2ui.org/specification/v0_9/catalogs/basic/catalog.json', 'theme': {'primaryColor': '#0055A5', 'agentDisplayName': 'Acme Airlines Concierge'}}} is not valid under any of the given schemas
Context failures:
- '0.9.1' is not one of ['v0.9', 'v0.9.1']
- '0.9.1' is not one of ['v0.9', 'v0.9.1']
- 'updateComponents' is a required property
- Additional properties are not allowed ('createSurface' was unexpected)
- '0.9.1' is not one of ['v0.9', 'v0.9.1']
- 'updateDataModel' is a required property
- Additional properties are not allowed ('createSurface' was unexpected)
- '0.9.1' is not one of ['v0.9', 'v0.9.1']
- 'deleteSurface' is a required property
- Additional properties are not allowed ('createSurface' was unexpected)
healthcare_patient_intake_triage | Algorithmic: PASS | Judging: C | Inference Time: 22.63s
==================================
Dataset Summary:
core_v0_9_1 : 42/51 passed (82.35%)
multi_turn_conversation_dataset: 3/4 passed (75.00%)
Inference Time - Average: 25.58s | Median: 22.90s
Pass percentage: 96.36%
Pass percentage check passed.
=======================================================
Determining pass percentage from log file: 2026-07-23T03-14-15-00-00_a2ui-v0-9-1-eval_GsSYibLu8eUnwxg4yFD2t7.eval
=== Evaluation Results Summary ===
--- Dataset: core_v0_9_1 ---
animalKingdomExplorer | Algorithmic: PASS | Judging: C | Inference Time: 9.83s
calendarEventCreator | Algorithmic: PASS | Judging: C | Inference Time: 4.46s
chatRoom | Algorithmic: PASS | Judging: C | Inference Time: 5.21s
checkoutPage | Algorithmic: PASS | Judging: C | Inference Time: 3.53s
cinemaSeatSelection | Algorithmic: PASS | Judging: C | Inference Time: 3.31s
clientSideValidation | Algorithmic: PASS | Judging: C | Inference Time: 3.32s
contactCard | Algorithmic: PASS | Judging: C | Inference Time: 6.00s
contextAwareUI | Algorithmic: PASS | Judging: C | Inference Time: 5.04s
courseSyllabus | Algorithmic: PASS | Judging: C | Inference Time: 3.69s
customYamlKeysTest | Algorithmic: PASS | Judging: I | Inference Time: 3.77s
[Judging Failure Reason (Grade I)]:
Based on the evaluation of the submitted answer against the specified criterion, here is the step-by-step reasoning:
dashboard | Algorithmic: PASS | Judging: C | Inference Time: 5.03s
deleteSurface | Algorithmic: PASS | Judging: C | Inference Time: 3.84s
dogBreedGenerator | Algorithmic: PASS | Judging: C | Inference Time: 5.57s
eCommerceProductPage | Algorithmic: PASS | Judging: C | Inference Time: 4.30s
fileBrowser | Algorithmic: PASS | Judging: P | Inference Time: 4.17s
[Judging Failure Reason (Grade P)]:
To assess the submission against the given criterion, let's break down the requirements step by step:
fitnessTracker | Algorithmic: PASS | Judging: C | Inference Time: 4.74s
flashcardApp | Algorithmic: PASS | Judging: C | Inference Time: 4.44s
flightBooker | Algorithmic: PASS | Judging: C | Inference Time: 3.66s
hotelSearchResults | Algorithmic: PASS | Judging: P | Inference Time: 5.22s
[Judging Failure Reason (Grade P)]:
To determine if the submission meets the criterion, let us analyze it step by step:
interactiveDashboard | Algorithmic: PASS | Judging: C | Inference Time: 4.19s
jobApplication | Algorithmic: PASS | Judging: C | Inference Time: 4.87s
kanbanBoard | Algorithmic: PASS | Judging: C | Inference Time: 4.58s
loginForm | Algorithmic: PASS | Judging: C | Inference Time: 5.91s
mcpAppProxy | Algorithmic: FAIL | Judging: I | Inference Time: 2.88s
[Algorithmic Failure Reason]:
A2UI tags '' and '' not found in response.
[Judging Failure Reason (Grade I)]:
To assess the submission, let's look at the requirement and what was provided:
musicPlayer | Algorithmic: PASS | Judging: C | Inference Time: 4.55s
nestedDataBinding | Algorithmic: PASS | Judging: C | Inference Time: 7.79s
nestedLayoutRecursive | Algorithmic: PASS | Judging: C | Inference Time: 4.56s
newsAggregator | Algorithmic: PASS | Judging: C | Inference Time: 5.32s
notificationCenter | Algorithmic: PASS | Judging: C | Inference Time: 4.82s
openUrlAction | Algorithmic: PASS | Judging: C | Inference Time: 2.92s
photoEditor | Algorithmic: PASS | Judging: C | Inference Time: 3.01s
podcastEpisode | Algorithmic: PASS | Judging: P | Inference Time: 6.39s
[Judging Failure Reason (Grade P)]:
To determine the appropriate grade for the submission, we will assess how well it meets each requirement of the criterion:
productGallery | Algorithmic: PASS | Judging: C | Inference Time: 6.08s
productGalleryData | Algorithmic: PASS | Judging: C | Inference Time: 4.48s
profileEditor | Algorithmic: PASS | Judging: C | Inference Time: 3.82s
recipeCard | Algorithmic: PASS | Judging: C | Inference Time: 4.38s
restaurantFinder | Algorithmic: PASS | Judging: I | Inference Time: 4.78s
[Judging Failure Reason (Grade I)]:
To assess the submission against the given criterion, we will break down the requirements step by step:
restaurantMenu | Algorithmic: PASS | Judging: C | Inference Time: 4.83s
rizzCharts | Algorithmic: PASS | Judging: C | Inference Time: 7.03s
settingsPage | Algorithmic: PASS | Judging: C | Inference Time: 3.72s
simpleCalculator | Algorithmic: FAIL | Judging: C | Inference Time: 4.23s
[Algorithmic Failure Reason]:
Validation failed: {'version': '0.9', 'createSurface': {'surfaceId': 'main', 'catalogId': 'https://a2ui.org/specification/v0_9/catalogs/basic/catalog.json', 'theme': {'primaryColor': '#10B981', 'agentDisplayName': 'Calculator'}}} is not valid under any of the given schemas
Context failures:
- '0.9' is not one of ['v0.9', 'v0.9.1']
- '0.9' is not one of ['v0.9', 'v0.9.1']
- 'updateComponents' is a required property
- Additional properties are not allowed ('createSurface' was unexpected)
- '0.9' is not one of ['v0.9', 'v0.9.1']
- 'updateDataModel' is a required property
- Additional properties are not allowed ('createSurface' was unexpected)
- '0.9' is not one of ['v0.9', 'v0.9.1']
- 'deleteSurface' is a required property
- Additional properties are not allowed ('createSurface' was unexpected)
smartHome | Algorithmic: PASS | Judging: C | Inference Time: 4.74s
socialMediaPost | Algorithmic: PASS | Judging: C | Inference Time: 6.72s
standardFunctions | Algorithmic: PASS | Judging: C | Inference Time: 4.58s
stockWatchlist | Algorithmic: PASS | Judging: C | Inference Time: 3.41s
surveyForm | Algorithmic: PASS | Judging: C | Inference Time: 3.92s
travelItinerary | Algorithmic: PASS | Judging: C | Inference Time: 6.25s
triviaQuiz | Algorithmic: PASS | Judging: C | Inference Time: 4.25s
updateDataModel | Algorithmic: PASS | Judging: C | Inference Time: 3.17s
videoCallInterface | Algorithmic: PASS | Judging: C | Inference Time: 5.06s
weatherForecast | Algorithmic: PASS | Judging: C | Inference Time: 5.32s
--- Dataset: multi_turn_conversation_dataset ---
banking_dispute_resolution | Algorithmic: PASS | Judging: C | Inference Time: 3.92s
corporate_travel_multi_city_rebooking | Algorithmic: PASS | Judging: C | Inference Time: 4.74s
flight_booking_flow | Algorithmic: PASS | Judging: C | Inference Time: 4.38s
healthcare_patient_intake_triage | Algorithmic: PASS | Judging: C | Inference Time: 3.77s
==================================
Dataset Summary:
core_v0_9_1 : 44/51 passed (86.27%)
multi_turn_conversation_dataset: 4/4 passed (100.00%)
Inference Time - Average: 4.70s | Median: 4.55s
Pass percentage: 96.36%
Pass percentage check passed.
Wrote evaluation summary to: /Users/jsimionato/development/a2ui_repos/datapoint/a2ui/eval/eval_summary.md
CI Passed: All evaluation strategies met the threshold.: Added and CLI flags.
Benchmark Evaluation Results
Evaluations across all datasets (, , and ) on :
Multi-Turn Benchmark Performance ()
All 4 complex multi-turn scenarios achieved 100.0% Algorithmic Pass Rate and 100.0% Grade C Judging:
Testing Instructions
============================= test session starts ==============================
platform darwin -- Python 3.14.2, pytest-9.1.1, pluggy-1.6.0
rootdir: /Users/jsimionato/development/a2ui_repos/datapoint/a2ui
configfile: pyproject.toml
plugins: anyio-4.14.1, asyncio-1.4.0
asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
collected 30 items
tests/test_dataset.py .... [ 13%]
tests/test_run_ci_evals.py ............ [ 53%]
tests/test_scorers.py ....... [ 76%]
tests/test_strategies.py ....... [100%]
=============================== warnings summary ===============================
eval/tests/test_scorers.py::test_scorer_valid_json_v091
eval/tests/test_scorers.py::test_scorer_valid_json
eval/tests/test_scorers.py::test_scorer_invalid_json
eval/tests/test_scorers.py::test_scorer_missing_root
eval/tests/test_scorers.py::test_scorer_duplicate_ids
eval/tests/test_scorers.py::test_scorer_broken_relationship
eval/tests/test_scorers.py::test_scorer_circular_reference
/Users/jsimionato/development/a2ui_repos/datapoint/a2ui/eval/a2ui_eval/scorers.py:78: DeprecationWarning: parse_response is deprecated. Please use format.parser.parse_response(...) on your InferenceFormat instance instead.
parts = parse_response(answer_text)
eval/tests/test_strategies.py::test_a2ui_express_solvers
/Users/jsimionato/development/a2ui_repos/datapoint/a2ui/.venv/lib/python3.14/site-packages/google/genai/types.py:42: DeprecationWarning: '_UnionGenericAlias' is deprecated and slated for removal in Python 3.17
VersionedUnionType = Union[builtin_types.UnionType, _UnionGenericAlias]
eval/tests/test_strategies.py::test_a2ui_express_solvers
eval/tests/test_strategies.py::test_a2ui_express_solvers
eval/tests/test_strategies.py::test_a2ui_express_solvers
eval/tests/test_strategies.py::test_a2ui_express_solvers
:106: DeprecationWarning: BaseAgentConfig is deprecated and will be removed in future versions.
eval/tests/test_strategies.py::test_a2ui_express_solvers
/Users/jsimionato/development/a2ui_repos/datapoint/a2ui/eval/a2ui_eval/strategies/format.py:59: UserWarning: [EXPERIMENTAL] ExpressFormat: This feature is experimental and may change or be removed in future versions without notice. It may introduce breaking changes at any time.
return ExpressFormat(catalog=catalog, surface_id=surface_id)
eval/tests/test_strategies.py::test_a2ui_express_solvers
/Users/jsimionato/development/a2ui_repos/datapoint/a2ui/agent_sdks/python/a2ui_agent/src/a2ui/inference_formats/experimental/express/prompt_generator.py:480: UserWarning: [EXPERIMENTAL] ExpressParser: This feature is experimental and may change or be removed in future versions without notice. It may introduce breaking changes at any time.
self.parser = ExpressParser(catalog) if catalog else None
eval/tests/test_strategies.py::test_a2ui_express_solvers
/Users/jsimionato/development/a2ui_repos/datapoint/a2ui/agent_sdks/python/a2ui_agent/src/a2ui/inference_formats/experimental/express/format.py:73: UserWarning: [EXPERIMENTAL] ExpressParser: This feature is experimental and may change or be removed in future versions without notice. It may introduce breaking changes at any time.
return ExpressParser(self.catalog, self.surface_id)
eval/tests/test_strategies.py::test_a2ui_elemental_solvers
/Users/jsimionato/development/a2ui_repos/datapoint/a2ui/eval/a2ui_eval/strategies/format.py:63: UserWarning: [EXPERIMENTAL] ElementalFormat: This feature is experimental and may change or be removed in future versions without notice. It may introduce breaking changes at any time.
return ElementalFormat(catalog=catalog, surface_id=surface_id)
eval/tests/test_strategies.py::test_a2ui_elemental_solvers
/Users/jsimionato/development/a2ui_repos/datapoint/a2ui/agent_sdks/python/a2ui_agent/src/a2ui/inference_formats/experimental/elemental/format.py:62: UserWarning: [EXPERIMENTAL] ElementalParser: This feature is experimental and may change or be removed in future versions without notice. It may introduce breaking changes at any time.
return ElementalParser(self.catalog, self.surface_id)
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
======================= 30 passed, 17 warnings in 3.05s ========================
2. Run Full Benchmark Suite:
Starting evaluation for multiple strategies...
Completed all tasks in 'logs' successfully
Evaluations complete. Logs saved to: /Users/jsimionato/development/a2ui_repos/datapoint/a2ui/eval/logs