feat(eval): standardize dataset schema, support messages format, and add multi-turn benchmarks

.agents/skills/a2ui-add-eval-datapoint/SKILL.md

@@ -0,0 +1,106 @@
+---
+name: a2ui-add-eval-datapoint
+description: Step-by-step workflow for adding and verifying new evaluation data points in the A2UI evaluation suite.
+---
+
+# Adding and verifying A2UI evaluation data points
+
+Use this skill when adding new data points or datasets to the A2UI evaluation framework in `eval/`.
+
+## References
+
+For full architecture and setup details, consult:
+
+- `eval/README.md`: Evaluation quickstart, Transcrypt setup, and CLI flags.
+- `eval/DESIGN.md`: Architecture, multi-stage scorers, and encryption-at-rest design.
+- `eval/datasets/dataset_schema.json`: Formal JSON Schema defining required fields and structural rules for evaluation data points.
+
+---
+
+## Step-by-step workflow
+
+### 1. Unlock Transcrypt
+
+Datasets in `eval/datasets/` are encrypted at rest. Unlock Transcrypt locally before authoring:
+
+```bash
+cd eval
+bin/transcrypt -y -c aes-256-cbc -p <PASSWORD>
+```
+
+### 2. Author the data point
+
+Create or update a dataset file in `eval/datasets/<dataset_name>.yaml` (e.g. `eval/datasets/my_dataset.yaml`).
+
+Every data point **must conform** to the JSON Schema in `eval/datasets/dataset_schema.json`.
+
+```yaml
+- name: sample_unique_identifier
+  dataset: my_dataset
+  description: Clear summary of what this scenario tests.
+  catalog: 'specification/{version}/catalogs/basic/catalog.json'
+  system_prompt: |
+    Optional domain-specific system instructions (e.g. company policies, triage rules).
+  messages:
+    - role: user
+      content: 'Initial user request...'
+    - role: assistant
+      content: 'Assistant response...'
+      tool_calls:
+        - id: call_001
+          type: function
+          function: query_database
+          arguments:
+            key: value
+    - role: tool
+      tool_call_id: call_001
+      function: query_database
+      content: '{"result": "data"}'
+    - role: user
+      content: 'Render the UI card with the retrieved data...'
+  target: |
+    Expected UI outcome and scoring criteria for the LLM judge.
+```
+
+### 3. Validate schema compliance
+
+Verify that all data points satisfy `eval/datasets/dataset_schema.json`:
+
+```bash
+cd eval
+uv run python -m pytest tests/test_dataset.py
+```
+
+### 4. Run an evaluation on the new data point
+
+Always execute an evaluation run on the newly added dataset to verify model inference and scoring:
+
+```bash
+cd eval
+# Quick validation on gemini-3.1-flash-lite
+uv run main.py --dataset my_dataset --sanity
+
+# Full evaluation check
+uv run main.py --dataset my_dataset
+```
+
+View the interactive traces and judging rationales:
+
+```bash
+uv run inspect view start
+```
+
+### 5. Verify encryption and commit
+
+When staging changes, Git applies the Transcrypt clean filter. Confirm that the staged file is encrypted ciphertext before committing:
+
+```bash
+# Stage the new dataset file
+git add eval/datasets/my_dataset.yaml
+
+# Verify staged content is encrypted ciphertext (not plaintext)
+git diff --cached eval/datasets/my_dataset.yaml
+
+# Commit the encrypted data point
+git commit -m "feat(eval): add my_dataset evaluation data points"
+```

eval/DESIGN.md

@@ -14,12 +14,12 @@ The framework adopts the three-pillar architecture of Inspect AI—Tasks, Solver
 
 | Inspect Component | A2UI Implementation Detail                                                 | Primary Responsibility                                            |
 | :---------------- | :------------------------------------------------------------------------- | :---------------------------------------------------------------- |
-| Dataset           | Encrypted archives resolved into EvalSet JSON.                             | Providing structured prompts and ground-truth UI targets.         |
-| Solver            | Uses `A2uiSchemaManager` to inject system prompt, then generates response. | Orchestrating the model's attempt to generate a valid UI payload. |
-| Scorer            | A hybrid of SDK-based validation and LLM-as-a-judge.                       | Determining the accuracy and usability of the generated JSON.     |
+| Dataset           | Encrypted YAML files conforming to `datasets/dataset_schema.json`.         | Providing structured `messages` turns, catalogs, and UI targets.  |
+| Solver            | Uses `A2uiSchemaManager` to inject system prompt and protocol rules.       | Orchestrating the model's attempt to generate a valid UI payload. |
+| Scorer            | Algorithmic `a2ui_scorer` validation and `measured_model_graded_qa` judge. | Determining technical validity and semantic intent quality.       |
 | Model Provider    | Support for Gemini, OpenAI, Anthropic, and local vLLM.                     | Standardizing API interactions across diverse model families.     |
 
-The interaction flow begins with the resolution of a dataset sample. A solver then prepares the environment, using `A2uiSchemaManager` to provide the model with the correct protocol schemas and a "catalog" of available A2UI components—a critical constraint of the A2UI security model. The model generates a response, which is then parsed and corrected using the SDK's parser tools, and finally passed through a series of Scorers to assess its technical validity and semantic alignment with the user's intent.
+The interaction flow begins with loading a dataset sample. Each sample specifies a `catalog` path, optional domain `system_prompt`, and a list of `messages` representing the conversation history (including user requests, assistant responses, and tool calls). The solver merges the domain system prompt with A2UI protocol instructions and passes the conversational context to the model. The generated UI payload is parsed and evaluated by `a2ui_scorer` for schema validity and `measured_model_graded_qa` for semantic quality.
 
 ### **Integration with A2UI Python SDK**
 

eval/README.md

@@ -1,15 +1,33 @@
 # A2UI Evaluation Framework
 
-This folder contains evaluation tests (aka evals) for the A2UI project.
-An evaluation test verifies that a prompt produces expected results conforming to the A2UI schema and semantic rules.
+This folder contains evaluation tests (aka evals) for the A2UI project using the [Inspect AI](https://inspect.aisi.org.uk/) framework.
+An evaluation test verifies that a prompt or conversational history produces expected UI results conforming to the A2UI schema and semantic rules.
 
 ## Design
 
-For a detailed overview of the design, secrets management, and contamination prevention, see the [DESIGN.md](DESIGN.md) file.
+For a detailed overview of the evaluation architecture, multi-stage scoring, and secret management, see [DESIGN.md](DESIGN.md).
+
+## Datasets and Schema
+
+Evaluation data points live in `datasets/*.yaml` files (e.g., `datasets/core_v0_9_1.yaml`, `datasets/core_v1_0.yaml`, `datasets/multi_turn_conversation_dataset.yaml`).
+
+Every dataset file must conform to the JSON schema defined in `datasets/dataset_schema.json`.
+
+### Dataset Structure
+
+Each sample in a dataset YAML file defines:
+
+- **`name`** (required): Unique identifier for the sample.
+- **`description`** (required): Human-readable summary of the test scenario.
+- **`catalog`** (required): Relative path to the component catalog (e.g., `"specification/{version}/catalogs/basic/catalog.json"`).
+- **`dataset`** (optional): Logical dataset grouping name (defaults to file basename).
+- **`system_prompt`** (optional): Domain-specific system instructions (e.g. clinical triage rules or travel policies).
+- **`messages`** (required): Chat conversation turns (`user`, `assistant` with optional `tool_calls`, `tool` returns, and `system`).
+- **`target`** (optional): Expected UI outcome description and grading criteria for the LLM judge.
 
 ## Running Evaluations
 
-To run the evaluations, you need to use the Inspect AI CLI via `uv`. Make sure you are in this directory (`evals/eval`).
+Make sure you are in the `eval/` directory.
 
 ### Prerequisites
 
@@ -20,21 +38,19 @@ To run the evaluations, you need to use the Inspect AI CLI via `uv`. Make sure y
    ```
 
 2. **Decrypt Datasets (First Time Setup)**:
-   The evaluation datasets are encrypted at rest in the repository to prevent base model contamination. To decrypt them in your repo for evaluation, you need to initialize Transcrypt with the shared password. From the `evals/eval` directory, run:
+   The evaluation datasets are encrypted at rest in the repository to prevent base model contamination. To decrypt them locally, initialize Transcrypt with the shared password:
 
    ```bash
    bin/transcrypt -p <PASSWORD>
    ```
 
-You can request the password from any member of the A2UI team (it's not really a secret, but it's also not going on Github in plaintext).
-
-After this one time setup, you will have local plaintext access to the decrypted datasets in the `datasets/` directory, and they will be encrypted and decrypted transparently by git.
+   After this setup, git transparently encrypts files on `git add` and decrypts them on checkout.
 
-### Upgrading transcrypt
+### Upgrading Transcrypt
 
 If you pull updates that change the encryption settings (such as transitioning from MD5 to PBKDF2), you may encounter decryption errors during `git pull` or see OpenSSL deprecation warnings.
 
-To upgrade your local transcrypt configuration to the latest settings:
+To upgrade your local Transcrypt configuration to the latest settings:
 
 1. Run the upgrade command:
 
@@ -45,57 +61,62 @@ To upgrade your local transcrypt configuration to the latest settings:
    This updates the local filter scripts in your `.git` directory while preserving your saved password.
 
 2. Force Git to re-decrypt the files:
+
    ```bash
    git checkout HEAD -- $(git ls-crypt)
    ```
+
    This runs the files through the newly upgraded smudge filter, decrypting them.
 
-## Dimensions
+### Executing Evals
+
+To run all datasets:
 
-Evaluations run across the following dimensions:
+```bash
+uv run main.py
+```
 
-- Strategies: Different ways of orchestrating A2UI generation
-- Samples: Data in `dataset/`
+To run a specific dataset or multiple datasets:
+
+```bash
+# Run a single dataset
+uv run main.py --dataset multi_turn_conversation_dataset
 
-### Run Evals
+# Run multiple datasets
+uv run main.py --datasets core_v0_9_1,multi_turn_conversation_dataset
+```
 
-To run the evaluations:
+To test across different inference formats (`direct` JSON, `express` XML tags, `elemental` DSL):
 
 ```bash
-uv run main.py
+uv run main.py --dataset multi_turn_conversation_dataset --strategies direct,express,elemental
 ```
 
-For a quick 2-sample validation using `gemini-3.1-flash-lite`, use the sanity flag:
+For a quick 2-sample validation using `gemini-3.1-flash-lite`:
 
 ```bash
 uv run main.py --sanity
 ```
 
 ## Viewing Evaluation Results
 
-Inspect AI provides a web-based log viewer to explore the results of your evaluations.
-
-To start the log viewer:
+Inspect AI provides a web-based log viewer to explore interactive traces and judge rationales:
 
 ```bash
 uv run inspect view start
 ```
 
-This will start a local web server (usually at `http://localhost:7575`) and open the viewer in your browser. It will automatically load logs from the `logs/` directory.
+This starts a local web server (usually at `http://localhost:7575`).
 
-## Listing Available Models
-
-To list the available Gemini models supported by your API key:
+To print a console summary or markdown table from an eval log file:
 
 ```bash
-uv run inspect eval tasks.py -T list_models=True --model google/gemini-3-flash-preview
+uv run python bin/report_evals.py logs/<log_filename>.eval
 ```
 
-(the `--model` flag is required even though it is ignored)
-
-## Running Unit Tests
+## Running Unit Tests & Schema Validation
 
-To run the unit tests for the evaluation framework (dataset loader, solvers, scorers):
+To run the unit tests and validate all dataset files against `datasets/dataset_schema.json`:
 
 ```bash
 uv run python -m pytest