Assertion generation for data-driven questions

[ha2trinh]

This PR adds assertion generation capabilities to the AutoQ pipeline for data-local and data-global questions. Assertions are factual statements that can be used to evaluate the accuracy of RAG system answers.

Key Changes

New Features

• Assertion Generation: Added LocalClaimAssertionGenerator and GlobalClaimAssertionGenerator with a map-reduce approach for processing claims into ranked assertions
• Assertion Validation: Optional validation step that scores assertions on grounding, relevance, and verifiability (enabled by default with min score 3/5)

Configuration

• New assertions config section with:
  • max_assertions: Maximum assertions per question (default: 10)
  • enable_validation: Enable/disable validation (default: true)
  • min_validation_score: Minimum score threshold (default: 3)
  • batch_size: Batch size for processing (default: 50)
  • max_data_tokens: Max tokens per batch (default: 32000)

AutoE Updates

• Updated assertion-based scoring:
  • Updated scoring prompt
  • Added options to run scoring on top-k assertions
  • Added utilities to run assertion evaluation for multiple question sets and RAG methods
  • Added utilities to visualize assertion scoring results

CLI Updates

• Added assertion configuration options to settings.yaml

Documentation

• Updated CLI docs with assertion configuration options
• Updated autoq notebook with assertion settings
• Added assertion_gen notebook for generating assertions retrospectively

Output Format

Assertions are saved to assertions.json in each question output folder:

{
  "question_id": "...",
  "question_text": "...",
  "assertions": [
    {
      "statement": "The response should state that...",
      "source_count": 2,
      "score": 10,
      "rank": 1,
      "validation": {
        "is_valid": true,
        "scores": {
          "grounding": 5,
          "relevance": 5,
          "verifiability": 5,
          "reasoning": "..."
        }
      }
    }
  ]
}

Testing

• Tested CLI end-to-end with AP news dataset
• Verified assertion generation for both data-local and data-global questions
• Confirmed validation scoring works correctly

[Copilot]

Pull request overview

This PR adds assertion generation capabilities to the AutoQ pipeline for data-local and data-global questions. Assertions are factual statements that can be used to evaluate the accuracy of RAG system answers.

Key Changes

• New Assertion Generation Framework: Introduced LocalClaimAssertionGenerator and GlobalClaimAssertionGenerator with a map-reduce approach for processing claims into ranked assertions
• Optional Validation: Added AssertionValidator that scores assertions on grounding, relevance, and verifiability (enabled by default with min score 3/5)
• Configuration System: New AssertionConfig and AssertionPromptConfig classes with configurable parameters for max assertions, validation settings, batch size, and token limits
• CLI Integration: Updated CLI to support assertion generation with new configuration options in settings.yaml
• Visualization Support: Added matplotlib and seaborn dependencies with utility functions for visualizing assertion scoring results

Reviewed changes

Copilot reviewed 43 out of 45 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
uv.lock, pyproject.toml	Added matplotlib>=3.10.5 and seaborn>=0.13.2 dependencies
benchmark_qed/autoq/question_gen/data_questions/assertion_gen/	New module with base classes, validators, generators, and ranking utilities
benchmark_qed/autoq/question_gen/data_questions/local_question_gen.py	Integrated assertion generation for local questions
benchmark_qed/autoq/question_gen/data_questions/global_question_gen.py	Integrated assertion generation for global questions
benchmark_qed/autoq/config.py	Added AssertionConfig and AssertionPromptConfig classes
benchmark_qed/autoq/cli.py	Added assertion configuration support to CLI
benchmark_qed/autoq/io/question.py	Added _save_assertions function to save assertions separately
docs/	Updated documentation and notebooks with assertion configuration examples

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.