Skip to content

CLAUDE.md

Latest commit

 

History

History
90 lines (70 loc) · 3.27 KB

CLAUDE.md

File metadata and controls

90 lines (70 loc) · 3.27 KB
Raw

CLAUDE.md — Agent Reliability Arena

Project

Agent Reliability Arena (ara) — production readiness benchmarking for AI agents. PyPI package: agent-reliability-arena. CLI entry point: arena.

Key Commands

make install      # pip install -e ".[dev]"
make run          # arena run configs/example_agent.yaml
make dashboard    # streamlit run dashboard/app.py
make lint         # ruff check arena/ dashboard/
make test         # pytest tests/ -v
make clean        # remove reports/ and cache

Direct CLI:

arena run configs/example_agent.yaml --output-dir reports

Architecture

ArenaRunner (runner.py)
  └─ loads ArenaConfig (config.py + models.py)
  └─ builds BaseAgentAdapter (agent_adapter.py)
       ├─ LangChainReActAdapter
       ├─ OpenAIAssistantsAdapter
       └─ CustomAgentAdapter
  └─ runs each BaseTrack (tracks/__init__.py)
       ├─ ConsistencyTrack      → SemanticSimilarityScorer
       ├─ RobustnessTrack       → ParaphraseInjector + SemanticSimilarityScorer
       ├─ ToolFailureTrack      → ToolFailureInjector + ToolRecoveryScorer
       ├─ MemoryDriftTrack      → LLMJudgeScorer
       └─ EnterpriseRealismTrack → all injectors + EnterpriseReadinessScorer (ERI)
  └─ assembles ArenaReport (models.py)
  └─ saves reports (reporter.py)

Key Files

File Purpose
arena/models.py All Pydantic data models — edit here for schema changes
arena/tracks/__init__.py Track logic + TRACK_MAP registry
arena/scorers/__init__.py Scoring implementations
arena/injectors/__init__.py Chaos injection (failure, paraphrase, schema drift)
arena/agent_adapter.py Agent framework adapters
arena/runner.py Orchestration + Rich progress display
arena/reporter.py Markdown + JSON report generation
configs/task_suites/ YAML task suite definitions
dashboard/app.py Streamlit UI

Environment Variables

ANTHROPIC_API_KEY   # Required for Robustness, Memory Drift, Enterprise Realism tracks
OPENAI_API_KEY      # Required for langchain_react / openai_assistants agent types

Conventions

  • All scorers return float in [0.0, 1.0]
  • All tracks return TrackResult; all adapters return AgentRun
  • Async throughout; sync wrapper run() in runner.py is for CLI use only
  • Use asyncio.get_running_loop() (not get_event_loop()) for thread executors
  • Task suites live in configs/task_suites/<name>.yaml; load_task_suite() in config.py resolves both built-in names and file paths

Adding a New Track

  1. Implement BaseTrack in arena/tracks/__init__.py
  2. Add the new TrackName enum value to arena/models.py
  3. Register in TRACK_MAP at the bottom of tracks/__init__.py

Adding a New Agent Adapter

  1. Subclass BaseAgentAdapter in arena/agent_adapter.py
  2. Implement async def run(self, prompt: str, track: TrackName) -> AgentRun
  3. Add new AgentType enum value to arena/models.py
  4. Register in build_adapter() mapping

Dependencies

Core: langchain, langchain-openai, langchain-anthropic, openai, anthropic, pydantic, pyyaml, sentence-transformers, rich, httpx, tenacity Dashboard: streamlit, plotly, pandas Dev: pytest, pytest-asyncio, ruff