Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions docs/docs/pages/advanced/custom-clients.mdx
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
---
title: Advanced Custom Client Configuration
description: Learn advanced techniques for configuring custom LLM clients with Scenario. Explore Azure API Gateway integration, custom OpenAI clients with headers and query parameters, and global ModelConfig settings for enterprise deployments.
---

import AzureExample from "../_generated/examples/test_azure_api_gateway.mdx";

# Custom Clients [Advanced configuration for custom LLM clients and parameters]
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/pages/agent-integration.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: Integrate Any AI Agent Framework
description: Integrate your agent with Scenario's AgentAdapter interface. Works with any framework - LangGraph, CrewAI, Mastra, OpenAI, custom agents.
description: Learn how to integrate your AI agent with Scenario's AgentAdapter interface. Framework-agnostic guide supporting LangGraph, CrewAI, Mastra, OpenAI, Pydantic AI, and custom agent implementations. Works with any LLM or agent architecture.
---

import { RefLink } from "../components/RefLink";
Expand Down
5 changes: 5 additions & 0 deletions docs/docs/pages/agent-integration/agentkit.mdx
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
---
title: Test Inngest AgentKit Agents
description: Learn how to test agents built with Inngest's AgentKit using Scenario. Complete guide to validating durable execution, workflow orchestration, stateful agent behavior, event-driven workflows, and multi-agent systems with comprehensive automated testing.
---

import { RefLink } from "../../components/RefLink";

# Inngest AgentKit Integration [Learn how to integrate AgentKit agents with the Scenario testing framework]
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/pages/agent-integration/agno.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: Test Agno Agents
description: Test Agno framework agents. Validate agent workflows, tool usage, and multi-step reasoning with simulation-based testing.
description: Learn how to test Agno framework agents with Scenario testing platform. Complete guide to validating agent workflows, tool usage, multi-step reasoning, and conversation history management with comprehensive automated testing.
---

import { RefLink } from "../../components/RefLink";
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/pages/agent-integration/crewai.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: Test CrewAI Multi-Agent Systems
description: Test CrewAI crews and multi-agent systems. Validate crew behavior, task delegation, inter-agent communication with automated simulations.
description: Learn how to test CrewAI crews and multi-agent systems with Scenario. Complete guide to validating crew behavior, task delegation, inter-agent communication, and collaborative workflows with automated simulations and comprehensive testing.
---

import { RefLink } from "../../components/RefLink";
Expand Down
5 changes: 5 additions & 0 deletions docs/docs/pages/agent-integration/google-adk.mdx
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
---
title: Test Google Agent Development Kit (ADK) Agents
description: Learn how to test Google Agent Development Kit (ADK) agents with Scenario. Complete guide to validating Google ADK agent behavior, tool interactions, session management, and Google Cloud integration with comprehensive automated testing.
---

import { RefLink } from "../../components/RefLink";

# Google ADK Integration [Learn how to integrate Google ADK agents with the Scenario testing framework]
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/pages/agent-integration/https.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: Test Agents via HTTPS Endpoints
description: Test AI agents deployed as HTTP/HTTPS services. Learn how to integrate REST APIs and production-deployed agents with Scenario testing framework.
description: Learn how to test AI agents deployed as HTTP/HTTPS services and REST APIs with Scenario. Complete integration guide for testing production-deployed agents, microservices, and serverless functions with blackbox testing methodology.
---

import { RefLink } from "../../components/RefLink";
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/pages/agent-integration/langgraph.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: Test LangGraph Agents
description: Complete guide to testing LangGraph agents. Handle streaming responses, state management, checkpointers, and tool calls in your LangGraph agent tests.
description: Learn how to test LangGraph agents comprehensively with Scenario. Complete guide covering streaming responses, state management, checkpointers, tool calls, and complex graph workflows. Test your LangGraph agents end-to-end with automated simulations.
---

import { RefLink } from "../../components/RefLink";
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/pages/agent-integration/litellm.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: Test LiteLLM Agents - Multi-Provider Testing
description: Test agents using LiteLLM with any LLM provider. Validate behavior across OpenAI, Anthropic, Azure, AWS Bedrock, and 100+ models.
description: Learn how to test agents built with LiteLLM across any LLM provider using Scenario. Comprehensive testing guide for validating agent behavior across OpenAI, Anthropic, Azure, AWS Bedrock, and 100+ supported models with unified interface testing.
---

import { RefLink } from "../../components/RefLink";
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/pages/agent-integration/mastra.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: Test Mastra Agents
description: Test Mastra framework agents. Validate workflows, tool integration, memory, and agent orchestration with simulation-based testing.
description: Learn how to test Mastra framework agents comprehensively with Scenario. Complete guide to validating agent workflows, tool integration, memory management, and agent orchestration with simulation-based testing and automated evaluation.
---

import { RefLink } from "../../components/RefLink";
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/pages/agent-integration/openai.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: Test OpenAI Assistants & Function Calling
description: Test OpenAI Assistant API, function calling, and custom OpenAI agents. Validate tool usage and multi-turn conversations.
description: Learn how to test OpenAI Assistants API, function calling, and custom OpenAI agents with Scenario. Complete guide to validating tool usage, multi-turn conversations, and OpenAI agent behavior with automated testing and user simulation.
---

import { RefLink } from "../../components/RefLink";
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/pages/agent-integration/pydantic-ai.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: Test Pydantic AI Agents
description: Test Pydantic AI agents with type-safe validation. Verify structured outputs, tool calls, and agent behavior with Scenario.
description: Learn how to test Pydantic AI agents with type-safe validation using Scenario. Complete guide to verifying structured outputs, tool calls, conversation history management, and agent behavior with comprehensive testing patterns.
---

import { RefLink } from "../../components/RefLink";
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/pages/basics/cache.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: Caching for Deterministic Agent Tests
description: Make scenario tests deterministic and faster with LLM call caching. Ensure consistent results across test runs while improving execution speed.
description: Learn how to make scenario tests deterministic and faster with LLM call caching. Comprehensive guide to ensuring consistent test results across test runs while dramatically improving execution speed and reducing API costs.
---

import { RefLink } from "../../components/RefLink";
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/pages/basics/concepts.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: Core Concepts
description: Learn scenarios, user simulators, judge agents, and scripted flows. Master the building blocks of simulation-based agent testing.
description: Master the building blocks of simulation-based agent testing with Scenario. Learn about scenarios, user simulators, judge agents, and scripted flows. Complete guide to understanding how these components work together for comprehensive AI agent testing.
---

import { RefLink } from "../../components/RefLink";
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/pages/basics/configuration.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: Configuration - Environment Variables & Models
description: Configure Scenario with environment variables, model settings, logging, and test runner integration for Python and TypeScript.
description: Learn how to configure Scenario with environment variables, model settings, logging levels, and test runner integration. Complete setup guide for both Python and TypeScript implementations with examples for all configuration methods.
---

import { Callout } from "vocs/components";
Expand Down
5 changes: 5 additions & 0 deletions docs/docs/pages/basics/custom-clients.mdx
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
---
title: Custom LLM Clients - Advanced Configuration Guide
description: Learn how to configure custom LLM clients and parameters for Scenario's built-in agents. Complete guide for integrating Azure API Gateway, custom OpenAI clients, and advanced LiteLLM parameters with UserSimulatorAgent and JudgeAgent.
---

# Custom Clients [Advanced configuration for custom LLM clients and parameters]

For advanced use cases, you can pass custom clients and additional parameters to Scenario's built-in agents.
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/pages/basics/debug-mode.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: Debug Mode - Interactive Agent Debugging
description: Step through scenarios interactively. Inspect agent responses, intervene with custom inputs, and debug complex conversations in real-time.
description: Learn how to debug AI agents interactively with Scenario's debug mode. Step through scenarios, inspect agent responses, intervene with custom inputs, and debug complex multi-turn conversations in real-time for faster development.
---

# Debug Mode [Step through scenarios interactively]
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/pages/basics/scripted-simulations.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: Scripted Simulations - Precise Control
description: Control exact conversation flow with scripted simulations. Orchestrate agent interactions, custom assertions, and deterministic test scenarios.
description: Master scripted simulations for precise control over conversation flow in agent tests. Learn how to orchestrate agent interactions, add custom assertions, implement conditional logic, and create deterministic test scenarios with complete control.
---

import { RefLink } from "../../components/RefLink";
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/pages/basics/test-runner-integration.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: Test Runner Integration - Vitest, Jest, Pytest
description: Integrate Scenario with popular test runners. Setup guides for Vitest, Jest, and pytest with rich scenario reporting.
description: Learn how to integrate Scenario with popular test runners for seamless testing workflows. Complete setup guides for Vitest, Jest, and pytest with rich scenario reporting, CI/CD integration, and best practices for both Python and TypeScript projects.
---

# Test Runner Integration
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/pages/basics/writing-scenarios.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: Writing Effective Agent Test Scenarios
description: Best practices for creating robust agent tests. Write clear descriptions, criteria, and assertions for reliable AI agent testing across frameworks.
description: Learn best practices for creating robust and maintainable agent test scenarios. Comprehensive guide covering clear descriptions, evaluation criteria, custom assertions, and testing strategies for reliable AI agent testing across all frameworks.
---

import { RefLink } from "../../components/RefLink";
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/pages/best-practices/domain-driven-tdd.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: Domain-Driven TDD - Build Agents with Test-First Development
description: Use TDD and Domain-Driven Design principles to build AI agents. Write scenarios in business language first, implement minimally, then refactor to reveal domain concepts.
description: Learn how to combine Test-Driven Development with Domain-Driven Design principles for building robust AI agents. Write scenarios in business language first, implement minimally with red-green-refactor cycle, then refactor to reveal domain concepts and improve agent architecture.
---

# Domain-Driven TDD [Using scenarios to drive implementation]
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: The Agent Testing Pyramid
description: Master the agent testing pyramid. Balance unit tests, evals, and simulations for comprehensive AI agent quality assurance and reliable deployments.
description: Master the agent testing pyramid for comprehensive AI agent quality assurance. Learn how to balance unit tests, evals, and simulations for reliable deployments. Three-layered approach to systematic agent testing inspired by traditional testing pyramids.
---

import { Authors } from "vocs/components";
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/pages/best-practices/the-vibe-eval-loop.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: The Vibe-Eval Loop - TDD for AI Agents
description: Rapidly iterate on agent behavior with the vibe-eval loop. Combine human intuition with automated testing for faster, more reliable agent development.
description: Learn the vibe-eval loop methodology for rapidly iterating on agent behavior. Combine human intuition with automated testing, capture edge cases systematically, and scale beyond POCs with a TDD-inspired workflow for faster and more reliable agent development.
---

import { Authors } from "vocs/components";
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/pages/community-support.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: Community & Support
description: Join Scenario community on Discord for help with agent testing. Connect with developers, share patterns, and get support for LangGraph, CrewAI, and custom agents.
description: Join the Scenario community on Discord for help with AI agent testing. Connect with developers, share testing patterns, get technical support for LangGraph, CrewAI, and custom agents. Access documentation, file issues, and stay updated on releases.
---

# Community & Support [Join Scenario community to connect with other users, get help, and explore the agent development ecosystem]
Expand Down
5 changes: 5 additions & 0 deletions docs/docs/pages/examples/multimodal/multimodal-files.mdx
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
---
title: Testing File Analysis Agents - PDF, CSV & Document Processing
description: Learn how to test AI agents that analyze and process files like PDFs, CSVs, and other documents. Comprehensive guide for building Scenario tests where agents parse user-provided files and respond appropriately.
---

# Multimodal File Analysis (Coming Soon)

This page will demonstrate how to build Scenario tests where the user provides **files** (PDF, CSV, etc.) as part of the conversation and the agent must parse and respond appropriately.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
---
title: Testing Image Generation Agents - AI Art & Creative AI
description: Learn how to test AI agents that generate images from text prompts. Comprehensive testing guide for evaluating prompt quality, image generation, and creative AI capabilities with automated judge evaluation.
---

# Multimodal Image Generation (Coming Soon)

This page will demonstrate how to build Scenario tests where the agent generates **images from text prompts** and the judge evaluates both the prompt quality and the generated image quality.
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/pages/examples/multimodal/overview.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: Multimodal Agent Testing - Images, Audio, Files
description: Test multimodal AI agents that process images, audio, and files. Learn patterns for testing voice agents, vision agents, and file processing.
description: Learn how to test multimodal AI agents that process images, audio, and files with Scenario. Comprehensive guide covering testing patterns for voice agents, vision agents, file processing, and combined multimodal inputs with automated evaluation.
---

# Multimodal Use Cases – Overview
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/pages/examples/testing-remote-agents.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: Testing Remote Agents - HTTP Adapter Patterns
description: Learn how to test HTTP-deployed AI agents with Scenario. Adapter patterns for JSON, streaming, SSE, and stateful APIs. Production-ready examples for REST endpoints.
description: Learn how to test HTTP-deployed AI agents comprehensively with Scenario. Complete guide covering adapter patterns for JSON responses, streaming, Server-Sent Events (SSE), and stateful APIs. Production-ready examples and patterns for REST endpoints.
---

# Testing Remote Agents [Adapters for HTTP-deployed agents]
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/pages/index.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: Agent Testing Framework
description: Test AI agents with simulation-based testing. LLM-powered user simulators for agent behavior testing, tool calling, and multi-turn conversations. Works with LangGraph, CrewAI, Pydantic AI.
description: Learn how to test AI agents with simulation-based testing using Scenario. LLM-powered user simulators enable comprehensive agent behavior testing, tool calling validation, and multi-turn conversation testing. Works seamlessly with LangGraph, CrewAI, Pydantic AI, and more.
---

import { Image } from "../components/Image";
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/pages/introduction/getting-started.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: Getting Started - Your First Agent Test
description: Build your first AI agent test in 5 minutes. Step-by-step tutorial with user simulators, automated evaluation, and visual debugging for LangGraph, CrewAI, and custom agents.
description: Learn how to build your first AI agent test in 5 minutes with Scenario. Complete step-by-step tutorial covering user simulators, automated evaluation, and visual debugging for LangGraph, CrewAI, and custom agent frameworks.
---

import { EmbeddedScript } from "../../components/EmbeddedScript";
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/pages/introduction/simulation-based-testing.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: Simulation-Based Testing for AI Agents
description: Why simulation-based testing is essential for AI agents. Learn the testing pyramid for agents and how simulations catch edge cases traditional testing misses.
description: Understand why simulation-based testing is essential for AI agents in production. Learn about the agent testing pyramid and discover how simulations catch edge cases, validate stateful behavior, and test multi-turn conversations that traditional testing methods miss.
---

# Simulation-Based Testing [Multi-agent applications require a new approach to testing]
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/pages/testing-guides/blackbox-testing.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: Blackbox Testing - Test Agents via Public Interfaces
description: Test AI agents through their public interfaces without mocking. Learn how to use HTTP adapters for production-like testing with real databases, authentication, and infrastructure.
description: Learn how to test AI agents through their public interfaces without mocking internal components. Complete guide to using HTTP adapters for production-like testing with real databases, authentication, middleware, and infrastructure - testing exactly how agents run in production.
---

# Blackbox Testing [Testing via public interfaces]
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/pages/testing-guides/fixtures.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: Test Fixtures for Deterministic Agent Tests
description: Create repeatable tests with static fixtures. Use images, audio, and data files for offline-friendly, deterministic agent testing.
description: Learn how to create repeatable, deterministic agent tests with static test fixtures. Complete guide to using images, audio clips, JSON payloads, and data files for offline-friendly testing that ensures consistent results and faster test execution.
---

# Fixtures [Static test assets for deterministic scenarios]
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/pages/testing-guides/mocks.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: Mocking External APIs in Agent Tests
description: Mock external dependencies for fast, reliable agent tests. Isolate agent behavior from third-party services with tool mocking patterns.
description: Learn how to mock external dependencies for fast, reliable agent tests. Complete guide to isolating agent behavior from third-party services with tool mocking patterns, database mocking, API service mocking, and testing tool failures for deterministic offline testing.
---

import { LanguageTabs } from "../../components/LanguageTabs";
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/pages/testing-guides/tool-calling.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: Testing AI Agent Tool Calls & Function Calling
description: Test and validate agent tool usage. Assert on function calls, mock tool responses, verify tool call arguments, and test tool calling flows.
description: Learn how to test and validate AI agent tool usage comprehensively. Complete guide covering assertions on function calls, mocking tool responses, verifying tool call arguments, testing tool calling flows, and validating agent tool selection behavior.
---

# Testing Tool Calls in Scenarios
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/pages/visualizations/index.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: Simulation Visualizer - Debug Agent Tests
description: Visualize agent test results with LangWatch. Browse conversations, debug tool calls, share results with your team, and improve agents faster.
description: Learn how to visualize agent test results with LangWatch's simulation visualizer. Browse conversations, debug tool calls, analyze agent behavior, share results with your team, and improve agents faster with visual debugging tools.
---

import { Callout } from "vocs/components";
Expand Down
2 changes: 1 addition & 1 deletion python/scenario/__init__.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
"""
Scenario: Agent Testing Framework through Simulation Testing
Access Scenario API root documentation to build simulation-based evaluations and structured AI agent testing flows.

Scenario is a comprehensive testing framework for AI agents that uses simulation testing
to validate agent behavior through realistic conversations. It enables testing of both
Expand Down
2 changes: 1 addition & 1 deletion python/scenario/agent_adapter.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
"""
Agent adapter module for integrating custom agents with the Scenario framework.
Explore the Scenario Python API to integrate custom agents into simulation-based AI agent tests within LangWatch.

This module provides the abstract base class that users must implement to integrate
their existing agents with the Scenario testing framework. The adapter pattern allows
Expand Down
Loading
Loading