pytest for LLMs — Write fast, zero-cost tests for your AI prompts, and optimize them automatically when they fail.

ghostrun gives you the two things most Gen AI developers end up building from scratch: a way to write reliable, offline-ready prompt tests, and a way to automatically optimize those prompts when they break. Both live in plain Python and plain pytest — no cloud dashboards, no complex YAML, no extra test harnesses.

Is this for you? · Install · Quickstart · Prompt Optimization · Documentation · Roadmap

---

Tip

TDD for LLMs (Test-Driven Prompts): Instead of guessing prompts in a notebook, write your test and criteria first (using ghostrun.expect), then let the optimizer (ghostrun craft) automatically tune the prompt instructions and few-shots to pass the test.

The Problem: Why LLM testing is frustrating

Building an AI application usually leaves you with three painful problems:

1. Slow & Expensive Tests: Every time you run pytest, your test suite calls OpenAI/Anthropic APIs, costing you money and taking forever to finish.
2. Brittle Exact Assertions: LLM outputs change slightly on every run. Writing assert reply == "expected" fails randomly because the LLM used a different word.
3. The Prompt Guessing Game: When you edit a prompt to fix one edge case, you have no easy way to know if you silently broke another output somewhere else.

How ghostrun fixes it

ghostrun solves all three issues by bringing standard software testing workflows to AI:

• Fast & Free Tests (Deterministic Replay): Wrap your test in @ghostrun.record. The first run hits the real API and saves the response; every run after replays it instantly from disk. Tests run in 0.05 seconds, cost nothing, and run fully offline in CI.
• No More Flakiness (Semantic Assertions): Assert on meaning and intent instead of exact text. ghostrun.expect(reply).tone_is("empathetic") is graded by a free, local judge (via Ollama) so your data stays private.
• Self-Healing Prompts (Optimizers): If your prompt fails the test, run ghostrun craft with your target criteria and training examples. The optimizer automatically searches for the best instructions and few-shot examples to pass your tests.

No SaaS dashboards. No complicated YAML. Just Python and pytest.

Search terms this project is built for: LLM evals in CI, LLM regression testing, pytest LLM evals, deterministic LLM tests, prompt engineering framework, prompt optimization, few-shot example selection, semantic assertions for LLM apps, and testing OpenAI or Anthropic applications with pytest.

---

Install

pip install ghostrun

For the default (local, free, private) judge, install Ollama and pull a small model:

ollama pull llama3.2:3b

If anything doesn't work, ghostrun doctor diagnoses the setup — see Configuration.

Fastest start: ghostrun init scaffolds a working first test against whatever LLM SDK it finds in your project (OpenAI, Anthropic, or a generic HTTP fallback) plus a .ghostrun.yaml — no config to author by hand:

ghostrun init
pytest test_ghostrun_example.py

Quickstart

# test_customer_support.py
import ghostrun
from my_app import generate_reply

@ghostrun.record(model="gpt-4o-mini")
def test_reply_generation():
    reply = generate_reply("Where is my refund?")

    ghostrun.expect(reply).contains_intent("apology")
    ghostrun.expect(reply).contains_intent("refund policy")
    ghostrun.expect(reply).does_not_contain_intent("arguing")
    ghostrun.expect(reply).tone_is("empathetic")

$ pytest test_customer_support.py
================================ test session starts ================================
collected 1 item

test_customer_support.py .                                                     [100%]
================================ 1 passed in 0.04s =================================

The 0.04s is the whole point — after the first record, calls replay from disk.

Note on the API: the assertion entry point is ghostrun.expect(...), not ghostrun.assert(...) — assert is a reserved Python keyword and cannot be a function name.

Record/replay alone needs no Ollama at all — the judge is only touched when you call a judge-backed assertion (contains_intent, tone_is, matches). Deterministic assertions (contains, is_valid_json) and tool-call assertions never invoke it.

Prompt Optimization (ghostrun craft)

Where @ghostrun.record and expect(...) test an existing prompt, ghostrun craft builds and optimizes one.

Instead of hand-writing brittle prompts and guessing edge cases, declare a typed input/output signature ("inputs -> outputs") and let ghostrun craft automatically search for winning instructions and discover high-value few-shot demonstrations:

from ghostrun.craft import craft

# Automatically synthesize prompt instructions & select passing few-shots
crafted = craft(
    name="refund_classifier",
    signature="customer_message -> is_refund_request, urgency",
    examples_path="dataset/support_queries.jsonl",
    criterion="Accurately flags refund requests and evaluates urgency",
    model="gpt-4o-mini",
    budget=10,  # Bayesian search over instruction candidates & demo bootstrapping
)

print(crafted.instructions)
# Resulting prompt artifacts and demos are saved locally for test replay

Or optimize directly from the CLI:

ghostrun craft refund_classifier \
    --signature "customer_message -> is_refund_request, urgency" \
    --examples dataset.jsonl \
    --criterion "Accurately flags refund requests and evaluates urgency" \
    --model gpt-4o-mini

Interactive Tactical Companion (ghostrun pet)

GhostRun includes a borderless, transparent desktop tactical companion with 9 real-time animated states that react to your development workflow:

ghostrun run test_app.py        # Runs tests & triggers victory celebration dance on 0.04s replay
ghostrun pet                    # Launches transparent floating companion on screen
ghostrun pet --anim thinking    # Hyper-speed thinking animation

• Interactive Controls: Left-click to cycle through all 9 animations (idle, running, jumping, waving, review, waiting, failed), drag anywhere across multiple monitors, or click the hover red ✕ button to dismiss.
• Workflow Triggers:
  • ghostrun init → 👋 waving welcome greeting.
  • ghostrun run → 🏆 jumping celebration on passing tests / 💀 failed on regression.
  • ghostrun craft → 🎖️ review tactical salute on discovering winning prompts.

Learn more in the Prompt Crafting Guide.

Is this for you?

Use ghostrun if you're writing pytest tests around code that calls an LLM (directly or via the OpenAI/Anthropic SDKs) and want that suite to run offline, free, and deterministically after the first recording.

Skip it if you need a hosted dashboard/observability platform for production traffic (see Langfuse/LangSmith/Braintrust instead), you're building a red-team/adversarial test suite (see Giskard), or you want 50+ pre-built judge metrics out of the box today (see DeepEval — more mature, more metrics, but doesn't intercept your app's own HTTP calls the way ghostrun does). See doc/comparison.md for the full, researched breakdown of where ghostrun is ahead and where it's duplicating existing work.

Documentation

Start here, in order:

Guide	What's in it
LLM regression testing	CI-native LLM evals for catching semantic and prompt regressions in real app code
Pytest LLM evals	How to write LLM evals as normal pytest tests instead of dashboard-only workflows
Test OpenAI apps offline	Record/replay OpenAI and Anthropic API calls so CI does not repeat live model calls
doc/guide/recording.md	How record/replay works, judge-verdict caching, supported providers, secret redaction, parallel test runs
doc/guide/assertions.md	Semantic assertions, judge reliability (benchmarked, not asserted), majority-vote verdicts, tool/function-call assertions
doc/guide/craft.md	Prompt synthesis, signatures (input -> output), Bayesian instruction search, and few-shot bootstrapping
doc/guide/configuration.md	.ghostrun.yaml, environment variables, pytest flags, ghostrun doctor, ghostrun init

Deeper reference, once you're past the basics

Guide	What's in it
doc/guide/regression-tracking.md	Snapshotting runs, ghostrun diff, posting a regression as a PR comment, JUnit CI integration
doc/guide/api-reference.md	Every public function, class, exception, and config field
doc/guide/why-not-diy.md	The actual bugs found building this — the case for a maintained package over a five-minute prompt
doc/judge-voting-benchmark.md	Full methodology and results for the majority-vote judge-caching benchmark
doc/comparison.md	Researched comparison against DeepEval, Promptfoo, Ragas, vcr-langchain, and 9 other tools
CHANGELOG.md	Release notes

A hosted, searchable version of this documentation is planned at ghostrun.parthmax.tech (config in mkdocs.yml, builds via .github/workflows/docs.yml).

Roadmap

• Deterministic HTTP record/replay (@ghostrun.record, .ghostrun_cache/)
• Semantic assertions (contains_intent, tone_is, matches) via local Ollama or an offline echo stub
• Judge-verdict caching, including majority-vote grading (judge.votes) with a benchmarked reliability tradeoff
• Tool/function-call assertions (expect_tool_calls)
• Prompt synthesis & optimization (ghostrun craft, Signature, BootstrapFewShot, BayesianSearch)
• Prompt regression tracking — ghostrun diff, PR-comment and JUnit CI output
• ghostrun init / ghostrun doctor — scaffolding and setup diagnostics in one command
• Secret redaction so the cache is safe to commit
• 16-provider HTTP coverage (OpenAI, Anthropic, Gemini, Bedrock, and more)
• Published to PyPI with trusted-publishing (OIDC) releases
• Hosted, searchable documentation site (mkdocs.yml ready; not yet deployed)
• Broader provider/framework integration guides

Contributing

See CONTRIBUTING.md — setup, test requirements, and where things live in the codebase.

Development

pip install -e ".[dev]"
pytest            # runs fully offline using the echo judge

License

MIT