ghostrun 0.1.1

pytest for LLMs: deterministic HTTP record/replay, local LLM-as-judge semantic assertions, and prompt regression diffing for testing GenAI applications.

ghostrun

pytest for LLMs. Deterministic record/replay and semantic assertions for GenAI apps — local-first, privacy-first, zero SaaS lock-in.

The problem

reply = generate_reply("Where is my refund?")
assert reply == "I'm sorry for the delay..."   # fails tomorrow: LLM never says the same thing twice

Every real test run also means a live API call — slow, costs money, and now your CI needs a secret API key just to run the test suite.

What ghostrun does about it

1. Deterministic replay — the first run records real LLM HTTP calls to a local .ghostrun_cache/; every run after replays them instantly from disk. Zero API cost, zero latency, zero flakiness, no key needed in CI.
2. Semantic assertions — assert on meaning, not exact text:

   ghostrun.expect(reply).contains_intent("apology")
   ghostrun.expect(reply).tone_is("empathetic")
   

   Graded by a local Ollama model by default — your prompts and data never leave your machine. That grading verdict gets cached too, so it's also free and deterministic after the first run.

No cloud dashboard. No custom CLI to learn. No dataset to author by hand. Just pytest.

---

Install

pip install ghostrun

For the default (local, free, private) judge, install Ollama and pull a small model:

ollama pull llama3.2:3b

If anything doesn't work, ghostrun doctor diagnoses the setup — see Configuration.

Fastest start: ghostrun init scaffolds a working first test against whatever LLM SDK it finds in your project (OpenAI, Anthropic, or a generic HTTP fallback) plus a .ghostrun.yaml — no config to author by hand:

ghostrun init
pytest test_ghostrun_example.py

Quickstart

# test_customer_support.py
import ghostrun
from my_app import generate_reply

@ghostrun.record(model="gpt-4o-mini")
def test_reply_generation():
    reply = generate_reply("Where is my refund?")

    ghostrun.expect(reply).contains_intent("apology")
    ghostrun.expect(reply).contains_intent("refund policy")
    ghostrun.expect(reply).does_not_contain_intent("arguing")
    ghostrun.expect(reply).tone_is("empathetic")

$ pytest test_customer_support.py
================================ test session starts ================================
collected 1 item

test_customer_support.py .                                                     [100%]
================================ 1 passed in 0.04s =================================

The 0.04s is the whole point — after the first record, calls replay from disk.

Note on the API: the assertion entry point is ghostrun.expect(...), not ghostrun.assert(...) — assert is a reserved Python keyword and cannot be a function name.

Record/replay alone needs no Ollama at all — the judge is only touched when you call a judge-backed assertion (contains_intent, tone_is, matches). Deterministic assertions (contains, is_valid_json) and tool-call assertions never invoke it.

Is this for you?

Use ghostrun if you're writing pytest tests around code that calls an LLM (directly or via the OpenAI/Anthropic SDKs) and want that suite to run offline, free, and deterministically after the first recording.

Skip it if you need a hosted dashboard/observability platform for production traffic (see Langfuse/LangSmith/Braintrust instead), you're building a red-team/adversarial test suite (see Giskard), or you want 50+ pre-built judge metrics out of the box today (see DeepEval — more mature, more metrics, but doesn't intercept your app's own HTTP calls the way ghostrun does). See doc/comparison.md for the full, researched breakdown of where ghostrun is ahead and where it's duplicating existing work.

Documentation

Start here, in order:

Guide	What's in it
doc/guide/recording.md	How record/replay works, judge-verdict caching, supported providers, secret redaction, parallel test runs
doc/guide/assertions.md	Semantic assertions, judge reliability (benchmarked, not asserted), majority-vote verdicts, tool/function-call assertions
doc/guide/configuration.md	.ghostrun.yaml, environment variables, pytest flags, ghostrun doctor, ghostrun init

Deeper reference, once you're past the basics:

Guide	What's in it
doc/guide/regression-tracking.md	Snapshotting runs, ghostrun diff, posting a regression as a PR comment, JUnit CI integration
doc/guide/api-reference.md	Every public function, class, exception, and config field
doc/guide/why-not-diy.md	The actual bugs found building this — the case for a maintained package over a five-minute prompt
doc/judge-voting-benchmark.md	Full methodology and results for the majority-vote judge-caching benchmark
doc/comparison.md	Researched comparison against DeepEval, Promptfoo, Ragas, vcr-langchain, and 9 other tools
CHANGELOG.md	Release notes

A hosted, searchable version of this documentation is planned at parthmax2.github.io/ghostrun (config in mkdocs.yml, builds via .github/workflows/docs.yml).

Contributing

See CONTRIBUTING.md — setup, test requirements, and where things live in the codebase.

Development

pip install -e ".[dev]"
pytest            # runs fully offline using the echo judge

License

MIT