llm-diff

A CLI tool and Python library for comparing LLM outputs — semantically, visually, and at scale.

llm-diff calls two LLM models in parallel, diffs their responses word-by-word, scores them semantically, and renders results in the terminal or as a self-contained HTML report. It scales to batch workloads, caches API responses, gates CI pipelines via --fail-under, and emits structured AgentOBS events for observability tooling.

What is llm-diff?

LLMs do not produce deterministic output. Evaluating models, iterating on prompts, or assessing the impact of a model upgrade all require you to compare responses — and doing that by hand does not scale.

llm-diff automates the entire workflow: it calls both models concurrently, produces a word-level diff, optionally scores semantic similarity via sentence embeddings, and outputs results to the terminal or as a shareable HTML report. It supports batch workloads from a YAML file, caches API calls so iterating on thresholds costs nothing, and emits exit code 1 when similarity falls below a threshold — making it a first-class citizen in CI/CD pipelines.

Version 1.2 adds LLM-as-a-Judge scoring, per-call USD cost tracking, multi-model (3–4 model) comparison, and structured JSON diff.

Version 1.3.0 adds EVAL_REGRESSION_DETECTED schema event emission — --fail-under gate failures now emit a structured llm.eval.regression.detected event (via make_eval_regression_event()) in addition to returning exit code 1, providing a full audit trail for CI regression gates.

Version 1.2.2 integrates AgentOBS as a built-in observability layer: every comparison, model call, cache lookup, cost record, judge evaluation, and --fail-under regression failure now emits a validated schema event that can be collected in memory, exported to JSONL, or forwarded to any custom backend.

Documentation

Quick Start

# Install with semantic scoring support
pip install "llm-diff[semantic]"

# Install with schema-events observability
pip install "llm-diff[semantic]" agentobs

# Set an API key
export OPENAI_API_KEY="sk-..."

# Compare two models on the same prompt
llm-diff "Explain recursion in one sentence." -a gpt-4o -b gpt-4o-mini --semantic

# Save a self-contained HTML report
llm-diff "Explain recursion." -a gpt-4o -b gpt-4o-mini --semantic --out report.html

# Run a batch from a YAML prompt file and gate on similarity
llm-diff --batch prompts.yml -a gpt-4o -b gpt-4o-mini --semantic --fail-under 0.85

See Getting Started for quick examples, or work through the Tutorials for a guided learning path covering prompt engineering, batch evaluation, CI/CD gating, LLM-as-a-Judge, cost tracking, and the Python API.

Getting Help

When filing a bug, please include: llm-diff --version, your OS, Python version, the full command you ran, and the complete error output.

Contributing

See CONTRIBUTING.md for development setup, running the test suite, code style guidelines, and pull request instructions.

License

llm-diff is distributed under the MIT License.