llm-diff 1.3.0

A CLI tool for comparing LLM outputs — semantically, visually, and at scale

llm-diff

A CLI tool and Python library for comparing LLM outputs — semantically, visually, and at scale.

---

llm-diff calls two LLM models in parallel, diffs their responses word-by-word, scores them semantically, and renders results in the terminal or as a self-contained HTML report. It scales to batch workloads, caches API responses, gates CI pipelines via --fail-under, and emits structured llm-toolkit-schema events for observability tooling.

What is llm-diff?

LLMs do not produce deterministic output. Evaluating models, iterating on prompts, or assessing the impact of a model upgrade all require you to compare responses — and doing that by hand does not scale.

llm-diff automates the entire workflow: it calls both models concurrently, produces a word-level diff, optionally scores semantic similarity via sentence embeddings, and outputs results to the terminal or as a shareable HTML report. It supports batch workloads from a YAML file, caches API calls so iterating on thresholds costs nothing, and emits exit code 1 when similarity falls below a threshold — making it a first-class citizen in CI/CD pipelines.

Version 1.2 adds LLM-as-a-Judge scoring, per-call USD cost tracking, multi-model (3–4 model) comparison, and structured JSON diff.

Version 1.3.0 adds EVAL_REGRESSION_FAILED schema event emission — --fail-under gate failures now emit a structured llm.eval.regression.failed event (via make_eval_regression_event()) in addition to returning exit code 1, providing a full audit trail for CI regression gates.

Version 1.2.2 integrates llm-toolkit-schema as a built-in observability layer: every comparison, model call, cache lookup, cost record, judge evaluation, and --fail-under regression failure now emits a validated schema event that can be collected in memory, exported to JSONL, or forwarded to any custom backend.

Documentation

Guide	Description
Getting Started	Installation, API keys, first diff
Tutorials	Step-by-step learning path from first run to Python API (12 tutorials)
CLI Reference	All flags, option groups, exit codes, YAML format
Python API	All public functions, dataclasses, and field descriptions
Schema Events	Observability integration with llm-toolkit-schema
Configuration	.llmdiff TOML schema, env vars, config priority
Provider Setup	OpenAI, Groq, Mistral, Ollama, LM Studio, Anthropic
HTML Reports	Report anatomy, batch reports, judge card, cost table
CI / CD Integration	GitHub Actions examples, threshold recommendations

Quick Start

# Install with semantic scoring support
pip install "llm-diff[semantic]"

# Install with schema-events observability
pip install "llm-diff[semantic]" llm-toolkit-schema

# Set an API key
export OPENAI_API_KEY="sk-..."

# Compare two models on the same prompt
llm-diff "Explain recursion in one sentence." -a gpt-4o -b gpt-4o-mini --semantic

# Save a self-contained HTML report
llm-diff "Explain recursion." -a gpt-4o -b gpt-4o-mini --semantic --out report.html

# Run a batch from a YAML prompt file and gate on similarity
llm-diff --batch prompts.yml -a gpt-4o -b gpt-4o-mini --semantic --fail-under 0.85

See Getting Started for quick examples, or work through the Tutorials for a guided learning path covering prompt engineering, batch evaluation, CI/CD gating, LLM-as-a-Judge, cost tracking, and the Python API.

Getting Help

Bug reports	Open an issue
Feature requests	Open a feature request
Questions & discussion	GitHub Discussions
Open issues	github.com/veerarag1973/llmdiff/issues
PyPI project page	pypi.org/project/llm-diff
Roadmap	IMPLEMENTATION_PLAN.md
Changelog	CHANGELOG.md

When filing a bug, please include: llm-diff --version, your OS, Python version, the full command you ran, and the complete error output.

Contributing

See CONTRIBUTING.md for development setup, running the test suite, code style guidelines, and pull request instructions.

License

llm-diff is distributed under the MIT License.