model-diff

Compare LLM model outputs side-by-side with rich diff visualization.

Run the same prompt on multiple models simultaneously and see exactly what each model says differently.

Installation

pip install model-diff

Or install from source:

git clone https://github.com/yourname/model-diff
cd model-diff
pip install -e .

Requirements

Set the API keys for the providers you want to use:

export OPENAI_API_KEY=sk-...
export ANTHROPIC_API_KEY=sk-ant-...

Missing keys are handled gracefully — models without a key are skipped with a warning.

Usage

# Default: compare GPT-4o vs Claude Sonnet
model-diff "What is the best way to handle errors in Python?"

# Specify models explicitly
model-diff "Explain recursion" --models gpt-4o,claude-sonnet-4-6

# Use a prompt file
model-diff --prompt prompt.txt --models gpt-4o,claude-haiku-4-5-20251001,claude-sonnet-4-6

# Word-level diff
model-diff "Explain recursion" --diff words

# Show only differences (hide matching sections)
model-diff "Explain recursion" --only-diff

# Deterministic outputs
model-diff "Explain recursion" --temperature 0.0

# Save results to JSON
model-diff "Explain recursion" --output results.json

Supported Models

Architecture

src/model_diff/
├── cli.py      # Click-based CLI entry point
├── models.py   # Provider-specific API callers, run concurrently via threading
└── differ.py   # difflib-based diff engine + Rich output formatter

Model calls are issued concurrently using threading, so wall time equals the slowest model rather than the sum of all models.

License

MIT