pluto-llm-diff 0.1.0

git diff, but for LLM behavior — behavioral regression testing for language models.

llm-diff ⚡

git diff, but for LLM behavior.

You swapped gpt-4o for gpt-4o-mini to cut costs. You upgraded Llama 3.1 to 3.2 because the benchmarks went up. Did your model's behavior actually get better — or did it just get chattier, start ignoring your formatting rules, and flip its answers when you rephrase a question?

Benchmarks measure capability. llm-diff measures behavior — and shows you exactly what changed, in the format every engineer already speaks: a diff.

- baseline   ollama/qwen2.5:0.5b
+ candidate  openai/gpt-4o-mini

One command. Three behavioral dimensions. Colored deltas. CI-friendly exit codes.

Install

pip install llm-diff

Quickstart

Compare a local Ollama model against an OpenAI model:

export OPENAI_API_KEY=sk-...
llm-diff ollama/qwen2.5:0.5b openai/gpt-4o-mini

Illustrative output (baseline vs candidate shown for clarity):

- baseline   ollama/baseline-model
+ candidate  openai/candidate-model

╭──────────────────────┬──────────────────┬──────────────┬──────────────┬────────╮
│ Dimension            │ Metric           │ - baseline   │ + candidate  │      Δ │
├──────────────────────┼──────────────────┼──────────────┼──────────────┼────────┤
│ Instruction Fidelity │ fidelity score   │ 1.00         │ 0.50         │  -0.50 │
│ Verbosity Profile    │ total words      │ 41           │ 118          │    +77 │
│                      │ preamble words   │ 0            │ 9            │     +9 │
│                      │ markdown headers │ 0            │ 3            │     +3 │
│ Reasoning Consistency│ consistency      │ 1.00         │ 1.00         │   0.00 │
│ Latency              │ avg latency (s)  │ 0.84         │ 1.92         │  +1.08 │
╰──────────────────────┴──────────────────┴──────────────┴──────────────┴────────╯
        Δ = candidate − baseline · green improvement · red regression

More recipes:

# Two local models, fully offline
llm-diff ollama/qwen2.5:0.5b ollama/llama3.2:1b

# Any OpenAI-compatible server: vLLM, LM Studio, llama.cpp, TGI, ...
llm-diff openai/my-finetune openai/my-base \
  --openai-base-url http://localhost:8000/v1

# Open-weight models via Hugging Face Inference Providers (OpenAI-compatible)
llm-diff "openai/meta-llama/Llama-3.3-70B-Instruct" "openai/Qwen/Qwen3-8B" \
  --openai-base-url https://router.huggingface.co/v1 --api-key $HF_TOKEN

# Machine-readable report for dashboards and CI artifacts
llm-diff ollama/qwen2.5:0.5b openai/gpt-4o-mini --json > diff.json

# Audit the raw responses behind every score
llm-diff ollama/qwen2.5:0.5b ollama/llama3.2:1b --show-responses

Model specs are backend/model. Unprefixed specs fall back to --backend (default: ollama), and only the first slash is parsed — so HuggingFace-style ids like meta-llama/Llama-3.3-70B-Instruct pass through intact.

The three dimensions (MVP)

Dimension	The question it answers	How it's scored
Instruction Fidelity	Does the model obey strict formatting constraints?	Probe demands exactly 3 bullets, no introduction. +0.5 for exactly 3 bullet lines, +0.5 for a clean start with zero preamble.
Verbosity Profile	How much does the model say, and how much of it is filler?	One open-ended explanation, measured three ways: total words, filler-preamble words ("Certainly! Here is…"), and markdown header count.
Reasoning Consistency	Does the same logic get the same answer when reworded?	Two isomorphic syllogisms with nonsense words (memorization-resistant). Score 1.0 if the Yes/No verdict survives the reframing, 0.0 if it flips.

Every diff runs at temperature=0 by default, so results are as reproducible as the backends allow.

Built for CI

llm-diff speaks exit codes: 0 all probes succeeded, 1 fatal config error, 2 usage error, 3 partial diff (some probes failed — shown as ERR rows, never a crash). Transient failures (timeouts, 429s, 5xx) are retried with exponential backoff; hard failures surface immediately with actionable messages.

# .github/workflows/model-swap.yml (sketch)
- run: llm-diff ollama/current-prod ollama/candidate --json > diff.json
- run: python scripts/check_thresholds.py diff.json   # gate on deltas

Native --fail-if "fidelity<0.8" gating is on the roadmap below.

Roadmap: from 3 dimensions to 7

The MVP battery is deliberately small — a fast smoke test, not a benchmark. The v1.0 goal is seven dimensions with a much larger, community-built probe set:

#	Dimension	Status
1	Instruction Fidelity	✅ shipped
2	Verbosity Profile	✅ shipped
3	Reasoning Consistency	✅ shipped
4	Hallucination Propensity — does the model invent facts, citations, and APIs under pressure?	🔜 planned
5	Refusal Sensitivity — does the model over-refuse benign requests, and did that change between versions?	🔜 planned
6	Domain Retention — did niche knowledge (SQL dialects, legal terms, medical vocab) survive the new release?	🔜 planned
7	Sycophancy Drift — does the model cave and change correct answers when the user pushes back?	🔜 planned

Also planned for v0.2: custom probe packs via YAML (llm-diff a b --probes my_domain.yaml), N-way comparison matrices, a native hf/ backend prefix for Hugging Face Inference Providers, and --fail-if threshold gating for CI.

Contributing

The most valuable thing you can contribute is a probe set. The three built-in probes prove the mechanism; the community can make it comprehensive. If you know how models should behave in your domain — medicine, law, code review, customer support, your native language — you can encode that knowledge as probes + scorers, no ML expertise required.

Great first contributions:

1. Probe packs — propose a set of prompts + expected behaviors for one dimension (open an issue with the probe-pack label to discuss the format).
2. Scorers — every scorer is a pure function of response text in llm_diff/scorers.py, trivially unit-testable. Sharpen the heuristics or add new ones.
3. Backends — the transport layer in llm_diff/core.py is ~100 lines per backend.

Dev setup:

git clone https://github.com/SiddharthFX/llm-diff
cd llm-diff
pip install -e ".[dev]"
pytest   # fully offline — tests run against a bundled mock backend

See CONTRIBUTING.md for guidelines. Adding a dimension end-to-end touches exactly three places: a prompt in probes.py, a score_* function in scorers.py, and one hook in evaluate() — the table, summary, JSON report, and exit codes pick it up automatically.

Research

This tool exists to answer a bigger question: do successive LLM releases trade behavioral fidelity for benchmark scores? Read the study design in WHITEPAPER.md — and help us run it.

License

MIT © 2026 Pluto AI Research Lab