whydunit 0.1.1

Find out why an LLM pipeline failed, by counterfactual replay instead of guesswork.

whydunit

Find out why an LLM pipeline failed — by replaying it under targeted counterfactuals, not by asking a model to guess.

Tracing tools show you what happened. They do not tell you what went wrong. whydunit takes failed runs, re-runs them with one thing changed at a time, and reports which cause is consistent with the evidence — with the arms and the p-value that produced each verdict.

No account, no cloud, no framework, no rewrite of your pipeline. Python 3.10+, zero runtime dependencies.

pip install https://github.com/alex-hahn/whydunit/releases/download/v0.1.1/whydunit-0.1.1-py3-none-any.whl
whydunit demo          # bundled traces, no API key, no cost

(Not on PyPI yet. Installing from the release wheel is the same package — there are no dependencies to resolve.)

---

The two-minute path

whydunit init                                  # find the traces you already record
whydunit diagnose runs.jsonl \
    --runner mypackage.pipeline:answer \
    --failures 50 --budget 400

48 failures; the largest group is distractor (7, 15%) — fix ranking or the
index: deduplicate near-copies, drop stale versions, rerank

! 6 failure(s) had two or more interacting causes; fixing any single one of
  them will not move those runs

where the failures are
  distractor        7   15%  ████████████████████
      43% of them alike: mentions 'time', 'maintenance', 'changeover'
  position          7   15%  ████████████████████
      43% of them alike: mentions 'time', 'lead', 'part'
  reasoning         7   15%  ████████████████████
      100% of them alike: mentions 'capacity', 'handbook', 'hours'
  representation    7   15%  ████████████████████
      57% of them alike: mentions 'handbook'
  retrieval         7   15%  ████████████████████
  scorer            7   15%  ████████████████████
  compound          6   12%  █████████████████

what to fix first
  1. position — 7 failures (15%)
     reorder the context: put the highest-scoring blocks at the edges, not the
     middle
  2. scorer — 7 failures (15%)
     fix the metric: normalise numbers and casing, or replace exact match with
     a predicate

48 failures diagnosed, 210 model calls (4.4 per failure), 88% attributed

That last section is the point. Not "quality 73%" — a number for a slide — but three findings that go to three different people.

What it actually does

For each failed run (successful ones are never re-run, so they cost nothing), whydunit tests one hypothesis at a time and stops at the first that holds:

intervention	if this repairs the run	the fix is
rescore under a laxer oracle (free)	scorer	the metric
move the strongest block to an edge	position	context ordering
re-serialise a block, same values	representation	the chunker
replace a suspect block with neutral filler	distractor	the index / ranker
insert a block stating the answer	retrieval	retrieval
nothing does, not even a perfect context	reasoning	the prompt or the model

Two things separate this from "show the trace to GPT and ask what went wrong":

Nothing is diagnosed by inspection. The verdict is never "this looks like a retrieval problem". It is "failed 5/5, passed 5/5 once a block stating the answer was inserted, p = 0.03". Nothing in a trace distinguishes "the answer was absent" from "the answer was present and overlooked" — only changing the context and running it again does.

One successful re-run is not evidence. Model outputs are stochastic; on a task with a 30% pass rate you will see a "repair" one attempt in three by doing nothing. Repairs are measured as a paired comparison under common random numbers, scored with an exact McNemar test. On a deterministic pipeline whydunit detects that, runs one seed, and labels the finding as structural rather than statistical — rather than dressing up a tautology as a p-value.

How often it is right

Measured, not asserted. make validate reproduces every number below in a few seconds, with no API key. See docs/validation.md for how the ground truth is constructed and verified.

regime	accuracy	macro-F1	calls/trace	undetermined	by-hand baseline
deterministic, one cause	100%	1.00	4.0	0	66%
plus interacting causes	100%	1.00	4.5	0	55%
stochastic pipeline (noise 0.35)	69%	0.73	15.0	37	66%
tight budget (6 calls/trace)	79%	0.83	3.8	30	66%

The deterministic matrix, n=140:

true \ predicted	distractor	position	reasoning	representation	retrieval	scorer
distractor	24	0	0	0	0	0
position	0	23	0	0	0	0
reasoning	0	0	23	0	0	0
representation	0	0	0	23	0	0
retrieval	0	0	0	0	24	0
scorer	0	0	0	0	0	23

And the one where it falls apart. Under a sampling pipeline, accuracy drops 31 points:

class	precision	recall
distractor	1.00	0.08
position	1.00	0.26
representation	1.00	0.83
retrieval	0.96	1.00
reasoning	0.82	1.00
scorer	1.00	1.00

Distractor and position collapse to undetermined: with five repeats and a noisy pipeline, a real repair often cannot clear p ≤ 0.05. The failure mode is abstention, not confident error — precision stays at or near 1.00 where recall falls off a cliff. The exception is worth naming: four distractor cases came back as reasoning because noise broke the perfect-context test too. That is 3% of the sample, and it is confidently wrong.

The comparison that matters is not against nothing. The surface baseline — is the answer anywhere in the retrieved context, and did the model refuse or assert? — is the check teams already run by hand, costs nothing, and gets 66% right. Under heavy sampling noise, whydunit's advantage over it nearly disappears. If that is your regime, raise --repeats or expect a report that mostly says undetermined.

What this cannot do

• It cannot diagnose a run it cannot re-run. A trace file alone is not enough. whydunit refuses rather than falling back to reading the text and guessing — that is the thing it exists not to do.
• It cannot address a context logged as one concatenated string. Blocks have to be addressable, or the distractor and position tests are silently inapplicable and everything comes back retrieval.
• It cannot separate retrieval from reasoning given only a final answer. Inserting a block that states the answer repairs both. The verdict carries that caveat; supplying gold_block_ids removes it.
• It cannot be better than its oracle. A judge below κ = 0.6 makes every downstream number a hypothesis, and whydunit prints that next to them.
• The numbers above are from a simulator. They bound the method's discriminative power where ground truth is known. Real corpora have skewed cause distributions, more than two interacting defects, and uncertain oracles.

Integration

Reads what you already have — OpenTelemetry spans, Langfuse / LangSmith / Braintrust exports, or a two-field JSONL schema:

{"query": "...", "context_blocks": [{"id": "c7", "text": "...", "source": "..."}],
 "output": "...", "reference": "..."}

No traces at all? Point it at the pipeline:

from whydunit import Diagnoser, FunctionRunner, load_traces, summarise

runner = FunctionRunner(lambda query, blocks: my_chain(query, blocks))
diagnoses = Diagnoser(runner).diagnose_all(load_traces("runs.jsonl"))
print(summarise(diagnoses).headline())

Reports come out as a terminal table, Markdown or JSON. There is no dashboard; anyone who wants one has one.

Documentation

taxonomy.md	the six causes, and why the cascade tests them in that order
interventions.md	why "remove the block" is a choice, not an operation
oracles.md	grading, and calibrating a judge before trusting it
validation.md	how the numbers were produced and what they omit
adapters.md	getting your runs in
architecture.md	the module map

Prior work

The counterfactual idea is not new — step-wise intervention to score causal responsibility in agent traces has been published, and "lost in the middle", distractor sensitivity and format brittleness are all documented effects. What is missing is a tool: something that runs over the traces you already have, costs four model calls per failure, and reports its own error rate. That is what this is.

License

MIT. See CONTRIBUTING.md to get started.