counterfix

Find out why an LLM call failed, and prove the fix works.

One line around the client you already use. counterfix names the likely failure type (F1 to F14) on every call. Ask it for a fix and it changes one thing at a time (raises max_tokens, asks for bare JSON, adds the source text, tells the model to check the question's premises, repeats a missed instruction) and re-runs the call. A change counts only if it passes clearly more often than re-running the call unchanged (a Fisher exact test against a plain retry). When none does, counterfix says so and lists what to try next. We call this counterfactual diagnosis.

counterfix works alongside your evaluation and tracing tools (DeepEval, Ragas, Promptfoo, Langfuse, MLflow, ...): they tell you that calls fail and how often; counterfix tells you why one failed and which change verifiably fixes it.

Status: alpha (latest version in the PyPI badge above). Everything below works today: live checks, judge checks, verified fixes, reports, regression tests, profiles, dataset and RAG evaluation, code evaluation, benchmarks, prompt-injection and personal-data checks, token confidence, and agent and tool-call checks. APIs may still change.

Known limits. Verified fixes are common for cut-off answers and broken JSON, rare for invented facts (3 of 14 on our seed set) and none yet for over-refusals or false premises: there counterfix says so and lists what to try instead. For code that gives wrong results, a small model (qwen2.5:3b) got a verified fix on 2 of 8 repeatable failures. For prompt injection, one fix (delimit_untrusted) was verified on 9 of 10 injected probes for gpt-oss-120b, none for qwen2.5:3b. The 20-probe tools benchmark is a smoke test (qwen2.5:3b passes all 20), not a model comparison. Shipped model profiles are small (54 probes; qwen2.5:3b only partly measured). The labels behind our accuracy numbers are reference labels; a human spot check agreed with 33 of 35 of them. See docs/results.md.

Install

pip install counterfix

The core has 4 small dependencies and needs no API key. counterfix is an independent open-source project, not affiliated with any model provider. If your project already has the OpenAI or Anthropic SDK, nothing else is needed; otherwise add it with pip install "counterfix[openai]" or "counterfix[anthropic]".

Quickstart

import counterfix
from openai import OpenAI

client = counterfix.guard(OpenAI())  # 1. wrap your client (same API as before)
r = client.chat.completions.create(model="gpt-4o-mini", messages=[...])
print(r.diagnosis.summary())  # 2. see why it failed (live checks, free)
print(counterfix.fix(r).explain())  # 3. get a fix that was re-run and verified

found: F1 Truncation (95%)
plain retry: passed 0/6
tried raise_max_tokens: passed 0/2 vs retry 0/6, p=1.000 -> no (still: F1 Truncation)
tried raise_max_tokens_more: passed 5/5 vs retry 0/6, p=0.002 -> VERIFIED
fix: Raise max_tokens a lot (long answers, or reasoning that uses the budget)
  change: {'max_tokens': 4096}
13 re-run(s); stopped at the first verified fix

That is the whole loop. Step 2 costs nothing: live checks make no extra calls. Step 3 re-runs the call (here 13 times) and may use a judge model for checks code cannot do; the judge comes from your own key (a free GROQ_API_KEY or GEMINI_API_KEY is enough; run counterfix setup to see what it finds). "Verified" means the change beat asking again on this call, not that it works on every prompt. How the test works: Verified fixes.

Try it with no key at all:

from counterfix import CallRecord, Message

record = CallRecord(
    messages=[Message(role="user", content="Answer in exactly one word: sky colour?")],
    response_text="The sky is usually blue on a clear day.",
)
print(counterfix.diagnose(record, judge=False).explain())

F6 Dropped instruction (90%)
  why: instructions not met: exactly 1 word (has 9); a one-word answer (has 9 words)
  try: Repeat the broken instructions at the end of the prompt as a short checklist. ...

Plug it into what you use

The guarded client behaves like the original: sync or async, stream=True, chat.completions.parse(), the Responses API, with blocks and with_options() all work and stay checked, and your type checker sees the client you passed. Two things differ: isinstance(client, OpenAI) is False (pass client.unwrapped where a library insists on the real object), and type checkers do not know response.diagnosis, so typed code can use counterfix.integrations.get_diagnosis(response). Callbacks only observe; they never change a reply. Details: Plug it in.

In production

client = counterfix.guard(
    OpenAI(),
    log="calls.jsonl",  # every call and its diagnosis, as JSON lines
    sample_rate=0.05,  # judge checks on every flagged call + 5% of the rest
    mode="retry",  # re-run a flagged call once with a known fix
)

Judge checks run on a background thread, capped per hour, so calls return as fast as before. Retry mode is off by default; without it counterfix only reports. For anything else, pass a counterfix.Config (the shortcuts above override it).

counterfix diagnose calls.jsonl --judge --html report.html   # a report you can send a teammate
counterfix fix calls.jsonl --judge --out fixes.jsonl          # search verified fixes in bulk
counterfix export-tests fixes.jsonl --out tests/test_prompt_regressions.py

Each verified fix becomes a pytest case that fails if the failure comes back after a prompt edit or a model change; counterfix.testing.assert_ok(response) does the same in your own tests. Guides: In production, Prompt regression tests.

The same checks over a dataset

The checks and fixes above also run over a whole dataset: the misses grouped by cause, and a verified fix searched for each. Install with pip install "counterfix[evals]".

counterfix eval qa.jsonl --model groq:openai/gpt-oss-120b --metric exact_match --judge --fix 5

counterfix works alongside your evaluation tools rather than replacing them. Metrics and benchmarks those tools already provide (BLEU, ROUGE, BERTScore, judge-scored RAG metrics, PII and toxicity, MMLU, GSM8K, HumanEval and the like) still work in 0.3 but are deprecated, with a warning that names the tool to use instead, and are planned for removal in 0.4.

More

Full docs: https://kansadwala.github.io/CounterFix/ (or build locally with mkdocs serve). Measured accuracy and fix rates: docs/results.md. A Colab-ready notebook on a free model: examples/quickstart.ipynb.

What it checks

Live checks make no extra LLM calls. They catch truncation (including reasoning models that spend the whole budget thinking), broken JSON or schema mismatches, broken tool-call arguments, refusals, empty answers, provider content filters, and broken instructions that code can verify exactly: sentence, word and bullet counts, letter case, reply language (by script), British or American spelling, banned words, "end with a question", "only the number", "one word" and similar (in the spirit of IFEval). On agent runs they also catch calls to tools that were not offered, repeated identical calls and missing expected calls; with a canary planted in untrusted text, they catch a followed injection for free.

Optional auto-retry: a fix that was re-run

from counterfix import Config, guard

client = guard(OpenAI(base_url=GROQ_URL, api_key=key), config=Config(mode="retry"))
r = client.chat.completions.create(
    model="qwen/qwen3.8-27b",
    messages=[{"role": "user", "content": "List the planets of the solar system, one per line."}],
    max_tokens=20,
)
for f in r.diagnosis.retried_from:
    print("before:", f.failure_type.value, f.evidence[0].detail)
for attempt in r.diagnosis.retries:
    print("retry: ", attempt.changes, attempt.outcome)
print("after: ", r.diagnosis.summary())

before: F1 the answer stopped at the max_tokens limit (20)
retry:  {'max_tokens': 1024} fixed
after:  no failure found by 5 checks

You get back the fixed response (all eight planets), and the record of what changed. Retry keeps whichever response is better, records what it tried, and never raises if the retry fails. Add your own fixes per model with RuleStore and FixRule.

Deeper checks with a judge (diagnose mode)

Some failures need meaning to spot: invented facts, false premises, content instructions ("mention the price"), ambiguity, wrong arithmetic. These checks ask a judge model, so they cost extra calls and never run in live mode; counterfix.diagnose() adds them when you want the deeper look (on failures, on a sample of traffic, or in tests). diagnose() takes a response from a guarded client or a CallRecord.

One mistake gets one cause: a wrong calculation is reported as F8, not also as an invented fact (F4), and a cut-off answer as F1, not also as dropped instructions (F6). The findings that were set aside are kept in report.suppressed.

The F8 check never trusts the judge's arithmetic: the judge writes one expression and counterfix evaluates it with a small whitelist-only evaluator. A sampling check (SamplingCheck) re-asks the model and flags answers that change between samples. An optional Jev back-end (counterfix[jev]) plugs into the same judge interface.

Bring your own key

counterfix never ships API keys. Live checks need none: they wrap your own client and make no calls of their own. Judge checks use a judge you choose, in this order:

1. counterfix.diagnose(response, judge="groq:qwen/qwen3.8-27b")
2. counterfix.configure(judge=...), once per process. A list of specs makes a voting panel, e.g. ["groq:qwen/qwen3.8-27b", "gemini:gemini-3.5-flash-lite", "ollama:llama3.2"]
3. the COUNTERFIX_JUDGE environment variable (comma-separated for a panel)
4. a key it finds: GROQ_API_KEY or GEMINI_API_KEY (both have free tiers), OPENAI_API_KEY, OPENROUTER_API_KEY, or a model in a local Ollama. A different model family from the one being checked is preferred.
5. otherwise the checked model judges itself, with a SelfJudgeWarning, because judges are lenient with their own mistakes.

See what counterfix finds on your machine (key values are never printed):

counterfix setup          # keys, local models, and the judge that will be used
counterfix setup --test   # also asks the judge one tiny question

Lint prompts before you send them

from counterfix import lint

for issue in lint("Return ONLY JSON with keys name and age for: Priya, 29."):
    print(issue.rule, issue.title, "->", issue.suggestion)

Or from the command line, e.g. in CI:

counterfix lint prompts/*.txt --fail-on-warning

Design rules:

• Fail open. If counterfix itself errors, your call still returns normally.
• Your errors stay yours. Exceptions from the provider reach your code unchanged.
• Safe defaults. Live mode only flags failures and makes no extra LLM calls.
• Light core. The core depends on 4 small libraries; providers and heavy tools are extras.

Failure types

API errors, rate limits and provider content filters are tracked separately, so they are never counted as the model's fault.

Development

git clone https://github.com/KANSADWALA/CounterFix.git
cd CounterFix
pip install -e ".[dev]"
pytest

Optional extras: openai, anthropic, litellm, langchain, llamaindex, otel, semantic, metrics, eval, code, benchmarks, safety, mcp, cache, cli, report, jev.

python bench/dev_walkthrough.py SPEC uses the library as a developer would against a real model (about 40 small calls) and prints OK or the error for each step.

Roadmap

Done:

• guard() for any OpenAI-compatible client, live checks, optional auto-retry ✅
• Prompt linter (counterfix lint) ✅
• Judge checks, diagnose(), bring your own key, counterfix setup ✅
• Reports: log_to(), counterfix diagnose LOG with terminal, JSON and HTML output ✅
• Background judge checks on a sample of traffic, with a cost cap ✅
• @counterfix.watch decorator for any function, sync or async ✅
• Anthropic SDK in guard(), LiteLLM and LangChain callbacks ✅
• Verified fixes: counterfix.fix() (counterfactual re-runs; a change must beat a plain retry) ✅
• Prompt regression tests: counterfix export-tests and a pytest plugin ✅
• Model profiles, compare, drift, route; fixes ordered by what worked per model ✅
• OpenTelemetry export and GenAI span reader; streams; RAG context; MCP server ✅
• Released on PyPI (pip install counterfix), docs site, releases by trusted publishing ✅
• Evaluation, wave 1: counterfix eval, reference metrics, TTFT and speed/cost, verified speed fixes ✅
• RAG evaluation: retrieval and judge metrics, F10 retrieval miss vs F11 context ignored, verified RAG fixes ✅
• OpenAI Responses API in guard(); evaluate your own app or RAG pipeline (--app) ✅
• Code evaluation: sandboxed tests, F12 with its causes, verified code fixes, pass@k ✅
• Benchmarks: MMLU, GSM8K, ARC, TruthfulQA, IFEval, HumanEval(+), MBPP(+), checked against lm-eval ✅
• Safety: prompt injection (F13, probes and judge check, verified fixes), PII, toxicity; token confidence and perplexity ✅
• Agents and tool calls: trajectories from any framework, F14 checks, tool_call metric, verified tool-choice fixes ✅
• Trust in the metrics: a labelled set and measured accuracy for every judge-based check, and a human spot check of the labels ✅

Next:

1. A public dataset of diagnosed failures with their verified fixes
2. Larger labelled sets and full profiles for more models
3. A BFCL loader, so the tool-use benchmark can compare models

License

MIT