counterfix

Find why an LLM call failed, and get a fix that was tested to work.

Most LLM evaluation tools tell you that a model failed. counterfix is built to tell you why, and to hand back a fix it has already checked. When a call fails, it changes one thing at a time (raises max_tokens, asks for bare JSON, adds the source text, tells the model to check the question's premises, repeats a missed instruction) and re-runs the call. A change counts only if it works in 2 of 3 re-runs. The change that fixes the failure is both the diagnosis and the prescription. We call this counterfactual diagnosis.

Status: alpha (v0.1.0). Everything below works today: live checks, judge checks, verified fixes, reports, regression tests, profiles. APIs may still change.

Known limits. Verified fixes are common for cut-off answers and broken JSON, rare for invented facts (3 of 14 on our seed set) and none yet for over-refusals or false premises: there counterfix says so and lists what to try instead. Shipped model profiles are small (54 probes; qwen2.5:3b only partly measured). The labels behind our accuracy numbers are mostly reference labels, with a small human spot check still to do. See docs/results.md.

Install

pip install "counterfix[openai]"

The core needs no API key. The openai extra is only for wrapping OpenAI-compatible clients (OpenAI, Groq, Gemini, OpenRouter, Ollama, vLLM, ...).

Quickstart

1. See it work in 10 seconds (no key)

import counterfix
from counterfix import CallRecord, Message

record = CallRecord(
    messages=[Message(role="user", content="Answer in exactly one word: sky colour?")],
    response_text="The sky is usually blue on a clear day.",
)
print(counterfix.diagnose(record, judge=False).explain())

F6 Dropped instruction (90%)
  why: instructions not met: exactly 1 word (has 9); a one-word answer (has 9 words)
  fix: Repeat the broken instructions at the end of the prompt as a short checklist. ...

judge=False runs only the live checks: free, instant, no extra calls. More cases in examples/quickstart.py.

2. Wrap your client (one line)

from openai import OpenAI
import counterfix

client = counterfix.guard(OpenAI())  # same API as before
response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Why did Einstein win his third Nobel Prize?"}],
)
print(response.choices[0].message.content)
print(response.diagnosis.summary())  # live checks, already run on this call

Any OpenAI-compatible endpoint works, for example a local Ollama server:

client = counterfix.guard(OpenAI(base_url="http://localhost:11434/v1", api_key="ollama"))

The Anthropic SDK works the same way (messages.create, sync or async; tool_use arguments are checked against each tool's input_schema):

from anthropic import Anthropic

client = counterfix.guard(Anthropic())

Already on LiteLLM or LangChain? Add a callback; it only observes, never changes a reply:

litellm.callbacks = [counterfix.litellm_callback()]
llm.invoke("...", config={"callbacks": [counterfix.langchain_callback()]})

Not using any of these? Decorate the function that calls your model (sync or async). It takes a prompt string or chat messages and returns text or a response:

@counterfix.watch(model="llama3.2")
def ask(prompt: str) -> str:
    return my_local_model(prompt)


answer = ask("Answer in one word: sky colour?")  # returned unchanged
print(ask.last_diagnosis.summary())

3. Ask a judge for the deeper look

report = counterfix.diagnose(response)  # adds judge checks: false premise, invented facts, ...
print(report.explain())

The judge comes from your own key (a free GROQ_API_KEY or GEMINI_API_KEY is enough; see Bring your own key). Run counterfix setup to see what it found. Full script: examples/diagnose_with_judge.py.

4. Get a fix that was tested to work

result = counterfix.fix(response)  # a guarded response, its diagnosis, or a CallRecord
print(result.explain())

found: F1 Truncation (95%)
plain retry: passed 0/2
tried raise_max_tokens: passed 0/2 -> no (still: F1 Truncation)
tried raise_max_tokens_more: passed 2/3 -> VERIFIED
fix: Raise max_tokens a lot (long answers, or reasoning that uses the budget)
  change: {'max_tokens': 4096}
7 re-run(s); stopped at the first verified fix

For each failure, counterfix changes one thing (raise max_tokens, ask for bare JSON, check the question's premises, repeat missed instructions, add the source text, ...), re-runs the call, and re-checks it. A change counts as the fix only if the failure is gone, with no new failure, in 2 of 3 re-runs, so luck does not count. It first re-runs the call unchanged: if a plain retry already passes, no change is claimed. The verified change is in result.fix (changed params, or full new messages) and on the finding's prescription (verified=True). max_runs caps the re-runs (15); judge calls come on top. When nothing passes, result.escalation lists what is left (retrieval, a larger model, a code tool, a clarifying question).

5. Log calls, then report on them

client = counterfix.guard(
    OpenAI(), config=counterfix.Config(on_diagnosis=counterfix.log_to("calls.jsonl"))
)

counterfix diagnose calls.jsonl                      # terminal summary, live checks only (free)
counterfix diagnose calls.jsonl --judge --html report.html --json results.jsonl
counterfix diagnose calls.jsonl --fail-on-failure    # exit code 1 if any call failed (CI)

The HTML report is one self-contained file you can send to a teammate: totals per failure type, then each call with its prompt, answer, why it was flagged and the suggested fix. The JSON lines ({"record": ..., "diagnosis": ...}) suit log pipelines, and can be read back with counterfix diagnose or counterfix.load_log().

6. Judge checks on production traffic, without slowing it down

config = counterfix.Config(
    sample_rate=0.05,  # every flagged call + 5% of the rest get judge checks
    max_judged_per_hour=60,  # cost cap; calls over it are skipped, never delayed
    on_judged=counterfix.log_to("judged.jsonl"),
)
client = counterfix.guard(OpenAI(), config=config)

Judge checks run on one background thread, so your call returns as fast as before. If the queue fills up, new calls are skipped rather than waited on; counterfix.background.stats counts what was judged and skipped. Call counterfix.background.flush() before a short script exits if you want the queued checks to finish.

7. Keep fixed failures fixed (pytest)

counterfix fix calls.jsonl --judge --out fixes.jsonl
counterfix export-tests fixes.jsonl --out tests/test_prompt_regressions.py
pytest tests/test_prompt_regressions.py

Each verified fix becomes a test case that fails if the failure comes back after a prompt edit or a model change. In your own tests, counterfix.testing.assert_ok(response) fails with the explanation when counterfix finds a failure.

More

Streams (stream=True)	every chunk passes straight through; the whole answer is checked at the end
with counterfix.context(source=..., reference=...)	tell the checks what the model was given (RAG) or the right answer
counterfix profile / compare / drift / route	measure models, compare an upgrade on your own calls, alert on drift, pick the cheapest safe model per task
counterfix.otel.export(), counterfix.otel.SpanChecker()	send diagnoses to OpenTelemetry, or check the GenAI spans you already have
counterfix mcp	an MCP server so coding agents can lint prompts and check calls

Full docs: docs/ (build with mkdocs serve). Measured accuracy and fix rates: docs/results.md. A Colab-ready notebook on a free model: examples/quickstart.ipynb.

What it checks

Live checks make no extra LLM calls. They catch truncation (including reasoning models that spend the whole budget thinking), broken JSON or schema mismatches, broken tool-call arguments, refusals, empty answers, provider content filters, and broken instructions that code can verify exactly: sentence, word and bullet counts, letter case, reply language (by script), British or American spelling, banned words, "end with a question", "only the number", "one word" and similar (in the spirit of IFEval).

Optional auto-retry: a fix that was re-run

from counterfix import Config, guard

client = guard(OpenAI(base_url=GROQ_URL, api_key=key), config=Config(mode="retry"))
r = client.chat.completions.create(
    model="qwen/qwen3.8-27b",
    messages=[{"role": "user", "content": "List the planets of the solar system, one per line."}],
    max_tokens=20,
)
for f in r.diagnosis.retried_from:
    print("before:", f.failure_type.value, f.evidence[0].detail)
for attempt in r.diagnosis.retries:
    print("retry: ", attempt.changes, attempt.outcome)
print("after: ", r.diagnosis.summary())

before: F1 the answer stopped at the max_tokens limit (20)
retry:  {'max_tokens': 80} fixed
after:  no failure found by 5 checks

You get back the fixed response (all eight planets), and the record of what changed. Retry keeps whichever response is better, records what it tried, and never raises if the retry fails. Add your own fixes per model with RuleStore and FixRule.

Deeper checks with a judge (diagnose mode)

Some failures need meaning to spot: invented facts, false premises, content instructions ("mention the price"), ambiguity, wrong arithmetic. These checks ask a judge model, so they cost extra calls and never run in live mode; counterfix.diagnose() adds them when you want the deeper look (on failures, on a sample of traffic, or in tests). diagnose() takes a response from a guarded client or a CallRecord.

One mistake gets one cause: a wrong calculation is reported as F8, not also as an invented fact (F4), and a cut-off answer as F1, not also as dropped instructions (F6). The findings that were set aside are kept in report.suppressed.

The F8 check never trusts the judge's arithmetic: the judge writes one expression and counterfix evaluates it with a small whitelist-only evaluator. A sampling check (SamplingCheck) re-asks the model and flags answers that change between samples. An optional Jev back-end (counterfix[jev]) plugs into the same judge interface.

Bring your own key

counterfix never ships API keys. Live checks need none: they wrap your own client and make no calls of their own. Judge checks use a judge you choose, in this order:

1. counterfix.diagnose(response, judge="groq:qwen/qwen3.8-27b")
2. counterfix.configure(judge=...), once per process. A list of specs makes a voting panel, e.g. ["groq:qwen/qwen3.8-27b", "gemini:gemini-3.5-flash-lite", "ollama:llama3.2"]
3. the COUNTERFIX_JUDGE environment variable (comma-separated for a panel)
4. a key it finds: GROQ_API_KEY or GEMINI_API_KEY (both have free tiers), OPENAI_API_KEY, OPENROUTER_API_KEY, or a model in a local Ollama. A different model family from the one being checked is preferred.
5. otherwise the checked model judges itself, with a SelfJudgeWarning, because judges are lenient with their own mistakes.

See what counterfix finds on your machine (key values are never printed):

counterfix setup          # keys, local models, and the judge that will be used
counterfix setup --test   # also asks the judge one tiny question

Lint prompts before you send them

from counterfix import lint

for issue in lint("Return ONLY JSON with keys name and age for: Priya, 29."):
    print(issue.rule, issue.title, "->", issue.suggestion)

Or from the command line, e.g. in CI:

counterfix lint prompts/*.txt --fail-on-warning

Rule	Flags	Tends to cause
L1	A constraint buried in the middle of a long prompt	F6
L3	A factual question with no "I don't know" option	F4, F5
L4	JSON/YAML/CSV asked for in words only, no example or JSON mode	F2
L7	Pasted outside text with no delimiters	prompt injection
L2	Instructions that conflict (needs --judge)	F6, F7
L5	System prompt and user message disagree (needs --judge)	F6, F3
L6	Request missing a key detail (needs --judge)	F9

Design rules:

• Fail open. If counterfix itself errors, your call still returns normally.
• Your errors stay yours. Exceptions from the provider reach your code unchanged.
• Safe defaults. Live mode only flags failures and makes no extra LLM calls.
• Light core. The core depends on 4 small libraries; providers and heavy tools are extras.

Failure types (v1)

Code	Failure
F1	Truncation
F2	Format break (JSON, schema)
F3	Over-refusal
F4	Hallucination from missing knowledge
F5	False-premise acceptance
F6	Dropped instruction
F7	Prompt fragility
F8	Computation or token-level error
F9	Ambiguous request

API errors, rate limits and provider content filters are tracked separately, so they are never counted as the model's fault.

Development

git clone https://github.com/KANSADWALA/CounterFix.git
cd CounterFix
pip install -e ".[dev]"
pytest

Optional extras: openai, anthropic, litellm, langchain, llamaindex, otel, semantic, cache, cli, report, jev.

Roadmap

Done:

• guard() for any OpenAI-compatible client, live checks, optional auto-retry ✅
• Prompt linter (counterfix lint) ✅
• Judge checks, diagnose(), bring your own key, counterfix setup ✅
• Reports: log_to(), counterfix diagnose LOG with terminal, JSON and HTML output ✅
• Background judge checks on a sample of traffic, with a cost cap ✅
• @counterfix.watch decorator for any function, sync or async ✅
• Anthropic SDK in guard(), LiteLLM and LangChain callbacks ✅
• Verified fixes: counterfix.fix() (counterfactual re-runs, 2 of 3 must pass) ✅
• Prompt regression tests: counterfix export-tests and a pytest plugin ✅
• Model profiles, compare, drift, route; fixes ordered by what worked per model ✅
• OpenTelemetry export and GenAI span reader; streams; RAG context; MCP server ✅

Next:

1. First release on PyPI (v0.1.0)
2. A larger probe suite from public datasets (after a licence check per dataset)
3. A public dataset of diagnosed failures with their verified fixes

License

MIT