counterfix

Find why an LLM call failed, and get a fix that was tested to work.

Most LLM evaluation tools tell you that a model failed. counterfix is built to tell you why, and to hand back a fix it has already checked. When a call fails, it changes one thing at a time (adds context, removes a false premise, rewords the prompt, gives the model a tool, raises max_tokens) and re-runs the call. The change that fixes the failure is both the diagnosis and the prescription. We call this counterfactual diagnosis.

Status: pre-alpha (v0.0.1). The core records, the guard() wrapper and the seed failure set are in place. Detectors and the counterfactual engine are being built. APIs will change.

What it plugs into

counterfix is middleware: it sits between your code and any text-in, text-out LLM.

from openai import OpenAI
from counterfix import guard

client = guard(OpenAI())  # same API as before
response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Why did Einstein win his third Nobel Prize?"}],
)
print(response.choices[0].message.content)
print(response.diagnosis.summary())  # what counterfix found

The same wrapper works with any OpenAI-compatible endpoint, for example Groq or a local Ollama server:

client = guard(OpenAI(base_url="http://localhost:11434/v1", api_key="ollama"))

Design rules:

• Fail open. If counterfix itself errors, your call still returns normally.
• Your errors stay yours. Exceptions from the provider reach your code unchanged.
• Safe defaults. Live mode only flags failures and makes no extra LLM calls.
• Light core. The core depends on 4 small libraries; providers and heavy tools are extras.

Failure types (v1)

API errors, rate limits and provider content filters are tracked separately, so they are never counted as the model's fault.

Install (development)

git clone https://github.com/KANSADWALA/CounterFix.git
cd CounterFix
pip install -e ".[dev]"
pytest

Optional extras: openai, anthropic, litellm, langchain, llamaindex, otel, semantic, cache, cli, report, jev.

Roadmap

1. Foundations and seed failure set ✅
2. Core records and the guard() plug ✅
3. Cheap live detectors (F1 to F3) and the prompt linter
4. LLM-judge and sampling detectors
5. Diagnose mode and HTML report
6. Counterfactual engine with verified fixes
7. Model failure profiles
8. More plugs: decorator, Anthropic, LiteLLM, LangChain, OpenTelemetry
9. Evaluation on public benchmarks
10. Release and paper

License

MIT