wiretruth

wiretruth replays real LLM provider responses through LLM SDKs and checks whether each SDK reports what actually happened.

Does your LLM SDK tell you the truth?

Version 0.1 tests one library, any-llm, on the OpenAI Chat Completions API with three truncation scenarios. More providers and libraries follow, and a public results matrix arrives in v0.4. Until then, the grades are in results/.

Why

A model hits its output token limit halfway through a JSON object. The provider says so plainly: "finish_reason": "length". What your code sees depends on the SDK in between. If the SDK raises a generic JSON parse error, nothing tells you that a higher limit would fix it. If it returns the partial object as a success, your pipeline stores an incomplete record.

wiretruth has this case recorded as truncation.structured-output on openai-chat. Replayed through any-llm:

wiretruth runs the same recorded edge cases through every library it supports and grades each one against what the provider sent.

What it checks

The scenarios are in suite/scenarios/, and the bug classes in suite/bug-classes.json.

Grades

Each scenario lists its requirements. MUST requirements set the grade. SHOULD requirements, such as the exact partial text in the truncation scenarios, add a note when they aren't met. How grades are computed: docs/grading.md.

Quickstart

You need Python 3.11 or newer and uv, which builds each library's own environment.

git clone https://github.com/kartsan03/wiretruth && cd wiretruth
uv run wiretruth run --lib any-llm

scenario                      openai-chat
truncation.structured-output  ✓ PASS
truncation.text               ✓ PASS
truncation.text-stream        ✓ PASS

any-llm 1.33.0: 3 cells, 3 PASS
stderr: .wiretruth/runs/20261010T193444.312067Z

No API keys are needed. Everything runs against recorded responses on 127.0.0.1. The first run downloads any-llm into .wiretruth/envs/; later runs work offline. uv run wiretruth run --shim shims/reference runs the built-in reference client, which must pass every cell. The scenarios, fixtures and shims live in this repository, so wiretruth runs from a clone.

Add your library

A shim is a small program. It reads a call spec as JSON on stdin, calls your library the way its docs show, and prints what the library reported as JSON on stdout. The any-llm shim is 150 lines. The protocol is in shims/README.md; to propose a library, open a new-library issue.

Related work

• Open Responses: a spec and compliance tests for servers that implement the Responses API. wiretruth tests clients across several native APIs.
• LLMConform: conformance tests for LLM gateways and APIs, run live.
• llmprobe and conform: conformance for inference engines (vLLM, llama.cpp, Ollama, …).
• aimock: a mock server for testing AI apps, with record and replay. Use it for your app's tests; wiretruth audits the SDK layer itself.
• The design borrows from JSONTestSuite, Wycheproof, toml-test and compat-table.

Author note

I've contributed fixes to libraries that wiretruth tests or will test: mozilla-ai/any-llm#1389 and TanStack/ai#1548. Grades are computed by the code in this repository. If you think a grade is wrong, please open a grade dispute.

Contributing

Scenario ideas, new libraries and grade disputes are all welcome. See CONTRIBUTING.md.

Citing

See CITATION.cff.

License

MIT. Fixtures contain model outputs recorded from provider APIs; docs/recording.md explains how they're recorded.