Skip to main content

infereval

CI Docs Release PyPI Python License: MIT

Inferentialist evaluation of LLMs: derive an implication frame from a language model's endorsement verdicts, then measure the model's agreement with an analyst-labeled benchmark via coverage and Cohen's / Fleiss' kappa. The agreement is evidence bearing on an inferential-mastery attribution — not a measurement of mastery itself (see Concepts).

infereval began as a way to turn Simonelli's stop-sign dialogue with GPT-4.1 into a repeatable evaluation procedure. It implements that procedure, β → η → (cov, κ_C, κ_F, κ_F*), for any analyst-supplied benchmark. β is the analyst-labeled benchmark, η is the model's derived endorsements, and the rest are the agreement measures described below.

Status

Beta (0.x, pre-1.0). The public Python API and CLI surface may shift between minor releases until 1.0. Methodology defaults are locked, and the JSON schemas are versioned independently (schema_version: "1.0") and promised stable from 1.0 onward regardless of the framework version. See the CHANGELOG for the current release.

Documentation

The docs site at https://www.bradleypallen.org/infereval/ covers a Concepts page (methodology mental model), Authoring benchmarks, Interpreting metrics (κ_C / κ_F / κ_F*, test-retest κ, decompositions, subsampling CIs, sensitivity sweeps), Providers (Anthropic seed handling, DeepSeek reasoning-token budgets, OpenRouter attribution), and Construct validity of the instrument — the R1–R22 requirements catalogue and end-to-end workflow in one place. Four executable tutorial notebooks (quickstart, authoring, paraphrase-axis triangulation, pulmonology visualization). Plus an auto-generated API reference, an Architecture dataflow diagram, a Glossary of the notation, and a JSON-schema reference.

Findings

A 13-model × 3-paraphrase-variant cross-family sweep against the stop-sign benchmark (2026-06-09) is committed at experiments/results/stop_sign_2026-06-09.md. Headline: under the original wording of δ(ra) ("a is red"), 12 of 13 frontier LLMs across six families reproduce Simonelli's analyst row exactly (κ_C = +1.000). That replicates the verdicts of Simonelli's dialogue across thirteen models, in fresh v0.15.2 captures. The perceptual variant is the cleavage axis. A companion 6-model pulmonary edema sweep (n=30 items) at experiments/results/pulmonology_2026-06-09.md characterizes within-day and day-out R22 reliability; the deepseek-v4-pro cell exhibits monotone κ decay across three time-scales (0.867 → 0.792 → 0.729) — the clearest published example of detectable across-update model drift the framework has produced.

Install

pip install infereval

Provider SDKs are optional extras (the framework runs without them — use the mock or replay providers):

pip install 'infereval[anthropic]'   # Anthropic Claude
pip install 'infereval[openai]'      # OpenAI + OpenRouter (OpenAI-API-compatible)
pip install 'infereval[all]'

From source (editable):

git clone https://github.com/bradleypallen/infereval
cd infereval
python -m venv .venv
source .venv/bin/activate
pip install -e '.[dev]'

60-second quickstart

Inspect the bundled stop-sign benchmark (a formalization of Simonelli's stop-sign dialogue), then run an evaluation against the deterministic replay fixture — no API key needed:

# 1. Look at the benchmark.
infereval describe examples/stop_sign/benchmark.json

# 2. Validate it against the JSON schema.
infereval validate examples/stop_sign/benchmark.json

# 3. Run a deterministic evaluation against the committed replay fixture.
infereval evaluate examples/stop_sign/benchmark.json \
    --replay-from tests/fixtures/stop_sign_replay.jsonl \
    --output /tmp/eta.json \
    --n-samples 5 \
    --log /tmp/run.jsonl

# 4. Compute metrics.
infereval metrics /tmp/eta.json --benchmark examples/stop_sign/benchmark.json

To run against a real model, replace step 3 with:

export ANTHROPIC_API_KEY=...
infereval evaluate examples/stop_sign/benchmark.json \
    --provider anthropic --model claude-haiku-4-5-20251001 \
    --output /tmp/eta.json --n-samples 5 --log /tmp/run.jsonl

The JSONL run log under /tmp/run.jsonl records one event per provider call (prompt hash, raw response, parsed verdict, usage, timing) so the evaluation is auditable end to end.

For the R22 test-retest discipline, infereval retest --auto --benchmark <b> --provider X --model Y [--interval-s N] [--save-etas DIR] (v0.11.0+) collapses the four-step manual workflow (evaluate, evaluate again, retest, optionally --claims) into one CLI call — runs the same evaluation twice with an optional inter-capture sleep and emits the standard RetestResult directly. v0.12.0+ accepts --interval-s multiple times for cumulative drift-since-baseline analysis in one orchestrated call. v0.13.0+ surfaces test-retest κ as a co-equal ### Reliability (R22) subhead in infereval report §2 alongside ### Agreement, and auto-detects single vs multi-interval retest shape (multi-interval renders a per-interval table with a worst-case overall verdict). v0.14.0+ adds the staged-composition pattern: --baseline-from <eta-path> runs one fresh capture against a saved baseline and --append-to <multi.json> appends a new pair to an existing multi-result — both compute interval_s from actual elapsed wall clock, so day-out / week-out R22 evidence can ship as incremental commits without the CLI process needing to stay alive for the elapsed window. See the stop-sign R22 capture for the worked end-to-end demo.

What this is and isn't

This is: a research tool that formalizes Simonelli's stop-sign dialogue into a repeatable evaluation procedure. Given (i) a bearer set, (ii) expression and context-construction functions, (iii) a benchmark of implications labeled by one or more analysts, the framework drives an LLM through endorsement-probing for each implication and reports the resulting agreement with analyst practice along three axes:

  • Coverage — how often the model takes a substantive position (cov(η)).
  • Cohen's kappa — agreement against a chosen reference (analyst consensus c_i or a single analyst v_{:,j}).
  • Fleiss' kappa — agreement with the model treated as the (m+1)th annotator, alongside the inter-analyst baseline κ_F*(β). The analysts' agreement with one another is the ceiling against which the model's agreement should be read.

Each metric can be decomposed by tag or by RSR target.

This is not: a factuality benchmark, a leaderboard, or an answer to whether LLMs are sapient. The methodology is carving-relative: results depend on the analyst-supplied bearer carving, context construction, and benchmark. The framework provides the machinery; the analyst supplies the practice the machinery is comparing against. Concepts discusses what carving-relativity buys and costs.

API surface

from infereval import (
    Verdict, Bearer, Implication,           # core data types
    DerivedFrame,                            # ⟨B, I_M⟩: bearers + the model's derived implications
)
from infereval.benchmark import Benchmark
from infereval.evaluation import Evaluation, evaluate, EndorsementConfig, ProviderParams
from infereval.providers import get_provider
from infereval.metrics import MetricsReport

bench = Benchmark.load("examples/stop_sign/benchmark.json")
provider = get_provider("anthropic", "claude-haiku-4-5-20251001")
eta = evaluate(bench, provider,
               config=EndorsementConfig(n_samples=5),
               params=ProviderParams(temperature=1.0),
               log_path="/tmp/run.jsonl")
report = MetricsReport(eta=eta, benchmark=bench)
print(report.to_dict())

Locked methodology defaults

These are framework defaults, overridable per evaluation:

Setting Default
n_samples 5 (odd, clean 3-way majority)
Tie-break abstain (configurable: good, bad, first)
Verification prompt default-v1 (GOOD/BAD/ABSTAIN tokens with brief glosses)
TeX in expressions Stripped at prompt time; LaTeX-source-friendly in benchmark JSON
Cohen's kappa reference Analyst consensus c_i (override with --reference analyst:<id>)
Provider seed Honored by OpenAI; ignored (with one-time warning) by Anthropic

See CLAUDE.md for the full list and the rationale behind each choice.

Development

pip install -e '.[dev]'
pytest                                # all unit + replay tests
pytest -m live                        # opt-in live provider tests (requires API keys)
mypy src/infereval
ruff check src tests

Live provider tests require RUN_LIVE_PROVIDER_TESTS=1 and the relevant API key in the environment. They are skipped by default.

Citation

If you use infereval in your work, please cite the software, giving the version you used:

@software{allen_infereval,
  author  = {Allen, Bradley P.},
  title   = {infereval: Inferentialist evaluation of {LLMs}},
  year    = {2026},
  version = {0.17.8},
  url     = {https://github.com/bradleypallen/infereval}
}

License

MIT — see LICENSE.

Metadata

Release files for infereval 0.17.9

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for infereval 0.17.9
File Size Uploaded Provenance Actions
infereval-0.17.9.tar.gz 454.0 kB With provenance Details for infereval-0.17.9.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for infereval 0.17.9
File Interpreter ABI Platform Actions
infereval-0.17.9-py3-none-any.whl Python 3 none any Details for infereval-0.17.9-py3-none-any.whl

Total release size: 694.5 kB

Release files / infereval-0.17.9.tar.gz

Download URL infereval-0.17.9.tar.gz
Size 454.0 kB
Tags Source
SHA-256 checksum
How to use checksums
64102743bbe3cc2991932f93d2f53e8fe7a740782d1400931fefb4c646c40efd
BLAKE2b-256 checksum
How to use checksums
5fefba8e73b6eb411a0b029703f0e108a4d2714ac473d0be774dfee0402e7029
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 12, 2026.

Transparency log

Release files / infereval-0.17.9-py3-none-any.whl

Download URL infereval-0.17.9-py3-none-any.whl
Size 240.5 kB
Tags Python 3
SHA-256 checksum
How to use checksums
87bf04af629580b8484547ad02cc2fa334356b196fce7707ed25de3865726ff0
BLAKE2b-256 checksum
How to use checksums
c505e2f81fe427db20f98af0bafe18b70d0c4af3f96780730c1719fedadc9c2f
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 12, 2026.

Transparency log

Security for infereval 0.17.9

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for infereval-0.17.9.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 12, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.17.9 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page