Skip to main content

infereval

CI Docs Release PyPI Python License: MIT

Inferentialist evaluation of LLMs: derive an implication frame from a language model's endorsement verdicts, then measure the model's agreement with an analyst-labeled benchmark via coverage and Cohen's / Fleiss' kappa. The agreement is evidence bearing on an inferential-mastery attribution — not a measurement of mastery itself (per the paper's Remark 8).

infereval is the executable companion to Note on Simonelli's Stop Sign Dialogue: An Implication-Space Instrument for Probing LLM Endorsement of Material Inferential Rules (Allen, 2026), which is maintained as a separate paper. The framework formalizes the procedure β → η → (cov, κ_C, κ_F, κ_F*) for any analyst-supplied benchmark.

Status

Beta (0.x, pre-1.0). The public Python API and CLI surface may shift between minor releases until 1.0. Methodology defaults are locked, and the JSON schemas are versioned independently (schema_version: "1.0") and promised stable from 1.0 onward regardless of the framework version. See the CHANGELOG for the current release.

Documentation

The docs site at https://www.bradleypallen.org/infereval/ covers a Concepts page (methodology mental model), Authoring benchmarks, Interpreting metrics (κ_C / κ_F / κ_F*, test-retest κ, decompositions, subsampling CIs, sensitivity sweeps), Providers (Anthropic seed handling, DeepSeek reasoning-token budgets, OpenRouter attribution), and Construct validity of the instrument — the R1–R22 requirements catalogue and end-to-end workflow in one place. Four executable tutorial notebooks (quickstart, authoring, paraphrase-axis triangulation, pulmonology visualization). Plus an auto-generated API reference, an Architecture dataflow diagram, a Glossary of paper symbols, and a JSON-schema reference.

Findings

A 13-model × 3-paraphrase-variant cross-family sweep against the stop-sign benchmark (2026-06-09) is committed at experiments/results/stop_sign_2026-06-09.md. Headline: under the paper-aligned δ(ra), 12 of 13 frontier LLMs across six families reproduce Simonelli's analyst row exactly (κ_C = +1.000) — a thirteen-model independent replication of the paper's empirical anchor under fresh v0.15.2 captures. The perceptual variant is the cleavage axis. A companion 6-model pulmonary edema sweep (n=30 items) at experiments/results/pulmonology_2026-06-09.md characterizes within-day and day-out R22 reliability; the deepseek-v4-pro cell exhibits monotone κ decay across three time-scales (0.867 → 0.792 → 0.729) — the clearest published example of detectable across-update model drift the framework has produced.

Install

pip install infereval

Provider SDKs are optional extras (the framework runs without them — use the mock or replay providers):

pip install 'infereval[anthropic]'   # Anthropic Claude
pip install 'infereval[openai]'      # OpenAI + OpenRouter (OpenAI-API-compatible)
pip install 'infereval[all]'

From source (editable):

git clone https://github.com/bradleypallen/infereval
cd infereval
python -m venv .venv
source .venv/bin/activate
pip install -e '.[dev]'

60-second quickstart

Inspect the bundled stop-sign benchmark (Example 1 of the paper), then run an evaluation against the deterministic replay fixture — no API key needed:

# 1. Look at the benchmark.
infereval describe examples/stop_sign/benchmark.json

# 2. Validate it against the JSON schema.
infereval validate examples/stop_sign/benchmark.json

# 3. Run a deterministic evaluation against the committed replay fixture.
infereval evaluate examples/stop_sign/benchmark.json \
    --replay-from tests/fixtures/stop_sign_replay.jsonl \
    --output /tmp/eta.json \
    --n-samples 5 \
    --log /tmp/run.jsonl

# 4. Compute metrics.
infereval metrics /tmp/eta.json --benchmark examples/stop_sign/benchmark.json

To run against a real model, replace step 3 with:

export ANTHROPIC_API_KEY=...
infereval evaluate examples/stop_sign/benchmark.json \
    --provider anthropic --model claude-haiku-4-5-20251001 \
    --output /tmp/eta.json --n-samples 5 --log /tmp/run.jsonl

The JSONL run log under /tmp/run.jsonl records one event per provider call (prompt hash, raw response, parsed verdict, usage, timing) so the evaluation is auditable end to end.

For the R22 test-retest discipline, infereval retest --auto --benchmark <b> --provider X --model Y [--interval-s N] [--save-etas DIR] (v0.11.0+) collapses the four-step manual workflow (evaluate, evaluate again, retest, optionally --claims) into one CLI call — runs the same evaluation twice with an optional inter-capture sleep and emits the standard RetestResult directly. v0.12.0+ accepts --interval-s multiple times for cumulative drift-since-baseline analysis in one orchestrated call. v0.13.0+ surfaces test-retest κ as a co-equal ### Reliability (R22) subhead in infereval report §2 alongside ### Agreement, and auto-detects single vs multi-interval retest shape (multi-interval renders a per-interval table with a worst-case overall verdict). v0.14.0+ adds the staged-composition pattern: --baseline-from <eta-path> runs one fresh capture against a saved baseline and --append-to <multi.json> appends a new pair to an existing multi-result — both compute interval_s from actual elapsed wall clock, so day-out / week-out R22 evidence can ship as incremental commits without the CLI process needing to stay alive for the elapsed window. See the stop-sign R22 capture for the worked end-to-end demo.

What this is and isn't

This is: a research tool that formalizes Simonelli's stop-sign dialogue into a repeatable evaluation procedure. Given (i) a bearer set, (ii) expression and context-construction functions, (iii) a benchmark of implications labeled by one or more analysts, the framework drives an LLM through endorsement-probing for each implication and reports the resulting agreement with analyst practice along three axes:

  • Coverage — how often the model takes a substantive position (cov(η)).
  • Cohen's kappa — agreement against a chosen reference (analyst consensus c_i or a single analyst v_{:,j}).
  • Fleiss' kappa — agreement with the model treated as the (m+1)th annotator, alongside the inter-analyst baseline κ_F*(β) (Remark 4 of the paper).

Each metric can be decomposed by tag or by RSR target.

This is not: a factuality benchmark, a leaderboard, or an answer to whether LLMs are sapient. The methodology is carving-relative: results depend on the analyst-supplied bearer carving, context construction, and benchmark. The framework provides the machinery; the analyst supplies the practice the machinery is comparing against. See the Discussion in the paper for what carving-relativity buys and costs.

API surface

from infereval import (
    Verdict, Bearer, Implication,           # core data types
    DerivedFrame,                            # ⟨B, I_M⟩ per Definition 3
)
from infereval.benchmark import Benchmark
from infereval.evaluation import Evaluation, evaluate, EndorsementConfig, ProviderParams
from infereval.providers import get_provider
from infereval.metrics import MetricsReport

bench = Benchmark.load("examples/stop_sign/benchmark.json")
provider = get_provider("anthropic", "claude-haiku-4-5-20251001")
eta = evaluate(bench, provider,
               config=EndorsementConfig(n_samples=5),
               params=ProviderParams(temperature=1.0),
               log_path="/tmp/run.jsonl")
report = MetricsReport(eta=eta, benchmark=bench)
print(report.to_dict())

Locked methodology defaults

These are framework defaults, overridable per evaluation:

Setting Default
n_samples 5 (odd, clean 3-way majority)
Tie-break abstain (configurable: good, bad, first)
Verification prompt default-v1 (GOOD/BAD/ABSTAIN tokens with brief glosses)
TeX in expressions Stripped at prompt time; LaTeX-source-friendly in benchmark JSON
Cohen's kappa reference Analyst consensus c_i (override with --reference analyst:<id>)
Provider seed Honored by OpenAI; ignored (with one-time warning) by Anthropic

See CLAUDE.md and the paper for the full list and the rationale behind each choice.

Development

pip install -e '.[dev]'
pytest                                # all unit + replay tests
pytest -m live                        # opt-in live provider tests (requires API keys)
mypy src/infereval
ruff check src tests

Live provider tests require RUN_LIVE_PROVIDER_TESTS=1 and the relevant API key in the environment. They are skipped by default.

Citation

@unpublished{allen2026inferential,
  author = {Allen, Bradley P.},
  title  = {Note on {S}imonelli's Stop Sign Dialogue: An Implication-Space Instrument for Probing {LLM} Endorsement of Material Inferential Rules},
  year   = {2026},
  note   = {University of Amsterdam}
}

License

MIT — see LICENSE.

Metadata

Release files for infereval 0.17.8

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for infereval 0.17.8
File Size Uploaded Provenance Actions
infereval-0.17.8.tar.gz 445.7 kB With provenance Details for infereval-0.17.8.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for infereval 0.17.8
File Interpreter ABI Platform Actions
infereval-0.17.8-py3-none-any.whl Python 3 none any Details for infereval-0.17.8-py3-none-any.whl

Total release size: 684.0 kB

Release files / infereval-0.17.8.tar.gz

Download URL infereval-0.17.8.tar.gz
Size 445.7 kB
Tags Source
SHA-256 checksum
How to use checksums
c9757052fd5580c09267876cf0a6be08816e2b58c21725b8da8e1845a0f99f2c
BLAKE2b-256 checksum
How to use checksums
a6458a6d716079758a08d4d78a91b48f7708015b60aea13b9edd4f3b972902f2
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 27, 2026.

Transparency log

Release files / infereval-0.17.8-py3-none-any.whl

Download URL infereval-0.17.8-py3-none-any.whl
Size 238.3 kB
Tags Python 3
SHA-256 checksum
How to use checksums
3f9825c25206fe3f84a8a80c9aca6a258bb3605e4403093f6cb97170c287a422
BLAKE2b-256 checksum
How to use checksums
96a650789c50633088e0b3db478a84aaf4f7191f962c08c8ae4c5b48094aaf91
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 27, 2026.

Transparency log

Security for infereval 0.17.8

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for infereval-0.17.8.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 27, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.17.8 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page