stated-confidence

Make LLM agents say how sure they are, and what that certainty rests on.

Ask a model to attach a confidence level and an evidential basis to each claim it produces and it fabricates less. The level alone is common practice, and it turns out to do most of the work of cutting the number of false claims. The basis is what happens to the claims the model still gets wrong: choosing between "documented fact", "expert opinion", and "educated guess" forces a provenance check, and a claim the model cannot back gets labelled as a guess instead of dressed up as a figure, so a reader can filter on it.

This package is that schema, the vocabulary behind it, prompt guidance for filling it, and the consistency checks that send hedging-without-meaning-it back to the model. It was extracted from a production knowledge-graph generator where it was observed to sharply reduce invented facts and figures, and then measured on four models.

Install

pip install stated-confidence            # pydantic only
pip install "stated-confidence[pydantic-ai]"

The shape

from stated_confidence import Confidence, DataAnalysisBasis


class Finding(BaseModel):
    text: str
    confidence: Confidence[DataAnalysisBasis]

A Confidence has:

SPECULATIVE is not the bottom of the scale. It is the level for an idea the model is offering rather than a claim it is making: an extension a source suggests, a hypothesis, a what-if. A generator that has somewhere honest to put an idea does not dress it up as a fact; take that room away and the ideas come back as fabricated "facts" or disappear altogether.

The level scale is fixed. The basis set is the extension point. Three ship:

• GeneralKnowledgeBasis for claims about the world: documented fact, scientific consensus, historical record, industry standard, expert opinion, theoretical framework, fictional universe, speculation, educated guess.
• DataAnalysisBasis for agents reasoning over data they retrieved: query result, computed, schema inference, general knowledge, assumption.
• DecisionBasis for recommendations: policy, measured outcome, precedent, trade-off analysis, expert judgment, assumption.

Define your own by subclassing BasisEnum with a __terms__ table and a __fallback__ value.

Prompting

from stated_confidence import confidence_instructions

system_prompt += confidence_instructions(DataAnalysisBasis, localized=True)

Pydantic AI

from pydantic_ai import Agent
from stated_confidence.pydantic_ai import confidence_output_validator

agent = Agent("openai:gpt-5", output_type=Report)
agent.output_validator(confidence_output_validator())

Every Confidence anywhere in the output is checked. A HIGH level on a basis of ASSUMPTION, a SPECULATIVE level on a basis of DOCUMENTED_FACT, or a non-HIGH level with empty reasoning comes back to the model as a retry listing what to fix. Pass extra (ctx, output) -> list[str] callables for rules that need run context, such as requiring that evidence names a column the agent's query actually returned.

Lenient parsing, strict prompting

Models drift. "High" parses. 0.9 parses (as HIGH). An invented basis value parses as the set's fallback and is logged at WARNING on the stated_confidence logger so the set can grow from what models actually say.

TypeScript

The same vocabulary, generated from taxonomy.json, with a Zod schema for structured output and the same display helpers and consistency checks:

npm install stated-confidence

import { DATA_ANALYSIS_BASIS, confidenceInstructions, checkAll, describe, tone } from 'stated-confidence';
import { confidenceSchema } from 'stated-confidence/zod';

See typescript/README.md. CI checks that the prompt instructions the two packages produce are word-for-word identical.

Measured

The harness in evals/ asks five models for six one-sentence claims about each of 48 subjects (16 well known, 16 real but obscure, 16 that do not exist) under three schemas: bare (text only), level (text plus a confidence level), and full (text plus Confidence[GeneralKnowledgeBasis]). A Gemini 3.1 Pro judge grades every claim as true, false, or unverifiable against ground-truth notes without seeing the confidence fields; a second judge from another family agrees with it on 90% of claims. The write-up is evals/RESULTS.md.

Two levels of pressure. With an escape hatch (a caveats field and permission to give fewer claims), all models keep false claims under 8% in every arm and mostly decline or hedge on the fictional subjects; the confidence fields change little there. The numbers that matter are from the forced run, where the model must produce exactly six claims and has no field to decline in. That is the situation of a structured generator (a graph must have nodes), and where the effect was first noticed.

False claims per 100 claims, forced run, three repeats, 864 claims per cell, 95% intervals in parentheses:

What the table says:

• Asking for a level cuts the false rate by two thirds or more on the models that were inventing. The cut is almost entirely on subjects the model does not know: false claims per fictional subject fall from 3.0 to 0.1 for claude-sonnet-5 and from 4.0 to 0.8 for gpt-5.4-mini. Detail errors on real subjects stay at a few percent in every arm.
• Asking for the basis as well lowers the count a little further on two models and not at all on the other three. What the basis does is mark what is still wrong: gemini-3.6-flash keeps inventing under pressure, but 91% of its false claims arrive labelled SPECULATIVE or with a SPECULATION basis, and false claims presented as HIGH fall from 29% of its output to 2%.
• The levels mean something. In the forced full arm, HIGH claims are 93-96% true on every model; SPECULATIVE claims are 4-53% true on the four models that used the level more than a handful of times.
• Strong-basis claims carried non-empty evidence 84-100% of the time. No model emitted a basis value outside the set.

Display

describe(value) returns a label and description for any level, type, or basis value; tone(level) maps to positive, neutral, caution, or warning with no CSS framework attached. taxonomy.json at the package root is the same vocabulary as data, for other languages to consume.