stated-confidence

Make LLM agents say how sure they are, and what that certainty rests on.

Ask a model to attach a confidence level and an evidential basis to each claim it produces and it fabricates less. The level alone is common practice; the basis is what does the work. Choosing between "documented fact", "expert opinion", and "educated guess" forces a provenance check the model would otherwise skip, and a claim it cannot back gets labelled as a guess or quietly dropped instead of dressed up as a figure.

This package is that schema, the vocabulary behind it, prompt guidance for filling it, and the consistency checks that send hedging-without-meaning-it back to the model. It was extracted from a production knowledge-graph generator where it was observed to sharply reduce invented facts and figures. The effect is observed, not yet measured; an evaluation harness is planned.

Install

pip install stated-confidence            # pydantic only
pip install "stated-confidence[pydantic-ai]"

The shape

from stated_confidence import Confidence, DataAnalysisBasis


class Finding(BaseModel):
    text: str
    confidence: Confidence[DataAnalysisBasis]

A Confidence has:

SPECULATIVE is not the bottom of the scale. It is the level for an idea the model is offering rather than a claim it is making: an extension a source suggests, a hypothesis, a what-if. A generator that has somewhere honest to put an idea does not dress it up as a fact; take that room away and the ideas come back as fabricated "facts" or disappear altogether.

The level scale is fixed. The basis set is the extension point. Three ship:

• GeneralKnowledgeBasis for claims about the world: documented fact, scientific consensus, historical record, industry standard, expert opinion, theoretical framework, fictional universe, speculation, educated guess.
• DataAnalysisBasis for agents reasoning over data they retrieved: query result, computed, schema inference, general knowledge, assumption.
• DecisionBasis for recommendations: policy, measured outcome, precedent, trade-off analysis, expert judgment, assumption.

Define your own by subclassing BasisEnum with a __terms__ table and a __fallback__ value.

Prompting

from stated_confidence import confidence_instructions

system_prompt += confidence_instructions(DataAnalysisBasis, localized=True)

Pydantic AI

from pydantic_ai import Agent
from stated_confidence.pydantic_ai import confidence_output_validator

agent = Agent("openai:gpt-5", output_type=Report)
agent.output_validator(confidence_output_validator())

Every Confidence anywhere in the output is checked. A HIGH level on a basis of ASSUMPTION, a SPECULATIVE level on a basis of DOCUMENTED_FACT, or a non-HIGH level with empty reasoning comes back to the model as a retry listing what to fix. Pass extra (ctx, output) -> list[str] callables for rules that need run context, such as requiring that evidence names a column the agent's query actually returned.

Lenient parsing, strict prompting

Models drift. "High" parses. 0.9 parses (as HIGH). An invented basis value parses as the set's fallback and is logged at WARNING on the stated_confidence logger so the set can grow from what models actually say.

TypeScript

The same vocabulary, generated from taxonomy.json, with a Zod schema for structured output and the same display helpers and consistency checks:

npm install stated-confidence

import { DATA_ANALYSIS_BASIS, confidenceInstructions, checkAll, describe, tone } from 'stated-confidence';
import { confidenceSchema } from 'stated-confidence/zod';

See typescript/README.md. CI checks that the prompt instructions the two packages produce are word-for-word identical.

Display

describe(value) returns a label and description for any level, type, or basis value; tone(level) maps to positive, neutral, caution, or warning with no CSS framework attached. taxonomy.json at the package root is the same vocabulary as data, for other languages to consume.