Skip to main content

Pinocchio

Fast uncertainty estimates for black-box language models.

Pinocchio is an external calibrator that estimates the correctness of responses from black-box API models. It needs only a single forward pass to generate an uncertainty estimate and requires no access to the target model's logits, weights, or internal states.

Paper: arxiv.org/abs/2609.24881 · Project page: pinocchio-uq.com · Accepted to COLM 2026

Quickstart

pip install pinocchio-uq
from openai import OpenAI
from pinocchio import Pinocchio

client = OpenAI()
judge = Pinocchio()  # load once

messages = [{"role": "user", "content": "What is the capital of France?"}]
response = client.chat.completions.create(model="gpt-5", messages=messages)

p_correct = judge.score(response, messages=messages)

judge = Pinocchio() and judge.score(...) are the two lines Pinocchio adds to existing code. The first call downloads the base model (Qwen/Qwen3.5-0.8B) and the Pinocchio adapter (KevinDavidHayes/pinocchio-0.8b) from the Hugging Face Hub, about 2 GB in total.

What it is

  • Black-box. Pinocchio reads the question and the response. It never touches the target model.
  • One forward pass. No sampling, no logit access, and no modification of the target model.
  • Lightweight. This package ships a text-only 0.8B checkpoint, which matches the AUROC of the paper's largest model. It runs on a CPU.

In the paper, Pinocchio reaches 0.863 AUROC predicting the correctness of held-out responses, compared with 0.649 for the strongest single-pass black-box baseline, and transfers zero-shot to thirteen unseen models across eight organizations (mean AUROC 0.814). Those results use the paper's 8B model; the text-only 0.8B checkpoint here reaches 0.867 AUROC on the held-out text responses.

Usage

Score an OpenAI response

p_correct = judge.score(response, messages=messages)

Score a question and answer from any source

p_correct = judge.score(question="What is 2+2?", answer="4")

Batch scoring

items = [
    {"question": "What is the capital of Japan?", "answer": "Tokyo"},
    {"question": "What is the capital of Japan?", "answer": "Kyoto"},
]
scores = judge.score_batch(items)

Abstain or escalate when confidence is low

if judge.score(response, messages=messages) < 0.5:
    escalate(response)  # e.g. route to a human or a stronger model

How it works

Pinocchio is a small calibrator trained jointly on responses from seven LLMs that span a wide capability range. It takes the question, the response, and the identity of the model that produced it, and predicts a single token, i (incorrect) or ii (correct). The correctness probability is the softmax over those two logits:

P(correct) = exp(z_ii) / (exp(z_i) + exp(z_ii))

The logits are the calibrator's own; the target model is never inspected.

API

Pinocchio(adapter, base_model, device_map, torch_dtype, use_placeholder_image)

Parameter Default Description
adapter "KevinDavidHayes/pinocchio-0.8b" Hugging Face Hub repo or local path of the adapter
base_model "Qwen/Qwen3.5-0.8B" Base model
device_map "auto" Device placement
torch_dtype bfloat16 on GPU, float32 on CPU Model precision
use_placeholder_image True Reproduce the training input, which paired every text example with a blank image

judge.score(response=None, *, question, answer, messages, source_model, benchmark)

Returns P(correct), a float in [0, 1].

Parameter Description
response OpenAI ChatCompletion; the answer and model name are read from it
question The question that was asked
answer The answer to score, if response is not given
messages OpenAI messages list; the last user message is used as the question
source_model Optional name of the model that produced the answer
benchmark Optional benchmark name

judge.score_batch(items, prompt_variant="combined", show_progress=True)

Returns a list of P(correct), one per item.

Limitations

  • Distribution shift. Pinocchio can be miscalibrated for data far from its training mix. The paper shows that recalibrating with about 100 labeled examples (Platt scaling) restores calibration on new targets without changing the ranking.
  • Easy questions. Pinocchio was trained on challenging benchmarks where models frequently make mistakes. On trivially easy questions its scores are less reliable: in our tests, the wrong answer "5" to "What is 2+2?" still scored 0.97.
  • Text only. This 0.8B release scores text questions and answers. Images in messages are ignored.
  • English. The training data is predominantly English.

Citation

@inproceedings{hayes2026pinocchio,
  title     = {Pinocchio: Fast Uncertainty Estimates for Black-Box Language Models},
  author    = {Hayes, Kevin David and Pal, Arka and Zhang, Haosong and
               Goldstein, Tom and Goldblum, Micah},
  booktitle = {Conference on Language Modeling (COLM)},
  year      = {2026}
}

License

Apache 2.0. See LICENSE.

Metadata

Release files for pinocchio-uq 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for pinocchio-uq 0.1.0
File Size Uploaded Provenance Actions
pinocchio_uq-0.1.0.tar.gz 19.3 kB Without provenance Details for pinocchio_uq-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for pinocchio-uq 0.1.0
File Interpreter ABI Platform Actions
pinocchio_uq-0.1.0-py3-none-any.whl Python 3 none any Details for pinocchio_uq-0.1.0-py3-none-any.whl

Total release size: 33.5 kB

Release files / pinocchio_uq-0.1.0.tar.gz

Download URL pinocchio_uq-0.1.0.tar.gz
Size 19.3 kB
Tags Source
SHA-256 checksum
How to use checksums
396e9c37a02e38f92006d3a188531c025a530047ca4ee768863fbbcd0091d9cc
BLAKE2b-256 checksum
How to use checksums
bc79daff21bbe4d41a14e73c31cecf5cec4d824d59912614c2a6c31f19b56630
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.9.6

Release files / pinocchio_uq-0.1.0-py3-none-any.whl

Download URL pinocchio_uq-0.1.0-py3-none-any.whl
Size 14.2 kB
Tags Python 3
SHA-256 checksum
How to use checksums
aa9026bbcf08e3ce033f43fc8fac23a12f81e0d82f3d92b16069ba45a93a1e42
BLAKE2b-256 checksum
How to use checksums
f0c7a2ea70bf43fc2a5acea880217192d571a2be2fa34e31b6600a0e345cd2c4
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.9.6

Security for pinocchio-uq 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page