validity-screen 0.1.0

Validity screening protocol for LLM confidence signals

validity-screen

Check whether an LLM's confidence signal carries information before you build on it.

Implements the screening protocol from:

Cacioli, J. P. (2026). Screen Before You Interpret: A Portable Validity Protocol for Benchmark-Based LLM Confidence Signals. arXiv.

Install

pip install validity-screen

Quick start (Python)

import numpy as np
from validity_screen import screen

# Your data: item-level correctness and confidence
correct    = np.array([True, True, False, True, False, True, True, False])
confidence = np.array([True, True, True,  True, False, True, False, False])

result = screen(correct, confidence, model_name="My Model")

print(result.tier)          # 'Valid', 'Indeterminate', or 'Invalid'
print(result.vrs_table())   # Complete reporting table

Quick start (command line)

# From a CSV with 'correct' and 'confidence' columns
validity-screen run --data my_data.csv --model-name "GPT-5.4"

# From separate files
validity-screen run --correct correct.txt --confidence confidence.txt

# Continuous confidence? Auto-binarised at median
validity-screen run --data my_data.csv --confidence-col prob --threshold 0.5

# JSON output for pipelines
validity-screen run --data my_data.csv --json

What it does

Before computing calibration metrics (ECE), metacognitive sensitivity (meta-d', AUROC), or selective prediction accuracy, this protocol checks whether the confidence signal carries item-level information about correctness. If it doesn't, those downstream metrics are fitting noise.

Five values from a 2x2 contingency table. Three possible outcomes.

Tier	Meaning	Action
Valid	Confidence tracks correctness	Proceed with downstream metrics
Indeterminate	Near threshold, uncertain	Compute but flag; consider more items
Invalid	Confidence does not discriminate	Do not interpret AUROC, ECE, selective prediction

Indices

Index	What it detects	Invalid threshold
L	Blanket confidence on errors	>= 0.95
Fp	Over-withdrawal of correct items	>= 0.50
RBS	Inverted monitoring direction	> 0 (CI excludes zero)
TRIN	Fixed responding	>= 0.95 (warning only)
r	Item-level sensitivity	Reported, not thresholded

Batch screening

from validity_screen import screen_batch, summary_table

models = {
    "GPT-5.4":  {"correct": correct_gpt,  "confidence": conf_gpt},
    "Claude":   {"correct": correct_claude, "confidence": conf_claude},
    "Gemini":   {"correct": correct_gemini, "confidence": conf_gemini},
}

results = screen_batch(models, benchmark_name="MMLU")
print(summary_table(results))

Continuous confidence

from validity_screen import screen, binarise

# Binarise at a fixed threshold
confidence_binary = binarise(confidence_continuous, threshold=50)

# Or at the sample median
confidence_binary = binarise(confidence_continuous, method='median')

result = screen(correct, confidence_binary)

Requirements

• Python >= 3.8
• NumPy >= 1.20
• SciPy >= 1.7

Citation

@article{cacioli2026screen,
  title={Screen Before You Interpret: A Portable Validity Protocol for
         Benchmark-Based LLM Confidence Signals},
  author={Cacioli, Jon-Paul},
  journal={arXiv preprint},
  year={2026}
}

@article{cacioli2026validity,
  title={Before You Interpret the Profile: Validity Scaling for
         LLM Metacognitive Self-Report},
  author={Cacioli, Jon-Paul},
  journal={arXiv preprint},
  year={2026}
}

License

MIT