truthanchor 0.1.0

Calibrate raw LLM uncertainty scores into truth-aligned scores.

TruthAnchor

TruthAnChor (TAC) calibrates raw uncertainty scores of LLM responses into truth-aligned scores with a lightweight MLP.

What It Does

TruthAnchor supports:

• response generation for benchmark datasets
• raw uncertainty score computation
• truth-anchored score mapping with a lightweight MLP
• optional CUE comparison
• metric reporting with AUROC, ECE, and PRR
• plotting for calibration and score comparison
• custom score-label CSV input

Installation

pip install truthanchor

For local development from this repository:

pip install -e .

Quickstart

Run the end-to-end example pipeline:

python3 examples/tac_eval.py

This runs:

1. generation
2. uncertainty scoring
3. mapper training
4. held-out evaluation

Custom Score-Label CSV Input

You can also run the pipeline on a custom CSV containing:

• first column: uncertainty scores
• second column: binary labels

Example:

python3 examples/tac_eval.py \
  --datasets my_dataset \
  --models custom \
  --custom_scores_csv data/my_dataset.csv \
  --custom_method_name my_score \
  --higher_worse true

Note: custom CSV labels are assumed to use 1 = correct and 0 = incorrect.

Outputs

The example pipeline writes results under:

outputs/<dataset>/<sanitized-model>/

Including:

• uncertainty_scores.npz
• mapper_eval/results.csv
• mapper_eval/scores/*.npz
• optional comparison and calibration plots

Notebook Walkthrough

A step-by-step notebook example is available at:

examples/tac_eval_walkthrough.ipynb