gaige 0.0.4

Calibration and drift receipts for AI measurement: your corpus, your thresholds, honest error bars, and proof the instrument has not changed.

gaige

Calibration and drift receipts for AI measurement. Your corpus, your thresholds, honest error bars, and a fingerprint that proves the instrument hasn't changed underneath you.

gaige.dev · PyPI · issues · fuel the instrument

Why

Any score you act on comes from an instrument, and that instrument's numbers depend on the model, the quantization, the device, the library versions, and the material it reads. Change any of those and you have a different instrument, whether or not anyone noticed. Almost nobody who relies on a score knows its actual error rate on their own material; they trust a marketing page. When the consequence lands on a real person, "the tool said 87%" is not evidence.

Point gaige at a corpus or a probe set, give it any pluggable scorer, and what comes back is a receipts report. Inside it: the ROC and AUROC with bootstrap confidence intervals, and operating thresholds computed two ways: an empirical target-FPR threshold whose in-sample rate is labeled as exactly that, and a conformal one carrying a finite-sample guarantee, stated honestly as marginal. Error rates arrive per subgroup with intervals, the base-rate harm arithmetic is done for you, and the whole thing sits on an instrument fingerprint: model, device, versions, corpus hash, the exact reproduce command, and quantization verified at load time, because some library combos silently ignore 4-bit and load fp16, which shifts scores, so gaige refuses to emit numbers from a load it can't prove.

Two applications of one machine:

• Detector calibration: what is this detector's real false-positive rate on your material, rather than on a vendor's marketing page. If you use a detector: know its error bars before you act. If you are judged by one: these receipts are what an auditable process looks like.
• Instrument drift: has the system you're measuring changed, or has your measuring pipeline? Distinguishing those two is the difference between a finding and an artifact, and it is the harder question. gaige is built to answer it with evidence rather than assertion.

And since 0.0.3, a third surface on the same honesty rules: gaige verify, a provenance evidence sweep (C2PA Content Credentials, image watermarks, keyed text schemes) that emits evidence statuses, never a score, and states in every result what a negative from it means.

0.0.4 adds gaige admit: corpus admission on the same rules. Point it at an accepted baseline receipt and an unlabeled candidate slice, and it measures divergence from what you already trust: a conformal novelty rate carrying an exact finite-sample false-alarm law, KS distance and quantile shifts with bootstrap intervals, per-document placements, and stratified where-it-differs, with refusal floors wherever the evidence is too thin. The baseline is whatever your organization has vetted, human or AI; admission measures divergence, not class membership, and the receipt never says admit or reject.

Scoring needs a model and ideally a GPU. Analysis does not: gaige analyze re-derives thresholds and reports from scores that already exist, so calibration runs on a laptop, or a CPU-only machine, or an isolated environment with no accelerator at all.

Quickstart

pip install gaige                 # analysis only — numpy + requests, no torch
pip install "gaige[gpu]"          # scoring too: torch/transformers/bitsandbytes/accelerate
gaige run --corpus hc3-mini --n 100 --detector fast-detect-gpt
# → reports/<ts>/report.md + scores.csv + roc.json + env.json
gaige run --corpus your-labeled.jsonl   # rows: {"text": ..., "label": "human"|"ai"}
pip install "gaige[verify]"             # provenance evidence: c2pa-python + PyWavelets + opencv
gaige verify picture.png                # evidence statuses (FOUND/ABSENT/INCONCLUSIVE/NEEDS_KEYS), never a score
gaige admit --baseline reports/<ts>-<detector>/ --candidate new-material.jsonl
# → divergence of new material from your accepted baseline: intervals + refusal floors, never a verdict

On a CPU machine the quickstart saturates: the auto-selected gpt2-large separates this small corpus completely (AUROC 1.0000 on hc3-mini), which demonstrates the pipeline, not the difficulty of detection. For a calibrated example with honest error bars, see the published GPU receipt: AUROC 0.9720 (95% CI 0.9458-0.9938), Fast-DetectGPT on falcon-7b, 4bit, hc3-mini(n=100,seed=17).

On a CPU-only machine the [gpu] extra pulls the multi-gigabyte CUDA build of torch by default. Install the small CPU build first and the extra will leave it alone:

pip install torch --index-url https://download.pytorch.org/whl/cpu && pip install "gaige[gpu]"

From a source checkout: pip install -e .

Python API

The CLI is the primary interface. For library use, import gaige carries the supported surface: the calibrate/conformal/analyze spine. It never imports torch, so it runs where a model never could: a laptop, a CI job, an air-gapped review machine.

from pathlib import Path
import numpy as np
import gaige

# Replay an existing report — recompute every statistic from its scores.csv.
rows, corpus, meta = gaige.load_report(Path("reports/2026-07-26T18-02-11Z"))
results = gaige.compute_results(rows)

# Or bring scores from any instrument of your own (gaige refuses under 50 per class —
# CorpusTooSmall — because a threshold fitted on less is a guess).
rng = np.random.default_rng(17)
scores = np.concatenate([rng.normal(0, 1, 100), rng.normal(2, 1, 100)])
labels = np.array(["human"] * 100 + ["ai"] * 100)
gaige.auroc(scores, labels)
gaige.threshold_at_fpr(scores, labels, target_fpr=0.01)   # measured, labeled in-sample
gaige.conformal_threshold(scores[labels == "human"], alpha=0.05)  # finite-sample bound;
# raises InsufficientCalibration when your n cannot support the alpha you asked for.

Everything not in gaige.__all__ is internal and may move without notice.

What it will never do

• Ship a universal threshold. There isn't one; that's the point.
• Emit a verdict. Scores + measured error rates only.
• Pretend one corpus generalizes. Every report says what it was measured on.

Where gaige sits

The detection literature of 2024–2026 converged on exactly what gaige does (fixed-FPR operating points, calibrated in-domain, reported with error bars) while producing steady evidence that nobody's fixed threshold survives contact with a new domain, a new model, or even a new decoding setting.

Default thresholds are the documented failure. RAID (ACL 2024, arXiv:2405.07940), the largest shared detection benchmark (6M+ generations), measured open-source detectors at naive default thresholds running 47–100% false-positive rate on human text (GLTR 100%, LLMDet 97.9%, Fast-DetectGPT 47.3%) while the commercial detectors it tested ship factory-calibrated at ≤1.7%. Its stated advice is gaige's thesis, published: "calibrate detectors on in-domain data before using them." RAID also fixes and discloses FPR for every number it reports, and calls itself one of the first shared resources to do so.

Accuracy does not survive a change of scene. In a 2026 cross-dataset audit (arXiv 2604.16607), every one of 15 detector variants fell to near-chance (AUROC ≤ 0.60) on at least one evaluation set (the best worst-case was that 0.60), and 12 of the 15 showed ≥15% FPR on human text on at least one dataset. An independent black-box evaluation on deliberately-unseen generators (arXiv 2412.05139) put the best detector it tested at 0.58 TPR at 1% FPR. Rankings reorder across benchmarks and metrics, so a single-benchmark accuracy claim is not defensible without its corpus disclosed, and detectors also move under plain configuration changes (sampling strategy, repetition penalty, an unseen model; measured in RAID) with no "attack" involved.

Even flagship thresholds are global. Binoculars (best worst-case in the audit above and best in the black-box evaluation) ships a single global threshold (0.901) optimized for accuracy on three reference datasets; its headline 0.01%-FPR figure is a point read off a ROC curve, not a deployment threshold. Measured here: the paper's global thresholds run at 16% and 3% FPR on our reference corpus (hc3-mini, n=100, seed=17, both scoring models quant-verified on the receipt). Thresholds do not transfer. That is not a flaw in Binoculars; it is the reason calibration tooling has to exist.

FPR-bounded calibration is published methodology with no product behind it. Conformal calibration (arXiv 2505.05084) bounds the false-positive rate with a finite-sample guarantee, and the bound held empirically at α from 0.2 down to 0.005 across seven detectors, but naive single-quantile conformal costs real TPR, which is why the trade must be printed, never hidden. gaige implements the single-quantile mode with the trade printed. Measured on the reference instrument (Fast-DetectGPT, falcon-7b, 4-bit, hc3-mini n=100 seed=17): α=0.01 → threshold 2.4446 at TPR 76%, beside the in-sample 1%-target pair (2.1229, TPR 86%); α=0.005 correctly refused at n=100: the guarantee's price, measured and shown.

Aggregates hide the numbers that matter. Under a static threshold, short human texts are flagged at measurably higher rates (arXiv 2502.04528: score distributions differ significantly by length and style, KS_max 0.3081, p<0.01), and the group-threshold flagship paper's own tables show its disparity cut bought with mean F1 collapsing 0.51→0.39, invisible in its headline metrics. Our first RAID receipt says the same about decoding: on a stratified slice (raid g2×d2×a2, per-cell reservoir seed 17; instrument Fast-DetectGPT, falcon-7b, 4-bit, cuda), aggregate AUROC 0.9285 and TPR@1%FPR 61.5% (against 86% on the reference corpus, same instrument, harder corpus) concealed a decoding split of greedy 87.6% vs sampled 39.7% TPR@1%FPR: a 48-point spread inside one aggregate. Per-subgroup rates with intervals, and refusals where n is too small to say, are the honest form of the answer.

So gaige is not a detector, and not a claim that detection works. It is the measurement layer this literature keeps asking for: your corpus, your instrument, its actual error rates, with the fingerprint that proves which instrument produced them, and a report that refuses to say more than was measured.

Status / roadmap

v0.0.1 (bursts 1-2a): Fast-DetectGPT (analytic single-model) · HC3 subsample corpus · ROC/AUROC/ threshold receipts with bootstrap CIs · quantization verification · CPU support + gaige analyze replay · resumable scoring · conformal thresholds (split conformal per arXiv:2505.05084; finite-sample bound P(human flagged) ≤ α, stated honestly: marginal over calibration draws, with the exact conditional Beta dispersion printed beside it) · subgroup-stratified receipts (length buckets always, metadata axes when the corpus carries them; every rate with a bootstrap interval, rates on fewer than 20 samples withheld: counts speak instead) · base-rate arithmetic in every report (FPR × your volume = wrongly flagged per year, plus PPV at assumed prevalences: the calculation Vanderbilt published when it disabled its detector) · Binoculars as detector #2 (released-implementation construction, both models quant-verified on the receipt; the paper's global thresholds measured at 16%/3% FPR on the reference corpus: calibrate in-domain, with receipts) · quantization A/B receipts (measured: 4-bit moves the 1%-FPR threshold ~10% where fp32-vs-fp16 agree to four decimals, so quantization is an instrument parameter) · the probe runner, run registry, P(True)/ECE, and drift monitors of the longitudinal apparatus (see RUNBOOK Workflow B) · providers local-hf / llama.cpp / ollama (attestation earned per artifact: GGUF sha256; ollama's manifest+weights chain re-hashed by gaige itself) · gaige plan (what this machine can run, at measured cost, with attestation, but no quality column, deliberately: separation lives in receipts).

v0.0.2 (the quality release): gaige export — a report's results and environment joined into one public receipt document (schema gaige-receipt-export/1, plus a rebuilt index), with fail-closed structural redaction: absolute paths, IPs, and non-allowlisted URL hosts refuse and name the offending field, because a receipt destined for a public surface must not leak the bench it was made on. Reports without an instrument fingerprint are refused outright: a public number without its instrument is what receipts exist to prevent · zero scoring drift proven byte-for-byte: 0.0.1-tagged code and 0.0.2 produce identical analyses of the reference receipts, only the version stamp differs.

v0.0.3 (the provenance release): gaige verify — C2PA Content Credentials (absence is a typed signal: only ManifestNotFound reads ABSENT, a validation failure is never "no manifest"), the publicly checkable dwtDct image watermark behind a per-image carrier self-test (a negative reads ABSENT only when a probe payload embedded into that same image survives a round trip, proving the carrier could have held the mark; otherwise INCONCLUSIVE), and keyed text schemes reported honestly as NEEDS_KEYS. The overclaiming result cannot be constructed; the type system refuses it. The dwtDct codec is vendored (upstream imports torch at package-import time for an unrelated scheme), bit-format cross-validated against the real library in both directions, and pinned by a real-encoder golden fixture in CI · measured per-instrument memory floors: the floor gaige run enforces and the needs gaige plan prints are the same single-sourced numbers, with --min-free-gb as the deliberate escape hatch, named in both refusal messages · "what the aggregate hides" (docs/): TPR@1%FPR 86.0% on the reference corpus, 61.5% aggregate on a harder RAID slice, and greedy 87.6% vs sampled 39.7% inside that aggregate.

v0.0.4 (the admission release): gaige admit — unlabeled candidate material measured against an accepted baseline receipt under the baseline's fingerprinted instrument. The guarantee-bearing number is a two-sided conformal novelty rate: the share of candidate documents outside the baseline's acceptance band, beside the exchangeable expectation, with each side's exact conditional Beta law printed and a floor of ceil(2/alpha)-1 reference scores (a tighter guarantee than your data supports is refused, not granted). Support: two-sample KS distance and quantile shifts with bootstrap intervals and deliberately no p-values, per-document placements with two-sided conformal p-values, and stratified where-it-differs with rates withheld under 20 documents per stratum. Live scoring refuses on any instrument mismatch; supplied scores are accepted for the no-GPU lane and labeled unattested. Verdict-free by construction: the receipt never says admit or reject · data_license on built-in corpora: hc3-mini and RAID slices now record the dataset card's declared license in every receipt's corpus fingerprint.

Full history: CHANGELOG.

Next (research-ranked, see PROGRESS.md): SynthID-Text watermark receipts (detection is per-configuration and Google delegates the two thresholds to deployers — calibration homework, which is this tool's home turf) · verify-receipt fields for revocation and trust-list state, with trust-list changes as a drift event class · C2PA-in-text reporting (spec 2.3 embeds manifests in unstructured text) · per-subgroup conformal thresholds (Mondrian; the guarantee-backed version of group-adaptive thresholding, needs larger calibration sets) · adversarial degradation panels · TH-Bench corpus adapter (the RAID adapter landed: gaige corpus prepare-raid) · drift canaries (does your instrument still measure what it measured last term?).

Licensing and the name

Code: AGPL-3.0 (LICENSE). Free to use, study, modify, and share, including inside an institution, in research, and in an appeal against a detector's verdict. If you offer a modified version to others as a network service, publish your modifications. A commercial license removes that obligation: see COMMERCIAL.md.

Name: not licensed. No open-source license grants trademark rights. You may fork freely, but under a different name. See TRADEMARK.md; the standards there exist so that "a gaige report" keeps meaning something.

Maintainership is deliberately closed (see CONTRIBUTING.md), but issues and receipt-backed reproductions very welcome; pull requests not accepted at this time.

Fuel

The fuel link is at ko-fi.com/gaigedev. A coffee's worth genuinely helps, and it goes where the roadmap says it goes: GPU hours mostly, corpus licensing when one comes up. gaige itself stays AGPL and free no matter what happens here, and no amount of support has ever changed a number this tool prints.