detectorproof

Measure the detectors before you trust any of them.

detectorproof is not a synthetic-speech detector and must not become one. It is an instrument for measuring detectors: what a declared panel of frozen checkpoints can and cannot do on your material, and what their judgments are reacting to.

One command is built: panel. It scores a declared set of frozen detectors over a labelled control set, reports the area under the ROC curve for each, and refuses to let an incompetent one contribute to anything downstream.

No audio is read and no model is loaded. It works on a score table you already have, has no dependencies, and runs on a machine that will never hold the corpus.

pip install detectorproof

Or from a clone, which is what you want if you intend to run the tests:

git clone https://github.com/Mormolykos/detectorproof.git
cd detectorproof
pip install .          # or: pip install -e ".[dev]" && pytest

Python 3.10 or newer, no runtime dependencies. Both forms install the detectorproof command and the importable package. See CHANGELOG.md for release status.

Why this exists

On one corpus, three widely benchmarked anti-spoofing detectors sat at chance while their model cards advertised sub-1% equal error rates. A panel that quietly averaged them in would have reported a confident number built partly on models that could not do the task at all.

Separately, and this is the finding the package is built around: genuine human speech can move strongly toward "synthetic" after ordinary benign processing that generates nothing. A neural-codec round trip of real studio recordings moved 9 of 13 frozen detectors past multiple-comparison correction, three of them on 47 of 47 utterances at the maximum a rank statistic can reach. A phase-retrieval algorithm from 1984 moved 12 of 13.

Those numbers are not asserted here. They come from a published, independently audited study, and every one of them regenerates from its deposit:

Gkilis, P. (2026). Reconstruction history, not synthesis: benign processing moves frozen detectors on genuine human speech. Zenodo. 10.5281/zenodo.22819223 · CC BY 4.0

⚠️ Read its scope limits before quoting it: three speakers, 47 utterances, neutral speech, one language; seven of the thirteen detectors come from a single research group; no mechanism is identified and no listening test was run. Thirteen of that paper's own claims were withdrawn during three rounds of independent adversarial audit, and the withdrawals are published in it.

So a detector score is not interpretable without knowing the provenance of the audio it was computed on — and before any of that, you need to know which detectors on your panel can do the task on your material at all. That is what panel answers.

Usage

detectorproof panel \
  --panel panel.json \
  --scores scores.csv \
  --labels labels.csv \
  --condition CTRL

panel.json — the declaration, fixed before the run.

[
  {"name": "det-a", "orientation": "higher_bonafide", "identity": "sha256:1f3a...", "source": "..."},
  {"name": "det-b", "orientation": "higher_synthetic", "identity": "rev:c1dbe2ae"},
  {"name": "det-c", "orientation": "unresolved", "identity": "commit:72c32e28"}
]

orientation is resolved from the model's own documentation or training-label path, never from what makes your benchmark look right. unresolved is a first-class value: the detector is scored and reported, and its sign is not interpreted.

identity pins the weights — a revision, a commit, a file hash. It is required. A panel that cannot say which checkpoint produced a number is not frozen.

scores.csv — detector,item,condition,score. One row per detector per item per condition. Scores are in each detector's native units. A missing measurement is written NOT_MEASURED (or n/a), never 0.

labels.csv — item,label, where label is bonafide or synthetic.

In Python:

from detectorproof import Detector, run_panel, render

report = run_panel(detectors, scores, labels, condition="CTRL")
print(render(report))

for result in report.competent_detectors():   # the only set you may aggregate over
    ...

The refusals

These are enforced in code, not described here. Each was bought by a specific measured failure.

refusal	the failure that bought it
A detector at or near chance is NOT COMPETENT ON THIS MATERIAL — reported in full, excluded from every aggregate	three benchmarked detectors at chance while advertising sub-1% EER
The panel is declared before the run. An undeclared detector in the scores is a fault; a declared detector with no scores stays in the report as NOT MEASURED	a panel assembled after seeing results is a selection, not a measurement
Two scores for one detector and item are a fault, not something to average	overlapping analysis windows treated as independent samples inflated one published proportion from 75% to 86%
NOT MEASURED is a third state — never zero, never an absence of effect	a missing measurement reported as zero is worse than reporting nothing
A near miss at the band edge is recorded, not resolved by widening the band	one detector landed at 0.447 against a declared band of [0.45, 0.55] and the band stayed where it was

There is deliberately no function that recommends a transformation to move a score, and there will not be one. In the work this came from, an intervention tuned against a chosen detector statistic made the audio more separable, not less — one detector's AUC moved from 0.576 to 0.763. The instrument reports; it does not advise. That refusal is the point, not an omission.

What a verdict means

NOT COMPETENT ON THIS MATERIAL is a statement about that checkpoint on your control set. It is never a statement about a detector's general quality. Every detector was trained by someone else, on other data, for other purposes.

An empty competent set is a reportable result about your panel and your material. It is not evidence that your audio is genuine, and it is not evidence that detection is impossible. Absence of evidence is not evidence of absence, and nothing downstream may run on an empty competent set.

Design

Zero dependencies in the core. Ties are handled by midranks everywhere, because a detector whose scores compress near zero produces long runs of equal values and a rank statistic that breaks ties arbitrarily reports a different answer depending on the order its input arrived in. AUC is computed from the rank sum rather than by counting pairs.

Every field this package reads has a declared domain — its type, its admissible values, and what absence means. Validation asks whether a value is inside that declaration; it does not carry a list of bad values anyone has seen. That list is unbounded and the domain is not.

The family

trainproof (training runs) · ttsproof (TTS failure modes) · spkproof (speaker-verification false rejection) · detectorproof (detector competence and robustness).

"Did the processing preserve speaker identity?" is a different question with different statistics, and spkproof answers it. Version 0.1.0 has no dependency on spkproof and does not call it — the two are used side by side, not chained.

Status

panel is built and tested. probe, identity and falsepositive are specified and not built. They are deliberately not built at the same time — panel is the gate the other three depend on, and it has to refuse correctly on a detector already known to be incompetent before anything is built on top of it.

Licence

MIT.