Calibrate scores. Control discoveries. Monitor change.
Documentation · Batch workflow · Sequential workflow · API reference · Paper
nonconform calibrates anomaly scores for discovery, threshold
certification, and sequential change monitoring. Wrap a
supported scikit-learn estimator, a PyOD model,
or a custom detector:
- Discovery: Use calibrated p-values directly or call
select(...)to apply false discovery rate (FDR) control. - Score thresholds: Build a simultaneous false-alarm/FPR certificate and choose a score threshold for future inliers.
- Stream: Use conformal martingales to accumulate evidence against exchangeability and trigger configured alarms.
Why nonconform?
- Calibrate anomaly scores into conformal p-values using reference data.
- Certify raw-score thresholds with simultaneous FPR certificates and target-driven threshold inversion.
- Control discoveries by accounting for multiple tests within a fixed family.
- Monitor streams for change with conformal martingales, anytime evidence against exchangeability, and configurable alarms.
- Keep your detector through support for PyOD, recognized scikit-learn estimators, and protocol-compliant custom models.
Installation
nonconform requires Python 3.12 or newer. Both discovery control and
sequential monitoring are included in the core installation.
pip install nonconform
For the PyOD detector collection and benchmark datasets:
pip install "nonconform[pyod,data]"
Optional extras
| Extra | Adds |
|---|---|
pyod |
PyOD anomaly detectors |
data |
oddball benchmark datasets and PyArrow support |
fdr |
Streaming FDR procedures from online-fdr |
probabilistic |
KDE-based probabilistic estimation and tuning |
all |
Every optional feature |
Quick start
Batch discovery control
This core-only example demonstrates the batch lane. The detector is trained on normal data, part of which is reserved automatically for conformal calibration.
import numpy as np
from sklearn.ensemble import IsolationForest
from nonconform import ConformalDetector, Split
rng = np.random.default_rng(42)
x_train = rng.normal(size=(3_000, 2))
x_test = np.vstack([
rng.normal(size=(200, 2)),
rng.normal(loc=5.0, size=(20, 2)),
])
detector = ConformalDetector(
detector=IsolationForest(random_state=42),
strategy=Split(n_calib=1_000),
seed=42,
).fit(x_train)
discoveries = detector.select(x_test, alpha=0.05)
p_values = detector.last_result.p_values
print(f"Selected {discoveries.sum()} of {len(x_test)} observations")
For threshold exploration in one fixed batch, use
certificate = detector.fdp_bounds(x_test, confidence=0.95) and query its
realized-FDP upper curve. certificate.select(0.01) applies the p-value cutoff
0.01; it does not request FDP at most 1%. The ordinary call prepares one
default envelope using fresh Monte Carlo sampling, and subsequent queries
reuse it. Coverage is joint over data and that sampling. Keep the scoring rule
and family fixed, and do not cherry-pick repeated constructions. The
FDP guide
shows a complete example and the separate expert interface for method settings.
Raw-score false-alarm control
If the operational requirement is “flag no more than 5% of future inliers,” use the raw-score certificate instead of treating an FDR target as a score threshold:
certificate = detector.fpr_bounds(confidence=0.95)
scores = detector.score_samples(x_test)
selected = certificate.select(scores, target_fpr=0.05)
print(f"Certified threshold: {certificate.threshold_for(0.05)}")
With probability at least 95% over the clean calibration draw, the certificate bounds the population inlier false-alarm rate at every threshold, conditional on the independently fitted score map. On that event, the chosen threshold has a future-inlier false-alarm probability at most 5%. Calibration and future inlier scores must be i.i.d. from the same distribution; exchangeability alone is insufficient.
The certificate uses the deterministic finite-sample one-sided KS distribution.
With 1,000 calibration scores at 95% confidence, its finite-threshold band floor
is about 3.85%, allowing the 5% target here. Smaller targets can produce an
infinite threshold and an empty selection. Native certificates support
unweighted Split calibration independently of the p-value estimator. This
does not guarantee a realized false-alarm proportion in a finite batch, FDR,
FDP, recall, or sequential Ville control.
Sequential change monitoring
A fitted Split detector can initialize the stream lane without refitting its
scoring model. The example is self-contained so it can be copied independently
of the batch example.
Show sequential monitoring example
import numpy as np
from sklearn.ensemble import IsolationForest
from nonconform import ConformalDetector, Split
from nonconform.martingales import AlarmConfig, SimpleJumperMartingale
from nonconform.monitoring import ExchangeabilityMonitor
rng = np.random.default_rng(42)
x_train = rng.normal(size=(1_000, 2))
detector = ConformalDetector(
detector=IsolationForest(random_state=42),
strategy=Split(n_calib=0.3),
seed=42,
).fit(x_train)
alpha = 0.05
monitor = ExchangeabilityMonitor.from_split_detector(
detector,
martingale=SimpleJumperMartingale(
alarm_config=AlarmConfig(restarted_ville_threshold=1 / alpha)
),
seed=42,
)
# Stable observations followed by a distribution shift
x_stream = np.vstack([
rng.normal(size=(50, 2)),
rng.normal(loc=3.0, size=(50, 2)),
])
for x_t in x_stream:
state = monitor.update(x_t)
if "restarted_ville" in state.triggered_alarms:
print(f"Change alarm at step {state.evidence_step}")
break
Under the sequential validity assumptions, the restarted Ville alarm at
1 / alpha controls the probability of ever crossing on one stream. It does
not control FDR across multiple streams. See the
sequential monitoring guide
for the full guarantee scope and other alarm statistics.
Choose a workflow
| Goal | Start with |
|---|---|
| Calibrate and select anomalies in a batch | Split and select(...) |
| Set a per-inlier raw-score false-alarm target | fpr_bounds(...) and FPR certificates |
| Aggregate evidence across random splits | DerandomizedSplits and e-BH |
| Monitor a stream for change | Exchangeability martingales |
| Reuse more data for fitting and calibration | CrossValidation or JackknifeBootstrap |
| Account for covariate shift | Weighted conformal inference |
| Calibrate groups with different normal behavior | Group-conditional (Mondrian) p-values |
| Certify realized FDP across p-value thresholds | fdp_bounds(...) and FDP certificates |
| Bring a custom or third-party detector | Detector compatibility |
Statistical scope
nonconform calibrates detector scores; it cannot make an unsuitable detector
or mismatched calibration set valid. Spatial or temporal dependence must be
handled explicitly before applying standard exchangeability-based claims. See
the guides to FDR control
and sequential monitoring
before relying on error-control statements in a new application.
Citation
If you use nonconform in academic work, please cite the
accompanying paper:
@inproceedings{Hennhofer_Conformal_Anomaly_Detection_2026,
author = {Hennhöfer, Oliver and Kirsch, Maximilian and Preisach, Christine},
booktitle = {Proceedings of the Fifteenth Symposium on Conformal and Probabilistic Prediction with Applications},
pages = {613--632},
publisher = {PMLR},
title = {{Conformal Anomaly Detection in Python: Moving Beyond Heuristic Thresholds with nonconform}},
url = {https://proceedings.mlr.press/v329/hennhofer26a.html},
year = {2026}
}
Project
Read the documentation, browse
the changelog,
or report a problem in the issue tracker.
Contributions are welcome; start with the
contributing guide.
nonconform is distributed under the
BSD 3-Clause License.
Metadata
Release files for nonconform 2.0.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| nonconform-2.0.0.tar.gz | 94.4 kB | Without provenance | Details for nonconform-2.0.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| nonconform-2.0.0-py3-none-any.whl | Python 3 | none | any | Details for nonconform-2.0.0-py3-none-any.whl |
Total release size: 207.0 kB
Release files / nonconform-2.0.0.tar.gz
| Download URL | nonconform-2.0.0.tar.gz |
|---|---|
| Size | 94.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
258212539e370edfe41eefb332fa36fedbde0bc07dc94d00806e0cf206a9c724
|
|
BLAKE2b-256 checksum How to use checksums |
d436c80bef0bf5a9ef967e125be23c330e47df9a37d4d0f13719fbc9eed7cef5
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.12.18 {"installer":{"name":"uv","version":"0.12.18","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":null,"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Release files / nonconform-2.0.0-py3-none-any.whl
| Download URL | nonconform-2.0.0-py3-none-any.whl |
|---|---|
| Size | 112.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
f079ea29fc009c9525e0f5d4f3f4c48dd1743b43abae01f74a08c49e47e1b158
|
|
BLAKE2b-256 checksum How to use checksums |
bc0bd98a3a2081e28275a08f6c286cc7793e30ab49b6290ed7fe74eeae4d7219
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.12.18 {"installer":{"name":"uv","version":"0.12.18","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":null,"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Security for nonconform 2.0.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.