Skip to main content

nonconform

Calibrate scores. Control discoveries. Monitor change.

PyPI version Supported Python versions Tests Code coverage Documentation License

Documentation · Batch workflow · Sequential workflow · API reference · Paper

nonconform calibrates anomaly scores for discovery, threshold certification, and sequential change monitoring. Wrap a supported scikit-learn estimator, a PyOD model, or a custom detector:

  • Discovery: Use calibrated p-values directly or call select(...) to apply false discovery rate (FDR) control.
  • Score thresholds: Build a simultaneous false-alarm/FPR certificate and choose a score threshold for future inliers.
  • Stream: Use conformal martingales to accumulate evidence against exchangeability and trigger configured alarms.

Why nonconform?

  • Calibrate anomaly scores into conformal p-values using reference data.
  • Certify raw-score thresholds with simultaneous FPR certificates and target-driven threshold inversion.
  • Control discoveries by accounting for multiple tests within a fixed family.
  • Monitor streams for change with conformal martingales, anytime evidence against exchangeability, and configurable alarms.
  • Keep your detector through support for PyOD, recognized scikit-learn estimators, and protocol-compliant custom models.

Works with     scikit-learn      PyOD      Custom AnomalyDetector protocol

Installation

nonconform requires Python 3.12 or newer. Both discovery control and sequential monitoring are included in the core installation.

pip install nonconform

For the PyOD detector collection and benchmark datasets:

pip install "nonconform[pyod,data]"
Optional extras
Extra Adds
pyod PyOD anomaly detectors
data oddball benchmark datasets and PyArrow support
fdr Streaming FDR procedures from online-fdr
probabilistic KDE-based probabilistic estimation and tuning
all Every optional feature

Quick start

Batch discovery control

This core-only example demonstrates the batch lane. The detector is trained on normal data, part of which is reserved automatically for conformal calibration.

import numpy as np
from sklearn.ensemble import IsolationForest

from nonconform import ConformalDetector, Split

rng = np.random.default_rng(42)
x_train = rng.normal(size=(3_000, 2))
x_test = np.vstack([
    rng.normal(size=(200, 2)),
    rng.normal(loc=5.0, size=(20, 2)),
])

detector = ConformalDetector(
    detector=IsolationForest(random_state=42),
    strategy=Split(n_calib=1_000),
    seed=42,
).fit(x_train)

discoveries = detector.select(x_test, alpha=0.05)
p_values = detector.last_result.p_values

print(f"Selected {discoveries.sum()} of {len(x_test)} observations")

For threshold exploration in one fixed batch, use certificate = detector.fdp_bounds(x_test, confidence=0.95) and query its realized-FDP upper curve. certificate.select(0.01) applies the p-value cutoff 0.01; it does not request FDP at most 1%. The ordinary call prepares one default envelope using fresh Monte Carlo sampling, and subsequent queries reuse it. Coverage is joint over data and that sampling. Keep the scoring rule and family fixed, and do not cherry-pick repeated constructions. The FDP guide shows a complete example and the separate expert interface for method settings.

Raw-score false-alarm control

If the operational requirement is “flag no more than 5% of future inliers,” use the raw-score certificate instead of treating an FDR target as a score threshold:

certificate = detector.fpr_bounds(confidence=0.95)
scores = detector.score_samples(x_test)
selected = certificate.select(scores, target_fpr=0.05)
print(f"Certified threshold: {certificate.threshold_for(0.05)}")

With probability at least 95% over the clean calibration draw, the certificate bounds the population inlier false-alarm rate at every threshold, conditional on the independently fitted score map. On that event, the chosen threshold has a future-inlier false-alarm probability at most 5%. Calibration and future inlier scores must be i.i.d. from the same distribution; exchangeability alone is insufficient.

The certificate uses the deterministic finite-sample one-sided KS distribution. With 1,000 calibration scores at 95% confidence, its finite-threshold band floor is about 3.85%, allowing the 5% target here. Smaller targets can produce an infinite threshold and an empty selection. Native certificates support unweighted Split calibration independently of the p-value estimator. This does not guarantee a realized false-alarm proportion in a finite batch, FDR, FDP, recall, or sequential Ville control.

Sequential change monitoring

A fitted Split detector can initialize the stream lane without refitting its scoring model. The example is self-contained so it can be copied independently of the batch example.

Show sequential monitoring example
import numpy as np
from sklearn.ensemble import IsolationForest

from nonconform import ConformalDetector, Split
from nonconform.martingales import AlarmConfig, SimpleJumperMartingale
from nonconform.monitoring import ExchangeabilityMonitor

rng = np.random.default_rng(42)
x_train = rng.normal(size=(1_000, 2))

detector = ConformalDetector(
    detector=IsolationForest(random_state=42),
    strategy=Split(n_calib=0.3),
    seed=42,
).fit(x_train)

alpha = 0.05
monitor = ExchangeabilityMonitor.from_split_detector(
    detector,
    martingale=SimpleJumperMartingale(
        alarm_config=AlarmConfig(restarted_ville_threshold=1 / alpha)
    ),
    seed=42,
)

# Stable observations followed by a distribution shift
x_stream = np.vstack([
    rng.normal(size=(50, 2)),
    rng.normal(loc=3.0, size=(50, 2)),
])

for x_t in x_stream:
    state = monitor.update(x_t)
    if "restarted_ville" in state.triggered_alarms:
        print(f"Change alarm at step {state.evidence_step}")
        break

Under the sequential validity assumptions, the restarted Ville alarm at 1 / alpha controls the probability of ever crossing on one stream. It does not control FDR across multiple streams. See the sequential monitoring guide for the full guarantee scope and other alarm statistics.

Choose a workflow

Goal Start with
Calibrate and select anomalies in a batch Split and select(...)
Set a per-inlier raw-score false-alarm target fpr_bounds(...) and FPR certificates
Aggregate evidence across random splits DerandomizedSplits and e-BH
Monitor a stream for change Exchangeability martingales
Reuse more data for fitting and calibration CrossValidation or JackknifeBootstrap
Account for covariate shift Weighted conformal inference
Calibrate groups with different normal behavior Group-conditional (Mondrian) p-values
Certify realized FDP across p-value thresholds fdp_bounds(...) and FDP certificates
Bring a custom or third-party detector Detector compatibility

Statistical scope

nonconform calibrates detector scores; it cannot make an unsuitable detector or mismatched calibration set valid. Spatial or temporal dependence must be handled explicitly before applying standard exchangeability-based claims. See the guides to FDR control and sequential monitoring before relying on error-control statements in a new application.

Citation

If you use nonconform in academic work, please cite the accompanying paper:

@inproceedings{Hennhoefer2026,
author = {Hennhöfer, Oliver and Kirsch, Maximilian and Preisach, Christine},
booktitle = {Proceedings of the Fifteenth Symposium on Conformal and Probabilistic Prediction with Applications},
pages = {613--632},
publisher = {PMLR},
title = {{Conformal Anomaly Detection in Python: Moving Beyond Heuristic Thresholds with nonconform}},
url = {https://proceedings.mlr.press/v329/hennhofer26a.html},
year = {2026}
}

Project

Read the documentation, browse the changelog, or report a problem in the issue tracker. Contributions are welcome; start with the contributing guide. nonconform is distributed under the BSD 3-Clause License.


Funding acknowledgement

Metadata

Release files for nonconform 2.0.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for nonconform 2.0.1
File Size Uploaded Provenance Actions
nonconform-2.0.1.tar.gz 94.4 kB With provenance Details for nonconform-2.0.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for nonconform 2.0.1
File Interpreter ABI Platform Actions
nonconform-2.0.1-py3-none-any.whl Python 3 none any Details for nonconform-2.0.1-py3-none-any.whl

Total release size: 207.0 kB

Release files / nonconform-2.0.1.tar.gz

Download URL nonconform-2.0.1.tar.gz
Size 94.4 kB
Tags Source
SHA-256 checksum
How to use checksums
214a0d5c42f65130d9d8749f3450008d45fe4cb6d3a035f66a7a768ddd18b242
BLAKE2b-256 checksum
How to use checksums
f463e4dc713035f934acfdb7cf8495527e9fb0d38ddcd1630aadca67571f6307
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 1, 2026.

Transparency log

Release files / nonconform-2.0.1-py3-none-any.whl

Download URL nonconform-2.0.1-py3-none-any.whl
Size 112.6 kB
Tags Python 3
SHA-256 checksum
How to use checksums
6b7fd1951a7ed174e430ff62cb2bb5c0a59b166bacf85f12c6b3deb76d171a27
BLAKE2b-256 checksum
How to use checksums
9ad9564d554a60e3f10c36164473aa7ccb1dfdd787a11d011587e819a18d13e9
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 1, 2026.

Transparency log

Security for nonconform 2.0.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for nonconform-2.0.1.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 1, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

2.0.1 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page