imgad — Image anomaly detection with foundation models, classical detectors and Anomalib

A modular framework for image-level anomaly detection in the one-class setting (training uses normal images only):

images ──► FeatureExtractor ──► preprocessing ──► classical detector ──► anomaly score
          (CLIP, SigLIP, DINOv2…   (l2, PCA,         (kNN, Mahalanobis,     (+ calibrated threshold)
           any HF model)            standard…)        LOF, OCSVM, IF…)

It also wraps end-to-end Anomalib models (PatchCore, PaDiM, EfficientAD…) with pixel-level anomaly maps, and includes a benchmarking module with repeated runs, confidence intervals, statistical tests and automatic markdown reports.

Installation

pip install imgad                    # core
pip install "imgad[anomalib]"        # + Anomalib models (PatchCore, PaDiM, EfficientAD…)
pip install "imgad[report]"          # + figures in benchmark reports (matplotlib)
pip install "imgad[all]"             # + open_clip, pyod, pandas, anomalib, matplotlib, pytest

From source: git clone https://github.com/joheras/imgad && pip install -e "imgad[dev]". Python ≥ 3.10.

Quick start

from imgad import AnomalyDetectionPipeline, HFFeatureExtractor, build_detector, data
from imgad.evaluation import evaluate_scores

ds = data.mvtec_ad("/path/to/mvtec_anomaly_detection", "bottle")

pipe = AnomalyDetectionPipeline(
    extractor=HFFeatureExtractor("openai/clip-vit-base-patch16"),
    detector=build_detector("knn", k=1),
    preprocessing=["l2"],
    cache_dir=".feature_cache",          # caches features per image
)
pipe.fit(ds.train_paths)                 # normal images only
scores = pipe.score(ds.test_paths)       # higher = more anomalous
preds  = pipe.predict(ds.test_paths)     # 0/1 with the calibrated threshold
print(evaluate_scores(ds.test_labels, scores, preds))

pipe.save("bottle.joblib")               # saves detector + config (not the model weights)
pipe = AnomalyDetectionPipeline.load("bottle.joblib")

Inference with a trained model

The .joblib file stores the fitted detector, the preprocessing, the threshold, the extractor configuration and metadata (number of training images, date, threshold origin). Hugging Face weights are not stored: they are reloaded from the Hub (or the local HF cache) on load.

from imgad import AnomalyPredictor

predictor = AnomalyPredictor.load("bottle.joblib")   # device auto (cuda/mps/cpu); device="cpu" to force it
r = predictor.predict_one("new.png")                 # path, PIL.Image or numpy array
r.is_anomaly, r.score, r.normal_percentile

results = predictor.predict("folder/")               # folder, list of paths/images, or a mix
predictor.save_results(results, "preds.csv")         # .csv or .json
for r in predictor.predict_iter("huge_folder/", chunk_size=256): ...   # constant memory

Each Prediction contains:

Tune the threshold on a labelled validation set (no retraining):

pipe = AnomalyDetectionPipeline.load("bottle.joblib")
pipe.calibrate_threshold(val_paths, val_labels, method="f1")   # f1 | youden | fpr | tpr
pipe.save("bottle_cal.joblib")
# Normal images only: method="fpr", target_fpr=0.01  -> ≤1% false alarms

Or on the fly: AnomalyPredictor.load("bottle.joblib", threshold=0.02).

Shortcuts: the extractor can be an HF id ("facebook/dinov2-base") and the detector a name ("mahalanobis") or a dict ({"name": "knn", "k": 5}).

Feature extractors

HFFeatureExtractor(model_name, pooling="auto", layer=None, normalize=True, ...) works with any vision model on the Hub that loads with AutoModel:

pooling options: auto, projection, pooler, cls, mean (mean of patch tokens), cls+mean. layer selects intermediate layers (layer=-4, or layer=[6, 9, 12] to concatenate several), which often capture local defects better than the more semantic last layer.

For open_clip checkpoints (hf-hub:laion/..., EVA-CLIP, DataComp…): OpenCLIPFeatureExtractor("hf-hub:laion/CLIP-ViT-B-32-laion2B-s34B-b79K").

Detectors

All detectors share the same API (fit, score, predict) and the convention higher score = more anomalous.

Threshold: after fit, threshold_ is set to the threshold_quantile (0.95 by default) of out-of-sample scores of the normal data: leave-one-out for kNN/LOF and 5-fold cross-validation for the rest. This avoids the optimistic threshold of detectors that overfit the training set (KDE, OCSVM, PCA…). It can be disabled with det.calibration = "train".

Anomalib models (with anomaly maps)

AnomalibPipeline wraps any one-class Anomalib model (patchcore, padim, efficient_ad, fastflow, stfpm, reverse_distillation, cflow, draem…; imgad list shows them) with the same API as the feature pipelines, and adds pixel-level anomaly maps:

from imgad import AnomalibPipeline, evaluate_maps

pc = AnomalibPipeline("patchcore", backbone="wide_resnet50_2", layers=["layer2", "layer3"], image_size=256)
pc.fit(ds.train_paths)                              # normal images only
scores, maps = pc.score_and_maps(ds.test_paths)     # maps: list of arrays (model resolution)
evaluate_maps(ds.test_masks, maps)                  # pixel_auroc, pixel_aupr, pixel_f1_max, aupro
pc.predict_masks(maps)                              # binary masks (map > pixel_threshold_)
pc.save("bottle_pc.joblib")                         # loaded by AnomalyPredictor.load / imgad predict as usual

• Anomalib provides the models and the training loop; its post-processing (normalisation and threshold learnt on a labelled validation set) is disabled. Scores and maps are the raw model outputs (higher = more anomalous).
• Threshold: as for the detectors, the threshold_quantile (0.95) of the scores of unseen normal images: calibration_split=0.2 holds out that 20% of the training images (not used for training). calibration_split=0 uses training scores (optimistic, especially for PatchCore). The pixel threshold (pixel_threshold_) is the same quantile of the maximum of each held-out normal map; with validation masks it can be tuned with calibrate_pixel_threshold(images, masks) (pixel F1). calibrate_threshold works as in the other pipelines.
• Images are resized to image_size without cropping, so the maps cover the whole image. score_and_maps(..., size="original") returns them at each image's own size.
• The .joblib stores the configuration and the weights (e.g. the PatchCore memory bank): nothing needs to be downloaded to load it.
• Model kwargs are those of the Anomalib constructor (backbone, layers, coreset_sampling_ratio, pre_trained…). For models trained over several epochs: max_epochs or trainer_kwargs={"max_steps": ...}. EfficientAD trains with batch size 1 and downloads imagenette.
• One-class models only; zero/few-shot models (WinCLIP, AnomalyDINO…) are not supported.

Inference with maps:

predictor = AnomalyPredictor.load("bottle_pc.joblib")
res = predictor.predict("folder/", return_maps=True)   # r.anomaly_map (original size), r.anomalous_area
predictor.save_maps(res, "maps/")                      # <folder>_<name>_heatmap.png + _mask.png
predictor.save_results(res, "preds.csv")               # includes the heatmap and mask paths

Heatmaps use a fixed scale per model (blue = typical normal, middle of the scale = pixel threshold, red = far above it), so normal images look blue instead of being rescaled.

Preprocessing

List of steps applied before the detector (fitted on the normal data): "l2", "standard", "center", "pca:0.95" (explained variance) or "pca:64" (components), "pca_whiten:<n>", or any sklearn transformer.

Data

data.mvtec_ad(root, "screw")                     # MVTec AD / MVTec layout (converted VisA, etc.)
data.from_folders(train_normal, test_normal, [test_anom_1, test_anom_2])
data.one_vs_rest(root, normal_class="cat")       # semantic AD from a classification dataset

Each returns an AnomalyDataset with train_paths, test_paths, test_labels, test_types and, for MVTec, test_masks (path of the ground-truth mask in ground_truth/<type>/<name>_mask.png, or None for normal images).

Evaluation and benchmarks

evaluate_scores returns AUROC, AUPR, F1-max and FPR@95%TPR (and F1/TPR/FPR at the threshold if predictions are given). per_type_auroc breaks the AUROC down by defect type.

evaluate_maps(masks, maps) gives the localisation metrics: pixel-AUROC, pixel-AUPR, pixel-F1-max and AUPRO (FPR ≤ 0.3, the standard MVTec AD metric). Masks are resized to each map's resolution.

benchmark(extractors, detectors, datasets, preprocessings, anomalib_models=[...]) evaluates the full grid, extracting features once per (extractor, dataset), plus each Anomalib model per dataset (with pixel metrics when masks are available). In YAML, use the anomalib: key (see configs/mvtec_benchmark.yaml).

Benchmark reports

imgad benchmark --report DIR writes a markdown report with figures:

imgad benchmark --config configs/mvtec_benchmark.yaml --report report/ --seeds 0 1 2 --bootstrap 1000

report/
  report.md       # the report
  figures/*.png   # every figure referenced in the report
  results.csv     # one row per dataset × configuration × seed
  config.yaml     # configuration used

The report contains:

Figures need matplotlib (pip install "imgad[report]"); without it the report contains tables only. The summary and discussion are generated from the numbers with simple rules: treat them as a first draft to check and complete. To rebuild a report without rerunning the benchmark:

imgad benchmark --config cfg.yaml --scores-out scores.joblib
imgad report --scores scores.joblib --config cfg.yaml --out report/ --title "My benchmark"

From Python: imgad.report.generate_report(rows, datasets, "report/", config=cfg) with rows = benchmark(..., keep_scores=True).

Statistical analysis: repetitions, confidence intervals and tests

Repetitions. benchmark(..., seeds=[0, 1, 2, 3, 4], train_subsample=0.8) runs every configuration once per seed (column seed). The seed sets the detectors' random_state (unless the spec fixes it), the calibration folds, the Anomalib seed and the training subset (train_subsample: a fraction if float, a number of images if int). Without subsampling, deterministic detectors (kNN, Mahalanobis…) give the same result for every seed. Features are extracted once and reused across seeds.

Confidence intervals. There are two sources of uncertainty, measured separately:

Statistical tests (imgad.stats):

To compare across datasets, seeds are first averaged within each dataset, and each dataset (e.g. each MVTec category) is a block, following Demšar (2006). Wilcoxon-Holm usually has more power with few configurations, while Nemenyi is the classical post-hoc test after Friedman and the one drawn in the CD diagram. DeLong needs per-image scores (keep_scores=True).

from imgad import benchmark, stats

rows = benchmark(extractors, detectors, datasets, seeds=range(5), train_subsample=0.8,
                 bootstrap=1000, keep_scores=True, anomalib_models=["patchcore"])
stats.summarize_seeds(rows)                          # mean ± std per (dataset, config)
res = stats.compare_configs(rows, metric="auroc")    # also aupr, pixel_auroc, aupro, fpr@95tpr…
print(stats.report_markdown(res))                    # mean ranks, Friedman, CD, pairs with Holm
stats.plot_cd_diagram(res, "cd.png")
stats.delong_vs_baseline(rows, baseline=res["configs"][0])

From the CLI:

imgad benchmark --config configs/mvtec_benchmark.yaml --seeds 0 1 2 3 4 --train-subsample 0.8 \
                --bootstrap 1000 --out results.csv --scores-out scores.joblib --stats
imgad stats --results results.csv --metric auroc --cd-plot cd.png --out stats.md
imgad stats --scores scores.joblib --baseline "openai/clip-vit-base-patch16 | l2 | knn(k=1)"   # + DeLong

CLI

imgad list
imgad fit --model openai/clip-vit-base-patch16 --detector knn -p k=1 --prep l2 \
          --train data/bottle/train/good --out bottle.joblib

# inference: images and/or folders; output to stdout, .csv or .json
imgad predict --model bottle.joblib new.png data/bottle/test --out preds.csv
imgad predict --model bottle.joblib folder/ --only-anomalies --threshold 0.02 --device cpu

# re-tune the threshold on labelled validation data (f1 | youden | fpr | tpr)
imgad calibrate --model bottle.joblib --normal val/good --anomalous val/defects --method f1 --out bottle_cal.joblib
imgad info --model bottle_cal.joblib         # configuration, threshold and metadata

imgad benchmark --config configs/mvtec_benchmark.yaml --out results.csv --report report/

# Anomalib models: -p sets the model kwargs; predict saves heatmaps and masks
imgad fit --anomalib patchcore -p backbone=wide_resnet50_2 "layers=[layer2,layer3]" \
          --image-size 256 --train data/bottle/train/good --out bottle_pc.joblib
imgad predict --model bottle_pc.joblib data/bottle/test --maps-dir maps/ --out preds.csv

Extending

Registering your own detector or extractor makes it available by name in Python, YAML and the CLI (see examples/extending.py):

from imgad import BaseDetector, register_detector

@register_detector("my_detector")
class MyDetector(BaseDetector):
    def _fit(self, X): ...
    def _score(self, X): ...   # higher = more anomalous

Example results

benchmarks/mvtec_4categories/ contains a benchmark of 5 extractors × 11 detectors on 4 MVTec AD categories (report, results CSV and configuration).

Project layout

src/imgad/
  features/      base.py (interface, registry, cache) · huggingface.py · open_clip.py
  detectors/     base.py (interface, registry, calibration) · classical.py
  anomalib_backend.py (AnomalibPipeline) · maps.py (masks, heatmaps)
  preprocessing.py · pipeline.py · inference.py · data.py · evaluation.py · stats.py · report.py · cli.py
configs/         example benchmark YAML files
examples/        quickstart.py · inference.py · compare_models.py · extending.py · anomalib_models.py
benchmarks/      example benchmark results
tests/           pytest (offline: a tiny CLIP generated locally; untrained ResNet-18 for Anomalib)

Possible next steps

• Zero-/few-shot detection with text prompts (WinCLIP, AnomalyCLIP) using the text encoder.
• Native PatchCore on patch features of foundation models (CLIP/DINOv2), to compare with Anomalib's (CNN).
• Test-time augmentation (TTA) and multi-scale aggregation.
• Few-shot sweep (train_subsample = 1, 2, 4, 8… images) reusing the seeds and the tests in imgad.stats.

License

Distributed under the Apache 2.0 license.

Acknowledgements

This work is partially supported by Agencia de Desarrollo Económico de La Rioja [ADER 2022-I-IDI-00015].