subselect 0.0.1

Subset-contrast unsupervised model selection for anomaly detection

subselect

Subset-contrast unsupervised model selection for anomaly detection.

Given a pool of trained anomaly-detection models with no ground-truth labels, pick the best one by measuring the contrast between each model's top-ranked points (predicted anomalies) and the rest of the data.

Why

In anomaly detection, practitioners must choose among many models (Isolation Forest, LOF, kNN, OCSVM, ...) with different hyperparameters. The standard way to evaluate models is supervised, using metrics like ROC-AUC that require labelled anomalies. In practice, labelled anomalies are rarely available; if they were, the problem would already be partly solved.

subselect selects a model from a pool without any labels. A good model concentrates real anomalies at the top of its score ranking, so its top subset looks distinct from the rest of the data in feature space; a bad model's top subset looks like ordinary data. Measuring that contrast with a density or distance metric gives an unsupervised ranking that correlates with the hidden supervised ROC-AUC ranking.

Install

pip install subselect

Requires Python 3.10+ with numpy, scipy, pandas, and scikit-learn.

Quickstart

import subselect as ss

# scores: (n_samples, n_models) anomaly score per (sample, model); higher = more anomalous
# X:      (n_samples, n_features) original feature matrix
# model_names: length-n_models identifiers aligned with the columns of scores
best = ss.Evaluator().select(scores=scores, X=X, model_names=model_names)
print("Picked:", model_names[best])

Evaluator() selects with a single default metric (Mahalanobis distance). Pass a list of metrics and ensemble=True to combine several and add their rank-aggregated ensemble (the strongest configuration in our benchmarks):

ev = ss.Evaluator(metrics=["mahalanobis_distance", "gmm_likelihood"], ensemble=True)
results = ev.evaluate(scores, X, model_names)
print(results.selected_per_metric)   # pick per metric, plus 'ensemble'

Shipped metrics

Five contrast metrics across the density and distance families:

name	family	direction
mahalanobis_distance	distance, global covariance	higher = better
knn_avg_distance	distance, local	higher = better
lof_score	local density (LOF)	higher = better
kde_log_likelihood	density, non-parametric	lower = better
gmm_likelihood	density, parametric	lower = better

ss.list_metrics() lists them and ss.metric_sets["core"] is all five.

Custom contrast metric

Three ways to add your own:

# 1. Subclass - most flexible
class MyMetric(ss.ContrastMetric):
    direction = +1                          # higher value -> better model
    kind = "per-point"

    def fit_reference(self, X_ref):         # fit on the complement (the reference)
        return self

    def score_subset(self, X_sub):
        return {"mean": ..., "std": ..., "median": ...}

# 2. Decorator - one function
@ss.contrast_metric(direction=+1, kind="per-point", name="my_metric")
def my_metric(subset_X, reference_X):
    return float(...)

# 3. Inline callable - quickest
ss.Evaluator(metrics=[(my_metric_callable, +1)]).select(scores, X, model_names)

Citation

See CITATION.cff. A paper describing the method is forthcoming.

License

MIT. See LICENSE.