ModelDiff-ML

Your candidate model improved AUC from 0.84 to 0.85. But did it get better everywhere?

ModelDiff-ML compares two machine-learning models at the observation level and automatically discovers interpretable, statistically validated regions where the candidate behaves differently from the baseline. It is a small Python library built around NumPy, pandas, and SciPy—not an MLOps platform.

ordinary evaluation:  "The average metric improved."
ModelDiff-ML:          "It improved overall, but regressed for high-x1, low-x2 observations."

Installation

ModelDiff-ML requires Python 3.11 or newer. The distribution is named modeldiff-ml; its Python import package remains the concise modeldiff.

pip install modeldiff-ml

For local development:

python -m venv .venv
. .venv/bin/activate
pip install -e '.[dev]'
pytest

Quickstart

ModelDiff-ML supports fitted sklearn-style models:

import modeldiff

report = modeldiff.compare(
    baseline=old_model,
    candidate=new_model,
    X=X_test,
    y=y_test,
    task="binary_classification",
    metric="log_loss",
    random_state=42,
)

print(report.summary())
report.to_json("modeldiff-ml.json")
report.to_html("modeldiff-ml.html")

Binary classifiers must expose predict_proba(X) with two columns. ModelDiff-ML deliberately does not turn an estimator's uncalibrated decision_function into a probability. Regression estimators need only predict(X).

Raw predictions

The framework is model-agnostic when predictions are already available:

from modeldiff import ModelDiffConfig, compare_predictions

config = ModelDiffConfig(
    max_depth=3,
    min_region_fraction=0.02,
    n_bootstrap=1000,
    n_permutations=2000,
    random_state=42,
)

report = compare_predictions(
    baseline_predictions=old_probabilities,
    candidate_predictions=new_probabilities,
    X=X_test,
    y=y_test,
    task="binary_classification",
    metric="log_loss",
    config=config,
)

significant_regressions = report.regressions(significant_only=True)

X may be a pandas DataFrame or two-dimensional NumPy array. DataFrames produce more readable conditions. Arrays receive the names x0, x1, and so on.

Leakage-aware validation splits

Random splitting remains the default. For repeated entities or ordered data, select a strategy that keeps related observations out of both sides of the discovery/validation boundary:

# All rows for a customer remain in one partition.
group_report = compare_predictions(
    old_predictions,
    new_predictions,
    X,
    y,
    task="regression",
    metric="mae",
    validation_strategy="group",
    groups=customer_ids,
)

# Earlier timestamps are used for discovery and later timestamps for validation.
time_report = compare_predictions(
    old_predictions,
    new_predictions,
    X,
    y,
    task="regression",
    metric="mae",
    validation_strategy="time",
    time_values=event_timestamps,
)

# Use an existing partition. Integer indices and boolean masks are both accepted.
explicit_report = compare_predictions(
    old_predictions,
    new_predictions,
    X,
    y,
    task="regression",
    metric="mae",
    validation_strategy="explicit",
    discovery_indices=development_mask,
    validation_indices=holdout_mask,
)

Group and time boundaries are chosen as close as possible to discovery_fraction without splitting a group or equal timestamp. Explicit partitions must assign every observation exactly once. Invalid or overlapping partitions raise ValueError; small or substantially unbalanced partitions add a report warning. report.split and serialized reports contain the strategy, actual fraction, counts, and strategy-specific audit metadata.

Interpreting delta

ModelDiff-ML uses one convention everywhere:

delta_i = candidate_loss_i - baseline_loss_i

• Positive delta: the candidate is worse.
• Negative delta: the candidate is better.
• Zero: equal loss.

The supported classification losses are log loss, Brier loss ("brier"), and accuracy loss ("accuracy"). Classification reports also include accuracy, log loss, Brier score, and rank-based ROC AUC. Regression supports squared error ("squared_error" or "mse"), RMSE ("rmse"), and absolute error ("absolute_error" or "mae").

For RMSE, the overall metric is square-rooted after aggregation; region discovery still operates on observation-level squared error. For accuracy, the primary loss is misclassification rate while the aggregate metrics table also reports conventional accuracy.

How region discovery works

1. ModelDiff-ML computes a loss difference for every observation.
2. It deterministically divides the data into discovery and validation partitions.
3. Numeric candidate conditions come from empirical quantiles. Categorical candidates use frequent equality and inequality conditions.
4. A from-scratch beam search builds conjunctions up to max_depth, scoring each as abs(mean_delta) * sqrt(region_fraction) (with sign-aware variants for each direction).
5. Highly overlapping regions are removed using Jaccard similarity.
6. The exact selected conditions are applied unchanged to the validation partition.
7. Validation effects receive percentile bootstrap confidence intervals and paired sign-flip p-values. Benjamini–Hochberg correction produces q-values across all tested regions.

This separation matters: searching for a threshold and reporting a p-value from those same rows would understate selection bias. ModelDiff-ML's independent holdout is a practical first defense.

Classification example

examples/classification_example.py creates a candidate that is better overall but deliberately worse when x1 is high and x2 is low:

python examples/classification_example.py

This is the central ModelDiff-ML use case: aggregate improvement can hide a meaningful local regression. A generated standalone report is available at examples/classification_report.html.

Regression example

examples/regression_example.py compares two arrays of regression predictions and discovers a planted high-error slice.

CLI and CI

The CLI reads precomputed predictions from CSV. Unless excluded, columns other than the target and prediction columns are features:

modeldiff-ml compare \
  --data validation.csv \
  --target y \
  --baseline-pred old_prediction \
  --candidate-pred new_prediction \
  --task binary_classification \
  --metric log_loss \
  --html report.html \
  --json report.json

The CLI supports the same split strategies with --validation-strategy and a control column:

modeldiff-ml compare ... --validation-strategy group --group-column customer_id
modeldiff-ml compare ... --validation-strategy time --time-column event_time
modeldiff-ml compare ... --validation-strategy explicit --split-column partition

An explicit split column must contain only discovery and validation. Split-control columns are automatically excluded from the region-search features.

Use --exclude COLUMN repeatedly to omit non-feature columns. CI-oriented checks return exit code 1:

modeldiff-ml compare ... --fail-on-significant-regression
modeldiff-ml compare ... --max-overall-regression 0.01

The JSON output has schema version 1.0, contains no model objects, and is suitable for CI artifact processing.

Configuration

ModelDiffConfig controls the discovery fraction, quantile grid, beam width, maximum depth, minimum and maximum support, overlap threshold, resampling counts, confidence and significance levels, classification threshold, numerical epsilon, and random seed. Defaults favor understandable output and avoid tiny slices. Missing feature values are never imputed; they evaluate as false for V1 conditions.

Statistical scope and limitations

Discovered subgroups are exploratory. Independent validation and FDR correction reduce false discoveries, but do not eliminate them. Results can still be unstable with small data, distribution shift, correlated repeated observations, or extensive repeated analyses by a human. A fresh final test set is appropriate before high-stakes deployment decisions.

V1 supports binary classification and regression only. It does not include multiclass evaluation, causal claims, fairness-specific analysis, distributed execution, SHAP, LLM evaluation, monitoring infrastructure, model registries, dashboards, or automated remediation.

Roadmap

• Multiclass observation-level losses
• Cross-validated discovery and stability summaries
• Pluggable, documented region scoring policies
• Richer calibrated-score adapters

License

MIT