rashomon-py

Does your conclusion survive every equally-good model?

Many parameter vectors fit a training set about as well as the one your solver returned. If some of them reverse your conclusion (a coefficient's sign, a feature ranking, a decision for one row), the conclusion depends on which optimum the solver landed on, not on the data. rashomon-py checks this for a fitted scikit-learn logistic or linear regression.

from sklearn.linear_model import LogisticRegression
from rashomon import audit

model = LogisticRegression().fit(X, y)       # your existing model (Pipelines work too)
report = audit(model, X, y)                  # pandas in -> feature names out

print(report.summary())
report.plot()
report.flipped                               # bool mask: rows whose prediction can flip
report.coefficients                          # DataFrame: estimate, low, high, sign_stable

Output for the ten "mean" features of the Wisconsin breast-cancer data with the default LogisticRegression():

Stability audit: LogisticRegression (classification), n=569, 10 features
Equally-good models: training log-loss within 0.01704 of the optimum 0.1435
  (one standard error of the 5-fold cross-validated log-loss, whose mean is 0.1458; models closer than this cannot be told apart by CV)
Method: hit-and-run sampling of the exact set (with ellipsoid proposals); 2,000 models, min ESS = 218 -> reliable

Predictions (flip test exact; disagreement over the sampled models)
  Flip under some equally-good model:           17.4%  (99 of 569)
  Worst single-model disagreement with yours:    5.1%

Coefficients (exact range across all equally-good models)
  feature                  estimate        low       high  sign
  mean radius               -0.9976     -4.634      2.621  UNSTABLE
  mean texture               -1.399     -2.527    -0.4664  stable
  mean perimeter            -0.9124     -4.613      2.775  UNSTABLE
  mean area                  -1.298     -5.061      2.433  UNSTABLE
  ...
Sign stable: 1 of 10 features.

The model is 95% accurate. Still, one diagnosis in six is reversed by some model that cross-validation cannot distinguish from it, and only the coefficient on texture keeps its sign across all of them. Radius, perimeter and area are near-duplicates, so the data fixes their combined effect but not how to split it between them. A claim like "tumour radius lowers the odds" would not survive.

Install

pip install rashomon-py          # Python 3.9+; depends on numpy, scipy, scikit-learn, pandas, matplotlib

How this differs from a confidence interval

A confidence interval or bootstrap asks how much the estimate would move under a new sample from the population. rashomon-py asks how many different models fit the data you have about equally well, and whether they agree with yours. The first is sampling uncertainty; the second is model multiplicity (Breiman's "Rashomon effect"). A coefficient can have a narrow confidence interval and still change sign across equally-good models when features are collinear, because the loss is nearly flat along the collinear directions. See Bootstrap, Rashomon, and Bayesian intervals for a worked comparison.

What "equally good" means

The tolerance sets how much worse than optimal a model may be and still count. audit() accepts four forms and prints the one in use:

The CV default is permissive, so expect larger sets than with a 1% rule. Results at two or three tolerances say more than any single one; see Choosing the tolerance.

What the report contains

Coefficient ranges and the flip test are exact: each coefficient range is the solution of a convex program over the true set, and each row not already flipped by a sampled model is settled by one more convex program (automatic when the problem is small enough: n · d² ≤ 5·10⁶ for ranges, n_undecided · n · d² ≤ 10⁹ for flips; force with exact_ranges=True / exact_flips=True). The disagreement figure and the prediction ranges come from models sampled from the exact set, so they are lower bounds that tighten as n_samples grows. The report states the effective sample size and whether it is reliable.

Validation

For an unpenalized logistic regression, the range of a coefficient over the Rashomon set with tolerance χ²₁(0.95)/(2n) is by definition its 95% profile-likelihood confidence interval, so the exact-range machinery can be checked against a standard statistical result. On the UCLA graduate-admissions data (admit ~ gre + gpa + rank, n = 400), coef_extremes() reproduces all twelve bounds of R's confint() output to within 1e-4 (R's own interpolation precision) and the MLE to 5e-7; the test also agrees with an independent profile root-finder to 2e-6. A second test grids a two-dimensional Rashomon set and checks the exact ranges, the exact flip test and the sampler's moments against brute force. See tests/test_validation.py.

Supported models

LogisticRegression / LogisticRegressionCV (binary; L2 or unpenalized; with or without class_weight), Ridge / RidgeCV, LinearRegression, and a Pipeline whose last step is one of these. If the model was fitted with sample_weight, pass the same weights to audit(..., sample_weight=w). The regularization strength, the unpenalized intercept and the row weights are converted exactly, and the audit checks that your fitted coefficients sit at the optimum of the reconstructed objective. A mismatch is reported.

Not supported: multiclass (planned), L1 and elastic-net penalties (the level set of a non-smooth objective is not a convex set of the same kind), trees, neural networks. The method needs a convex, twice-differentiable loss, so other L2-penalized GLMs (Poisson, multinomial) are possible extensions.

The method works in small to moderate dimension. Hit-and-run sampling mixes well up to a few dozen features. Above about 60 features the report will show a low effective sample size; reduce the dimension or sample longer. Exact coefficient ranges do not depend on sampling.

Expert API

RashomonSet gives direct access to ε calibration (percent_loss, LR_alpha, absolute), the membership oracle, hit-and-run sampling (with optional ellipsoid proposals), the Hessian-ellipsoid approximation, functional_range / coef_extremes, model class reliance, Shapley-VIC, and bootstrap and Bayesian comparisons. RashomonSet.from_sklearn(model, X, y, ...) builds one from a fitted model. RashomonSet(C=...) is not scikit-learn's C: it is 1/λ for the mean-loss objective, and scikit-learn's C equals C/n. audit and from_sklearn do the conversion.

from rashomon import RashomonSet
rs = RashomonSet.from_sklearn(model, X, y, epsilon=0.02, random_state=0)
rs.functional_range(x_row)          # exact prediction range for one row, on the logit scale
rs.sample_hitandrun(2000, ellipsoid_mix=0.5)

Documentation

• Quickstart
• When to use this
• Choosing the tolerance
• Interpreting instability
• Bootstrap, Rashomon, and Bayesian intervals
• Ellipsoid approximation vs sampling
• Tutorial: when equally-good models disagree
• Evaluation on real datasets
• API reference

References

• Breiman, L. (2001). Statistical modeling: The two cultures. Statistical Science, 16(3), 199-231.
• Fisher, A., Rudin, C., & Dominici, F. (2019). All models are wrong, but many are useful. JMLR, 20(177), 1-81.
• Marx, C., Calmon, F., & Ustun, B. (2020). Predictive multiplicity in classification. ICML.
• Dong, J., & Rudin, C. (2020). Exploring the cloud of variable importance for the set of all good models. Nature Machine Intelligence, 2, 810-824.
• Coker, B., Rudin, C., & King, G. (2021). A theory of statistical inference for ensuring the robustness of scientific results. Management Science, 67(10), 6174-6197.
• Rudin, C. (2019). Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1, 206-215.
• Semenova, L., Rudin, C., & Parr, R. (2022). On the existence of simpler machine learning models. FAccT.

License

MIT