imodels 3.0.2

Python package for concise, transparent, and accurate predictive modeling.
All sklearn-compatible and easy to use.
Check out our new packages! Interpretability in text: imodelsX, interpretability tools for tabular data with agents: agentic-imodels

📚 docs • 📖 demo notebooks

Modern machine-learning models are increasingly complex, often making them difficult to interpret. This package provides a simple interface for fitting and using state-of-the-art interpretable models, all compatible with scikit-learn. These models can often replace black-box models (e.g. random forests) with simpler models (e.g. rule lists) while improving interpretability and computational efficiency, all without sacrificing predictive accuracy! Simply import a classifier or regressor and use the fit and predict methods, same as standard scikit-learn models.

from imodels import get_clean_dataset, HSTreeClassifierCV # import any imodels model here
from sklearn.model_selection import train_test_split

# prepare data (a sample clinical dataset)
X, y, feature_names = get_clean_dataset('csi_pecarn_pred')
X_train, X_test, y_train, y_test = train_test_split(
    X, y, random_state=42)

# fit the model
model = HSTreeClassifierCV(max_leaf_nodes=4)  # initialize a tree model and specify only 4 leaf nodes
model.fit(X_train, y_train, feature_names=feature_names)   # fit model
preds = model.predict(X_test) # discrete predictions: shape is (n_test, 1)
preds_proba = model.predict_proba(X_test) # predicted probabilities: shape is (n_test, n_classes)
print(model) # print the model

------------------------------
Decision Tree with Hierarchical Shrinkage
Prediction is made by looking at the value in the appropriate leaf of the tree
------------------------------
|--- FocalNeuroFindings2 <= 0.50
|   |--- HighriskDiving <= 0.50
|   |   |--- Torticollis2 <= 0.50
|   |   |   |--- value: [0.10]
|   |   |--- Torticollis2 >  0.50
|   |   |   |--- value: [0.30]
|   |--- HighriskDiving >  0.50
|   |   |--- value: [0.68]
|--- FocalNeuroFindings2 >  0.50
|   |--- value: [0.42]

Installation

Install with pip install imodels (see here for help).

Supported models

🗂️ Docs   📄 Research paper   🔗 Reference code implementation

Demo notebooks

Demos are contained in the notebooks folder

What's the difference between the models?

The final form of the above models takes one of the following forms, which aim to be simultaneously simple to understand and highly predictive:

Rule set	Rule list	Rule tree	Algebraic models

Different models and algorithms vary not only in their final form but also in different choices made during modeling, such as how they generate, select, and postprocess rules:

Rule candidate generation	Rule selection	Rule postprocessing

model = FIGSClassifier().fit(X, y, categorical_features=['pet', 'city'])
model.predict(X)

import dtreeviz
from imodels import FIGSClassifier, shadow_tree

model = FIGSClassifier(max_rules=6).fit(X, y)
viz = dtreeviz.trees.DTreeVizAPI(shadow_tree(model, X, y))
viz.view()

from imodels import FIGSClassifier

model = FIGSClassifier(max_rules=4).fit(X_train, y_train, feature_names=feature_names)
model.get_rules()

                                               rule  prediction  tree
0                        FocalNeuroFindings2 <= 0.5       0.117     0
1                         FocalNeuroFindings2 > 0.5       0.427     0
2                             HighriskDiving <= 0.5      -0.008     1
3                              HighriskDiving > 0.5       0.550     1
4  PainNeck2 <= 0.5 and AlteredMentalStatus2 <= 0.5      -0.083     2
5   PainNeck2 <= 0.5 and AlteredMentalStatus2 > 0.5       0.048     2
6                                   PainNeck2 > 0.5       0.058     2

Extras

Our favorite methods

After developing and playing with imodels, we developed a few new models to overcome limitations of existing interpretable models.

FIGS: Fast interpretable greedy-tree sums

📄 Paper, 🔗 Post, 📌 Citation

Fast Interpretable Greedy-Tree Sums (FIGS) is an algorithm for fitting concise rule-based models. Specifically, FIGS generalizes CART to simultaneously grow a flexible number of trees in a summation. The total number of splits across all the trees can be restricted by a pre-specified threshold, keeping the model interpretable. Experiments across a wide array of real-world datasets show that FIGS achieves state-of-the-art prediction performance when restricted to just a few splits (e.g. less than 20).

Example FIGS model. FIGS learns a sum of trees with a flexible number of trees; to make its prediction, it sums the result from each tree.

GPGam: additive Gaussian processes over binned features

🔗 Post, 🗂️ API

GPGam fits a generalized additive model with pairwise interactions. Every shape function in it is a Gaussian process over the quantile bins of its feature.

Binning is what makes this practical. Once the features are binned, the exact GP marginal likelihood depends on the data only through the bin co-occurrence counts Z'Z, the bin sums Z'y, and y'y. One pass over the data computes those, and every optimizer step after that costs the same whether the data had a thousand rows or a hundred thousand.

That one likelihood settles every choice a GAM usually leaves to the user. How smooth each shape function should be follows from a mixture of two kernels whose lengthscales are learned and shared across features. Features that explain nothing get amplitudes near zero and drop out, and a hierarchical prior shrinks each kernel's amplitudes toward their centre across features, so that choice is stable across splits. Interactions are screened on the residual; up to 48 are fit jointly, and above a thousand rows the rest, up to five per feature, are backfit on the joint model's residual with each surface's grid resolution picked by comparing likelihoods. Nothing is set by cross-validation and nothing is random, so two fits on the same data give the same model.

from imodels import GPGamRegressor
model = GPGamRegressor().fit(X_train, y_train)

grid, values, std = model.shape_function(0, return_std=True)   # feature 0's curve, with its band
model.kernel_weights(0)                                        # the smooth/rough split the likelihood chose
model.interaction_terms()                                      # the pairs it chose to include

Because the model is a Gaussian process, each curve arrives with a posterior band, so you can see which parts of a shape function the data actually pins down. The post walks through a model fit to California housing, curve by curve and interaction by interaction.

On the development suite (65 datasets, at most 1,000 rows each) GPGam is the strongest interpretable model and second overall to TabPFN. On two held-out suites with every dataset shared with the development suite removed, TabArena and OpenML-CTR23, it is again the strongest interpretable model: first of eleven by mean rank on TabArena and second to TabPFN on CTR23, with a geometric-mean RMSE ratio of 0.97 against explainable boosting machines and wins on 22 of the 35 datasets. Every model in those comparisons was refit on identical preprocessing and the same split.

FastSmallTree: provably optimal small decision trees

🔗 Post, 🗂️ API

FastSmallTree fits the decision tree that minimizes misclassification rate plus a penalty per leaf, over every tree on the binarized features, and certifies that no other tree scores better. This is the objective optimal-tree packages such as GOSDT and STreeD solve. FastSmallTree came out of an autoresearch loop, and most of its speed comes from compiling the whole branch-and-bound search with numba, along with a few tighter bounds, each proved admissible in the post.

from imodels import FastSmallTreeClassifier
model = FastSmallTreeClassifier(regularization=0.05).fit(X_train, y_train)

model.optimal_      # True when the search finished: no other tree on these features scores better
model.objective_    # the proven minimum of error + 0.05 * leaves
model.estimator_    # the tree as an ordinary sklearn DecisionTreeClassifier (plot_tree, dtreeviz, ...)

regularization is the only hyperparameter: larger values give smaller trees. If the time limit is reached first, optimal_ is False and the model warns. The search needs numba (pip install numba); it compiles once per machine, in about 20 seconds, and is cached after that.

On held-out benchmarks built from TabArena, FastSmallTree matches the trees of existing optimal-tree packages while running faster, sometimes by more than 20×, and on the full-size datasets GOSDT and STreeD return no tree for several of them, mostly because they run out of memory. The post has the comparison and the proofs.

Hierarchical shrinkage: post-hoc regularization for tree-based methods

📄 Paper (ICML 2022), 🔗 Post, 📌 Citation

Hierarchical shrinkage is an extremely fast post-hoc regularization method which works on any decision tree (or tree-based ensemble, such as Random Forest). It does not modify the tree structure, and instead regularizes the tree by shrinking the prediction over each node towards the sample means of its ancestors (using a single regularization parameter). Experiments over a wide variety of datasets show that hierarchical shrinkage substantially increases the predictive performance of individual decision trees and decision-tree ensembles.

HS Example. HS applies post-hoc regularization to any decision tree by shrinking each node towards its parent.

MDI+: Flexible Tree-Based Feature Importance

📄 Paper, 🔗 Post, 📌 Citation

MDI+ is a novel feature importance framework, which generalizes the popular mean decrease in impurity (MDI) importance score for random forests. At its core, MDI+ expands upon a recently discovered connection between linear regression and decision trees. In doing so, MDI+ enables practitioners to (1) tailor the feature importance computation to the data/problem structure and (2) incorporate additional features or knowledge to mitigate known biases of decision trees. In both real data case studies and extensive real-data-inspired simulations, MDI+ outperforms commonly used feature importance measures (e.g., MDI, permutation-based scores, and TreeSHAP) by substantional margins.

References

Please cite the package if you use it in an academic work :)

@software{
	singh2021imodels,
	title        = {imodels: a python package for fitting interpretable models},
	journal      = {Journal of Open Source Software},
	publisher    = {The Open Journal},
	year         = {2021},
	author       = {Singh, Chandan and Nasseri, Keyan and Tan, Yan Shuo and Tang, Tiffany and Yu, Bin},
	volume       = {6},
	number       = {61},
	pages        = {3192},
	doi          = {10.21105/joss.03192},
	url          = {https://doi.org/10.21105/joss.03192},
}