RFE-cross-validation

Recursive Feature Elimination run inside every cross-validation fold — so you can see not just how many features to keep, but which ones, and how stable that choice is.

Motivation

scikit-learn gives you two related but incomplete tools:

• RFECV cross-validates the number of features to keep, but doesn't tell you how stable each individual feature's importance is.
• RFE ranks the features, but on a single fit — with no notion of variability.

rfe_cv fills the gap: it runs RFE separately in each CV fold and aggregates the per-fold rankings. A feature that is consistently ranked first across folds is a robust choice; one whose rank swings wildly is not. That fold-to-fold variability is what the shaded band communicates.

Figure: mean RFE rank across folds for each feature (1 = most important). The band shows the fold-to-fold variability of the rank.

How it works

1. Ranking (Figure 1). For each CV fold, fit RFE and record the ranking of every feature. Aggregate across folds into a mean rank (± a band).
2. Scoring (Figure 2). Order features by mean rank, then evaluate a cross-validated scoring metric using the top n features, for n from 1 up to max_features. This is the classic "score vs. number of features" curve, but built from the cross-validated ranking.

Install

pip install rfe-cv

Requires Python ≥ 3.9, plus numpy, pandas, scipy, matplotlib and scikit-learn (≥ 1.0), which pip installs for you.

Quick start

import pandas as pd
from sklearn.ensemble import RandomForestRegressor
from rfe_cv import rfe_cv

# df has feature columns plus a target column 'y'
features = ['x0', 'x1', 'x2', 'x3']

res = rfe_cv(df, features, 'y', RandomForestRegressor(),
             cv=5, scoring='r2')

res['rank_mean']          # mean RFE rank per feature (1 = best)
res['rank_std']           # rank std across folds
res['scores_mean']        # CV score using the n best features
res['selected_features']  # feature names selected at each step
res['fig1'], res['fig2']  # the two matplotlib figures

See example.py for regression, classification, comparing two models on the same axes, and max_features.

Parameters

Returns a dict with keys features, rank_mean, rank_std, scores_mean, scores_std, selected_features, fig1, fig2.

The shaded band (band)

Both figures draw a band around each curve, built from the standard deviation across folds. What that band means is your choice — and switching costs nothing extra, since the std is already computed:

⚠️ With few folds, the std estimate s is itself noisy — its relative error is about 1 / √(2(n−1)), roughly 35% at cv=5. Read narrow bands with caution; increasing cv is the cheapest way to tighten them reliably.

Comparing models on the same figures

Pass the figures from the first call into the next, and keep show=False until the final model:

res = rfe_cv(df, features, 'y', model_a, model_label='A', show=False)
rfe_cv(df, features, 'y', model_b, model_label='B',
       figs=[res['fig1'], res['fig2']], show=True)

Development

git clone https://github.com/attilalr/RFE-cross-validation.git
cd RFE-cross-validation
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
pytest

Releases are built and uploaded by the publish GitHub Actions workflow using PyPI Trusted Publishing — no API tokens are stored in the repository. Publishing a GitHub release uploads to PyPI; running the workflow manually with target: testpypi uploads to TestPyPI.

Roadmap

• Separate module file and example script
• Return structured results (a dict)
• Figure parameters (figsize, overlay via figs)
• English-only comments
• Return figures instead of only plotting
• Input type checks
• Consistent, leak-free scaling between selection and scoring (Pipeline)
• Choice of what the band represents (band)
• Packaged and published on PyPI
• Richer example / notebook walkthrough