rsoselect

Variable selection with the ridge selection operator (RSO) for linear, partially linear, and generalized partially linear models.

Ridge regression with an individual penalty per coefficient can select variables: if the penalty on a coefficient goes to infinity, that coefficient is exactly zero. RSO puts a budget on the inverse penalties and lets the data decide where to spend it (Wu, 2021). The same idea, applied to the inverse bandwidths of a kernel smoother, selects nonparametric predictors as well, so linear and nonlinear predictors are chosen in one optimization problem (Lu et al., 2024; Lu, Hu and Wu, 2025).

Installation

pip install rsoselect                # core
pip install "rsoselect[sklearn]"     # with scikit-learn integration

From source:

git clone https://github.com/luyouhanzju/rsoselect.git
cd rsoselect
pip install -e .

Requires Python 3.9+, NumPy and SciPy. scikit-learn is optional (needed only for criterion="cv" and for using RSO inside sklearn pipelines).

Quick start

from rsoselect import RSO, PLM, GPLM

# Linear model: drop-in replacement for sklearn's Ridge
m = RSO().fit(X, y)
m.support_          # selected predictors (boolean mask)
m.coef_, m.intercept_
m.predict(X_new)

# Partially linear model: X enters linearly, Z nonparametrically
m = PLM().fit(X, Z, y)
m.support_x_, m.support_z_
m.predict(X_new, Z_new)

# Generalized partially linear model (binomial, poisson or gaussian)
m = GPLM(family="binomial").fit(X, Z, y)
m.predict(X_new, Z_new)        # probabilities
m.predict_eta(X_new, Z_new)    # linear predictor

How it works

For a budget τ, RSO solves

min over gamma:   RSS( beta(gamma) )
subject to:       gamma_k >= 0,  sum_k gamma_k = tau

beta(gamma) = argmin_b  (1/n) ||y - X b||^2 + sum_k b_k^2 / gamma_k

so γ_k = 0 gives β_k = 0 exactly. In the PLM and GPLM the nonparametric part is a kernel smoother with inverse bandwidths λ_j under a second budget ∑λ_j = τ₂; λ_j = 0 removes Z_j. The GPLM is fit by local scoring, and (γ, λ) minimize the negative log-likelihood. All three models are optimized by the same modified coordinate descent (rsoselect.mcd).

The budgets are tuned over a grid:

For time series, use RSO(criterion="cv", cv=TimeSeriesSplit(5)).

criterion="loo" is the closed-form leave-one-out error (PRESS), e_i / (1 - h_ii) with H the hat matrix of the reported fit. It costs one fit per budget and needs no scikit-learn. It holds gamma fixed at the full-sample estimate, so it is optimistic relative to exact leave-one-out, which is RSO(criterion="cv", cv=LeaveOneOut()) (one fit per observation and budget). Like AIC, LOO targets prediction error and may keep a few more variables than BIC.

Stochastic discount factors (MSRR)

Ridge estimates of SDF weights minimize mean((1 - F @ lam)**2) + penalty, where F holds the factor returns (T x P). That is a regression of a vector of ones on F with no intercept, so RSO applies directly, and "loo" gives leave-one-period-out selection of the budget:

P = F_train.shape[1]
z = 10.0 ** np.arange(-10, 4)      # ridge penalties, e.g. Kelly et al.'s grid
m = RSO(criterion="loo", taus=P / z, fit_intercept=False, standardize=False,
        refit=False)
m.fit(F_train, np.ones(len(F_train)))
lam = m.coef_                      # SDF / portfolio weights, zero for dropped factors
r_sdf = F_test @ lam               # out-of-sample SDF portfolio returns

Keep fit_intercept=False: an intercept would absorb the vector of ones and set every weight to zero. With uniform gamma, the RSO penalty is (P / tau) * lam @ lam, so taus = P / z puts the budgets on the same scale as a ridge penalty z (the default taus assume standardized columns and are too small for raw returns). P / m.tau_ is the ridge-equivalent penalty of the selected budget.

Main parameters

RSO

PLM / GPLM

Shared optimizer settings: grid=(0, .2, .5, .8, 1), tol=1e-4, standardize=True. After fitting, path_ holds every budget on the grid with its gamma, lambda, selected sets and criterion values, which is enough to draw solution paths.

Reproducing the published results

The defaults implement the estimators as written in the papers. Setting r_compat=True reproduces the original R implementation used for the numerical results in the papers; the test suite checks this line by line against ports of the R functions. The two modes differ in:

Tests

pip install -e ".[test]"
pytest

References

If you use this package, please cite:

• Wu, Y. (2021). Can't ridge regression perform variable selection? Technometrics, 63(2), 263–271. https://doi.org/10.1080/00401706.2020.1791254
• Lu, Y., Dong, Y., Hu, J. and Wu, Y. (2024). A unified approach to variable selection for partially linear models. Journal of Computational and Graphical Statistics, 33(1). https://doi.org/10.1080/10618600.2023.2216254
• Lu, Y., Hu, J. and Wu, Y. (2025). A new approach to select linear and nonparametric predictors simultaneously for generalised partially linear models. Journal of Nonparametric Statistics, 37(4), 1298–1316. https://doi.org/10.1080/10485252.2025.2556412

@article{wu2021ridge,
  author  = {Wu, Yichao},
  title   = {Can't Ridge Regression Perform Variable Selection?},
  journal = {Technometrics},
  volume  = {63},
  number  = {2},
  pages   = {263--271},
  year    = {2021},
  doi     = {10.1080/00401706.2020.1791254}
}

@article{lu2024plm,
  author  = {Lu, Youhan and Dong, Yushen and Hu, Juan and Wu, Yichao},
  title   = {A Unified Approach to Variable Selection for Partially Linear Models},
  journal = {Journal of Computational and Graphical Statistics},
  volume  = {33},
  number  = {1},
  year    = {2024},
  doi     = {10.1080/10618600.2023.2216254}
}

@article{lu2025gplm,
  author  = {Lu, Youhan and Hu, Juan and Wu, Yichao},
  title   = {A New Approach to Select Linear and Nonparametric Predictors Simultaneously
             for Generalised Partially Linear Models},
  journal = {Journal of Nonparametric Statistics},
  volume  = {37},
  number  = {4},
  pages   = {1298--1316},
  year    = {2025},
  doi     = {10.1080/10485252.2025.2556412}
}

An R implementation of linear RSO by other authors is available on CRAN as RSO.

License

BSD 3-Clause. See LICENSE.