ReHLine-Python: Efficient Solver for ERM with PLQ Loss and Linear Constraints

Fast, scalable, and scikit-learn compatible optimization for machine learning

ReHLine-Python is the official Python implementation of ReHLine, a powerful solver for large-scale empirical risk minimization (ERM) problems with convex piecewise linear-quadratic (PLQ) loss functions and linear constraints. Built with high-performance C++ core and seamless Python integration, ReHLine delivers exceptional speed while maintaining ease of use.

See more details in the ReHLine documentation.

✨ Key Features

• 🚀 Blazing Fast: Linear computational complexity per iteration, scales to millions of samples
• 🎯 Versatile: Supports any convex PLQ loss (hinge, check, Huber, and more)
• 🔒 Constrained Optimization: Handle linear equality and inequality constraints
• 📊 Scikit-Learn Compatible: Drop-in replacement with GridSearchCV, Pipeline support
• 🐍 Pythonic API: Both low-level and high-level interfaces for flexibility

📦 Installation

Quick Install

pip install rehline

Release wheels and CI cover standard CPython 3.10–3.14 on these platforms:

We do not publish wheels for 32-bit systems, Alpine/musl, Intel macOS, Linux ARM, Windows ARM, or free-threaded Python. Alpine/musl lacks compatible scikit-learn wheels, and the extension has not been validated for free-threaded Python. Intel macOS and the other ARM targets are outside the current CI matrix. Source distributions remain available; builds on other targets are not covered by the release tests.

Development Install

For contributors and developers:

git clone https://github.com/softmin/ReHLine-python.git
cd ReHLine-python
pip install -e ".[test]"

To run tests:

pytest tests/

🚀 Quick Start

Sparse input

The solver and estimators accept SciPy sparse matrices and two-dimensional sparse arrays for X and constraint matrix A, including CSR, CSC and COO. Each may independently be dense or sparse. Training normalizes sparse inputs to float64 CSR.

from scipy.sparse import csr_matrix
from rehline import plq_Ridge_Classifier

X = csr_matrix([[1., 0., 0.], [0., 1., 0.], [1., 0., 1.], [0., 1., 1.]])
model = plq_Ridge_Classifier(loss={"name": "svm"}, C=0.1, tol=1e-8, max_iter=100000)
model.fit(X, [1, -1, 1, -1])
prediction = model.predict(X)

Sparse X also works with quantile regression, ElasticNet, multiclass models, CQR, warm starts, C paths and to_inference() snapshots. Intercept augmentation preserves sparsity. In sklearn pipelines, use StandardScaler(with_mean=False).

Sparse A works through ReHLine_solver, ReHLine, the PLQ Ridge/ElasticNet estimators, custom constraints, and constrained C paths. Constraints retain the convention A @ beta + b >= 0. For example:

from scipy.sparse import eye

# Nonnegative feature coefficients, without constructing a dense identity.
model = plq_Ridge_Classifier(loss={"name": "svm"}, A=eye(X.shape[1]),
                             b=[0.] * X.shape[1], tol=1e-8, max_iter=100000)
model.fit(X, [1, -1, 1, -1])

Constraint normalization, dual recovery and CD updates preserve sparse storage. Nonnegativity (nonnegative / >=0) and monotonicity (monotonic / monotonicity, including decreasing=True) always construct CSR constraints, even for dense X. Their storage grows linearly with the number of features. Mixed constraint blocks stay sparse. Fairness computes dense covariance rows without centering the full design. With fit_intercept=True, a custom A with d columns constrains feature coefficients; d+1 columns constrain the features and actual intercept, including intercept_scaling.

Loss parameters, b, rho, coefficients and dual variables remain dense. Dimensions and stored-entry counts must fit in int32; representable int64 indices are converted safely. Conversion and native binding may copy sparse buffers using storage proportional to nnz. The optional precision-floor polish_primal step may densify a small active constraint submatrix, capped at 1,000,000 entries for sparse A; above this size it continues CD without that correction. The objective-gap and feasibility stopping requirements are unchanged.

Scikit-Learn Style API (Recommended)

ReHLine provides plq_Ridge_Classifier and plq_Ridge_Regressor that work seamlessly with scikit-learn:

from rehline import plq_Ridge_Classifier
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

# Generate dataset
X, y = make_classification(n_samples=1000, n_features=20, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

# Simple usage
clf = plq_Ridge_Classifier(loss={'name': 'svm'}, C=1.0)
clf.fit(X_train, y_train)
print(f"Accuracy: {clf.score(X_test, y_test):.3f}")

# Use in Pipeline
pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('classifier', plq_Ridge_Classifier(loss={'name': 'svm'}))
])
pipeline.fit(X_train, y_train)

# Hyperparameter tuning with GridSearchCV
param_grid = {
    'C': [0.1, 1.0, 10.0],
    'loss': [{'name': 'svm'}, {'name': 'sSVM'}]
}
grid_search = GridSearchCV(plq_Ridge_Classifier(loss={"name": "svm"}), param_grid, cv=5)
grid_search.fit(X_train, y_train)
print(f"Best params: {grid_search.best_params_}")

See more details in ReHLine with Scikit-Learn.

Low-Level API for Custom Problems

from rehline import ReHLine
import numpy as np

# Generate sample data
np.random.seed(42)
X = np.random.randn(100, 5)
y = np.random.choice([-1, 1], size=100)
n, d = X.shape
C = 1.0

# Define custom PLQ loss parameters
clf = ReHLine()
# Set custom U, V matrices for ReLU loss
# and S, T, tau for ReHU loss
## U
clf._U = -(C*y).reshape(1,-1)
## V
clf._V = (C*np.ones(n)).reshape(1,-1)

# Set custom linear constraints A*beta + b >= 0
X_sen = X[:,0] - X[:,0].mean()
tol_sen = 0.1
clf._A = np.repeat([X_sen @ X], repeats=[2], axis=0) / n
clf._A[1] = -clf._A[1]
clf._b = np.full(2, tol_sen)

clf.fit(X)

See more detailed in Manual ReHLine Formulation.

🎯 Use Cases

ReHLine excels at solving a wide range of machine learning problems:

⚡ Performance Benchmarks

ReHLine delivers exceptional speed compared to state-of-the-art solvers. Here are speed-up factors on real-world datasets:

Speed Comparison vs. Popular Solvers

Note: "∞" indicates the competing solver failed to produce a valid solution or exceeded time limits. Results from NeurIPS 2023 paper.

Reproducible Benchmarks (powered by benchopt)

All benchmarks are reproducible via benchopt at our ReHLine-benchmark repository.

The current quick/dense suites and shared objective checks are maintained in ReHLine-benchmarking. See test setup for the Python repository's external test dependency.

🤝 Contributing

We welcome contributions! Whether it's bug reports, feature requests, or code contributions:

• 🐛 Open an issue
• 💬 Start a discussion
• 🔀 Submit a pull request

📚 Citation

If you use ReHLine in your research, please cite our NeurIPS 2023 paper:

@inproceedings{dai2023rehline,
  title={ReHLine: Regularized Composite ReLU-ReHU Loss Minimization with Linear Computation and Linear Convergence},
  author={Dai, Ben and Qiu, Yixuan},
  booktitle={Thirty-seventh Conference on Neural Information Processing Systems},
  year={2023}
}

🔗 ReHLine Ecosystem

Source distributions include Eigen 5.0.1 headers and licenses, so compiling a published source package does not require an Eigen download. A C++ compiler and the Python build dependencies are still required.

Git checkouts do not include Eigen. Editable installs, wheel builds and source distribution builds automatically download the pinned release when needed and check its SHA-256 and individual file checksums against tools/eigen-5.0.1.json. No separate preparation command is required. The files in vendor/eigen-5.0.1/ are ignored by Git and reused after verification on later builds. For offline preparation, use python tools/prepare_eigen.py --archive /path/to/eigen-5.0.1.zip; the same checksums are required. Running python tools/prepare_eigen.py without --archive is an optional way to download Eigen ahead of time. Builds from published source packages use only the bundled headers and do not download Eigen, including when their contents are missing or corrupted.

Set EIGEN3_INCLUDE_DIR to a local directory containing Eigen/Core to compile against a different local Eigen installation without preparing the default headers. Creating an sdist still prepares the pinned headers automatically so the resulting package remains self-contained, regardless of this override. To build an sdist:

python -m build --sdist

The sdist command verifies all pinned headers and licenses before packaging. An incomplete or modified existing copy fails verification instead of being silently reused. CI exercises automatic preparation and rebuilds the sdist with Python network access disabled.