Skip to main content

deskit

This is not a usage guide, but an overview of deskit. For a usage guide, see the documentation

deskit is a flexible, lightweight, and easy-to-use ensembling library that implements Dynamic Ensemble Selection (DES) algorithms for ensembling multiple ML models on a given dataset.

The library works entirely with data, taking as input a validation dataset along with precomputed predictions and outputting a dictionary of weights per model. This means that it can be used with any library or model without requiring any wrappers, including custom models, popular ML libraries, and APIs.

deskit includes several DES algorithms, and it works with both classification and regression.

See the full documentation here.

Dynamic Ensemble Selection

Ensemble learning in machine learning refers to when multiple models trained on a single dataset combine their predictions to create a single, more accurate prediction, usually through weighted voting or picking the best model.

DES refers to techniques where the models or their voting weights are selected dynamically for every test case. This selection bases on the idea of competence regions, which is the concept that there are regions of feature space where certain models perform particularly well, so every base model can be an expert in a different region. Only the most competent, or an ensemble of the most competent models is selected for the prediction.

Through empirical studies, DES has been shown to perform best on representative, diverse, and heterogeneous datasets, as well as non-stationary data (concept drift), models that haven't perfected a dataset, and when used on an ensemble of models with differing architectures and perspectives.

However, DES is not an automatic improvement. It tends to perform worse when datasets are homogeneous or have low diversity, when the validation set isn't a good representation of the test set, when using very high dimensional data or few training samples, or when a single model dominates a dataset.


Installation

pip install deskit

# The library runs with Nearest Neighbors from sklearn for exact KNN
pip install scikit-learn

# Alternatively, ANN can be used for faster runtimes at the cost of
# slightly lower accuracy. The following three are supported;
# Install the one you want to use.
pip install faiss-cpu   # FAISS (good default for most datasets)
pip install annoy       # Annoy (memory-efficient, simple)
pip install hnswlib     # HNSW (best for high-dimensional data)

Dependencies

Python (>= 3.8)

NumPy (>= 1.21)


Quick start

For a more detailed understanding of how to use the library, consult the usage guide.

from deskit.des.knorau import KNORAU

# 1. Train your models
models = {"rf": rf, "xgb": xgb, "mlp": mlp}

# 2. Get predictions on a held-out validation set
#    Regression: scalar arrays
#    Classification: probability arrays OR hard predictions
val_preds = {name: m.predict_proba(X_val) for name, m in models.items()}

# 3. Fit the router
router = KNORAU(task="classification", metric="accuracy", mode="max", k=20)
router.fit(X_val, y_val, val_preds)

# 4. Get test predictions from your models
test_preds = {name: m.predict_proba(X_test) for name, m in models.items()}

# 5. Route and blend test samples instantly
#    Returns final ensembled predictions for the entire batch
#    Alternatively, predict_weights can be used to get an array of weights dicts
#    Weights dict example:{"rf": 0.7, "xgb": 0.2, "mlp": 0.1}
predictions = router.predict(X_test, test_preds, temperature=0.1)

For classification with probability arrays, blend the output the same way to get a final probability distribution, then take the argmax.


Batch vs. Streaming Usage

deskit is fully polymorphic and handles both offline batch processing and real-time streaming data natively:

  • Batch Mode: Pass a 2D feature matrix and a dictionary of prediction arrays (as shown above) to process everything at vectorized speeds.
  • Streaming Mode: Pass a single 1D feature array and a dictionary of individual model outputs. router.predict() will automatically detect it and return a single final prediction instead of an array.

If you want to handle the ensembling math yourself or inspect the raw dynamic routing weights, use predict_weights:

# Returns a dictionary of weights (or array of weights for a batch)
# e.g., {"rf": 0.7, "xgb": 0.2, "mlp": 0.1}
weights = router.predict_weights(X_test, temperature=0.1)

Hyperparameter tuning

deskit supports hyperparameter tuning of KNN-based methods using Leave One Out (LOO) tuning. This technique excludes the closest data point with a negligent distance from the one being used in predict(), allowing for deskit methods to be tuned from the same DSEL set it was fit on and removing the need for cross-validation as it sees the entire set throughout the tuning process.

results = router.predict(X_val, val_preds, loo=True)

Why deskit?

Most DES libraries are tied to scikit-learn. deskit only ever sees a numpy feature matrix and a dict of prediction arrays, so the models themselves are never touched after training. This allows for more flexibility and a lighter library.

Furthermore, deskit works with both classification and regression, while the majority of DES libraries and literature is focused only on classification tasks.

# PyTorch example 
with torch.no_grad():
    val_preds  = {name: m(X_val_t).cpu().numpy()  for name, m in models.items()}
    test_preds = {name: m(X_test_t).cpu().numpy() for name, m in models.items()}

router = KNORAU(task="classification", metric="accuracy", mode="max", k=20)
router.fit(X_val, y_val, val_preds)

# Get final blended predictions
X_test_np = X_test_t.cpu().numpy()
predictions = router.predict(X_test_np, test_preds, temperature=0.1)

Algorithms

Full explanation of the algorithms, syntax, and parameters is available in the documentation. If you're struggling to decide on which algorithm to use, see the algorithm selection guide.

Method Best for Notes
DEWS-U Regression Softmax over neighborhood-averaged scores. Temperature controls sharpness.
DEWS-I Regression Like DEWS-U but scores are inverse-distance weighted.
DEWS-T Both Like DEWS-I but fits a weighted trend line over neighbor scores.
DEWS-V Regression Like DEWS-U but scores are variance-penalized.
DEWS-IV Regression Like DEWS-V but scores are also inverse-distance weighted.
LWSE-U Both Per-sample NNLS weight estimation over the local neighbourhood.
LWSE-I Both Like LWSE-U but rows are inverse-distance weighted.
KNORA-U Classification Each model earns one vote per neighbor it correctly classifies.
KNORA-E Classification Only models correct on all neighbors survive; falls back to smaller neighborhoods.
KNORA-IU Classification Like KNORA-U but votes are inverse-distance weighted.
OLA Both Hard selection: only the single best model in the neighborhood contributes.

ANN backends

deskit supports three Approximate Nearest Neighbour backends plus exact search:

Preset Backend Install Notes
exact sklearn KNN scikit-learn Exact, no extra deps
balanced FAISS IVF faiss-cpu ~98% recall, good default
fast FAISS IVF faiss-cpu ~95% recall, faster queries
turbo FAISS flat faiss-cpu Exact via FAISS, GPU-friendly
high_dim_balanced HNSW hnswlib Best for >100 features, balanced
high_dim_fast HNSW hnswlib Best for >100 features, faster

Annoy is also available as a custom backend — memory-efficient and simple, good for datasets that need to be persisted to disk.

# Exact search (no extra deps)
router = KNORAU(..., preset="exact")

# High-dimensional data
router = KNORAU(..., preset="high_dim_balanced")

# Custom FAISS config
router = KNORAU(..., preset="custom", finder="faiss",
                index_type="ivf", n_probes=50)

# Annoy
router = KNORAU(..., preset="custom", finder="annoy",
                n_trees=100, search_k=-1)

Custom metrics

Any callable (y_true, y_pred) -> float works:

def pinball(y_true, y_pred, alpha=0.9):
    e = y_true - y_pred
    return alpha * e if e >= 0 else (alpha - 1) * e

router = DEWSU(task="regression", metric=pinball, mode="min", k=20)

Built-in metric strings: accuracy, mae, mse, rmse, log_loss, prob_correct.

Alternatively, pass metric=None and populate the dictionary passed to fit() with your own per-sample computed scores instead of predictions.


Data types

deskit can be used with non-tabular data types like images, time series, and more. However, when used, the passed features either need to be run through a feature extractor beforehand, such as a CNN backbone for images.

Benchmark results

Regression-specific benchmark

To benchmark how the DEWS family specifically performs on regression tasks, since this was an identified gap in DES literature and implementations, DEWS was benchmarked against single best, simple average, and various stacking baselines. For this benchmark, 35 datasets were used, pulled from the OpenML-CTR23 regression suite, benchmarked with 10 folds and 3 seeds. All the methods were tuned with the exact same Random Search budget of 50. The full results can be seen in the documentation.

ΔSB = % improvement over SingleBest (positive = better). Gap = DEWS ΔSB − best non-DEWS ΔSB (negative = DEWS behind non-DEWS).

| Pool | DEWS mean | DEWS ΔSB | Best non-DEWS | |---|---|---:|---:|---|---:|---:| | Bagging-DT | 7798.74 | +32.23% | ValWeightedAverage | | Bagging-KNN | 9341.37 | +9.34% | ValWeightedAverage | | Subspace-DT | 8354.85 | +27.43% | RidgeStacking | | Subspace-KNN | 8704.69 | +12.64% | StackingRF |

SingleBest = 0% baseline. "Best non-DEWS" excludes all DEWS variants and Oracle; it may include stacking, simple/weighted averaging, OLA, or SingleBest itself.

As evident, DEWS does best compared to competition in Bagging pools, achieving up to a 32% improvement over single best, and beats stacking and averaging baselines, showing that in some cases, DES can have performance advantages.

General heterogeneous benchmark

20-seed benchmark (seeds 0–19) on standard sklearn and OpenML datasets. "Best Single" is the best individual model selected on the validation set. "Simple Average" is uniform equal-weight blending, included as a baseline.

It is important to consider that these experiments were run with the default hyperparameters, meaning that they could vary greatly with different values, and results could improve with tuning. For a more detailed benchmark breakdown, see the benchmark in the documentation. To see the full results, see results.txt in the tests folder.

This pool was selected for having variability in architectures while avoiding a single dominant model.

deskit algorithms tested: OLA, DEWS-U, DEWS-I, DEWS-T, DEWS-V, DEWS-IV, LWSE-U, LWSE-I, KNORA-U, KNORA-E, KNORA-IU.

Regression (MAE, lower is better)

Pool: KNN, Decision Tree, SVR, Ridge, Bayesian Ridge.

% shown as delta vs Best Single. 20-seed mean.

Dataset Best Single Simple Avg deskit best
California Housing (sklearn) 0.3956 +7.99% −2.54% (DEWS-I)
Bike Sharing (OpenML) 51.678 +47.77% −6.86% (DEWS-I)
Abalone (OpenML) 1.4981 +1.14% +1.47% (KNORA-U/KNORA-IU)
Diabetes (sklearn) 44.504 +3.18% +0.86% (DEWS-IV)
Concrete Strength (OpenML) 5.2686 +23.66% −5.41% (LWSE-I)

deskit beats best single and simple averaging on 3/5 regression datasets. This shows how DES can provide a strong boost if used on the right dataset, but it might be counterproductive if used blindly.

KNORA variants are designed for classification, which explains the poor performance on regression datasets; However, some exceptions can occur in certain datasets when the target is discrete and classification-like (like in Abalone).

DEWS-I and LWSE-I show the largest improvements on their respective datasets.

Classification (Accuracy, higher is better)

Pool: KNN, Decision Tree, Gaussian NB, SVM-RBF, Logistic Regression.

% shown as delta vs Best Single. 20-seed mean.

Dataset Best Single Simple Avg deskit best
HAR (OpenML) 98.24% −0.33% +0.16% (DEWS-T)
Yeast (OpenML) 58.87% +0.77% +1.66% (KNORA-IU)
Image Segment (OpenML) 93.70% +1.40% +2.25% (DEWS-T / DEWS-IV)
Waveform (OpenML) 85.91% −0.98% −0.39% (DEWS-T)
Vowel (OpenML) 89.95% −2.05% +2.95% (LWSE-I)

deskit beats or matches best single and simple averaging on 4/5 classification datasets.

Speed (mean ms fit + predict, 20 seeds, all tested algorithms combined)

Consider that usually it is recommended to only use one algorithm at a time, this benchmark ran eleven of them at the same time, so with a single one runtime is expected to be about 11x faster. For this benchmark, preset='balanced' was used, so the backend was an ANN algorithm with FAISS IVF.

Dataset deskit (11 algorithms)
California Housing 351.0 ms
Bike Sharing 283.5 ms
Abalone 72.9 ms
Diabetes 14.0 ms
Concrete Strength 22.5 ms
HAR 693.1 ms
Yeast 44.7 ms
Image Segment 69.9 ms
Waveform 124.5 ms
Vowel 39.0 ms

deskit caches all model predictions on the validation set at fit time and reads from that matrix at inference.


Contributing

Issues and PRs welcome.

Metadata

Release files for deskit 1.3.2.3

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for deskit 1.3.2.3
File Size Uploaded Provenance Actions
deskit-1.3.2.3.tar.gz 37.6 kB Without provenance Details for deskit-1.3.2.3.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for deskit 1.3.2.3
File Interpreter ABI Platform Actions
deskit-1.3.2.3-py3-none-any.whl Python 3 none any Details for deskit-1.3.2.3-py3-none-any.whl

Total release size: 88.0 kB

Release files / deskit-1.3.2.3.tar.gz

Download URL deskit-1.3.2.3.tar.gz
Size 37.6 kB
Tags Source
SHA-256 checksum
How to use checksums
fd68dafc5aa73d7cf44928a65a491c37fa9d97716360bf15fdadd37e34310fe9
BLAKE2b-256 checksum
How to use checksums
30f1eb50609c431f16496180586531a540cfe132f7c03079f2973cdd0b1bbfc2
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.12.3

Release files / deskit-1.3.2.3-py3-none-any.whl

Download URL deskit-1.3.2.3-py3-none-any.whl
Size 50.4 kB
Tags Python 3
SHA-256 checksum
How to use checksums
623d6124eba3152291a044271f72cd4d870c157ec78da17f13f0ee0e0f230bbb
BLAKE2b-256 checksum
How to use checksums
5ca68aa22983aa3bf78fdc75653cf644a368a89ab12bf005ce6ceaac39260b55
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.12.3

Security for deskit 1.3.2.3

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page