Pairwise Difference Learning (pdll)

A scikit-learn meta-estimator that makes tree models more accurate on small tabular datasets by learning from pairs of points.

Pairwise Difference Learning (PDL) turns a multiclass problem into a binary one. Instead of asking "what class is this point?", the base model learns "are these two points of the same class?" from every pair of training points. A new point is then compared with the training points (the anchors), whose labels are known. With n training rows the model sees n² examples, which helps most when data is scarce.

pdll wraps any scikit-learn classifier (or regressor) and keeps the usual fit / predict / predict_proba / score API. It implements Belaid, Rabus & Hüllermeier, Pairwise Difference Learning for Classification, Discovery Science 2024 (arXiv).

✨ Highlights

• 🔌 Drop-in: works in Pipeline, GridSearchCV and cross_val_score, passes scikit-learn's own estimator checks, and never modifies the estimator you pass in.
• 🧾 Raw DataFrames welcome (pandas, polars or pyarrow): string, category, bool and nullable columns are encoded for you; missing values go straight to base learners that support them (e.g. HistGradientBoosting).
• 📈 More accurate tree models: in the paper, PDL improves tree-based learners on 99 OpenML datasets (macro-F1). In the end-to-end suite that CI runs on every change, it wins 18 of 30 dataset/learner pairs (11 losses) and raises mean macro-F1 from 0.873 to 0.894.
• 🧪 Validated on 131 real datasets (results): raw Kaggle, OpenML, scikit-learn and Zindi data, with no crashes and at most 2.6 GB of memory.
• 🛡️ Tested: Python 3.9–3.14, numpy ≥ 1.23, pandas ≥ 1.5, scikit-learn ≥ 1.2 (Kaggle and Colab stacks included), Linux, Windows and macOS.

📖 Contents

Installation · Usage · When to use it · How does it work? · Evaluation · Citation

Installation

pip install -U pdll

Usage

from sklearn.datasets import load_wine
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score

from pdll import PairwiseDifferenceClassifier

X, y = load_wine(return_X_y=True)

rf = RandomForestClassifier(class_weight="balanced", random_state=0)
pdc = PairwiseDifferenceClassifier(estimator=rf)

print("RandomForest     :", cross_val_score(rf, X, y, cv=5).mean())
print("PDC(RandomForest):", cross_val_score(pdc, X, y, cv=5).mean())

pdc.fit(X, y)
pdc.predict(X[:3])            # array of labels, like any scikit-learn classifier
pdc.predict_proba(X[:3])      # ndarray (n_samples, n_classes), columns follow pdc.classes_

Raw DataFrames (strings, categories, missing values) work directly:

from sklearn.datasets import fetch_openml
from sklearn.ensemble import HistGradientBoostingClassifier

X, y = fetch_openml("titanic", version=1, as_frame=True, return_X_y=True)
X = X[["pclass", "sex", "age", "sibsp", "parch", "fare", "embarked"]]   # drop identifiers such as name or ticket

pdc = PairwiseDifferenceClassifier(HistGradientBoostingClassifier(class_weight="balanced"))
pdc.fit(X[:600], y[:600]).score(X[600:900], y[600:900])

Regression uses the same idea: the base model learns the target difference between two points.

from pdll import PairwiseDifferenceRegressor
from sklearn.tree import DecisionTreeRegressor

pdr = PairwiseDifferenceRegressor(DecisionTreeRegressor()).fit(X_train, y_train)
pdr.learn_anchor_weights(X_val, y_val, method="L2")   # optional: learn a weight per anchor

More in examples/: uncertainty from the per-anchor posteriors (predict_proba_samples), weighted regression, and more.

Main parameters

fit also accepts sample_weight. A pair (i, j) is weighted w_i·w_j, which is exactly equivalent to repeating rows.

When to use it

• ✅ Small tabular data: up to about 2,000 training rows. In the real-data benchmark PDL matches its base learner on average below 1,000 rows and beats it on many datasets. Above 2,000 rows it was usually less accurate than the base learner alone, and fit warns you.
• ✅ Tree-based base learners: DecisionTree, RandomForest, ExtraTrees, Bagging, (Hist)GradientBoosting. The weaker the learner, the larger the gain (a single decision tree gains 15 points on small digits data).
• ⚖️ Imbalanced classes: use class_weight="balanced". PDL ranks well (its ROC AUC is on par or better) but under-predicts minority classes by default.
• ⚠️ Parametric base learners (logistic regression, SVM, kNN, naive Bayes, MLP) rarely gain from PDL; pdll warns you.
• ⏱️ Cost: training sees up to max_pairs pairs with 3× the features, and prediction compares every point with up to max_anchors anchors. With the defaults, a typical dataset fits in seconds and stays well under 2 GB of memory, whatever its size. For more accuracy on a few thousand rows, raise max_pairs (e.g. 1_000_000), at about 10× the time and memory.

How does it work?

The PDL algorithm works by transforming the multiclass classification problem into a binary classification problem. The algorithm works as follows:

Example 1: Graphical abstract

Example 2: PDC trained on the Iris dataset

Evaluation

Real-world validation (131 datasets)

Before release, pdll ran with default settings on 131 raw real datasets: OpenML-CC18 and CTR23 (official task splits), Kaggle datasets, scikit-learn datasets and a Zindi dataset. 129 ran, and 2 were refused with an actionable message (sparse text, multi-target). There were no crashes, out-of-memory errors or timeouts, and peak memory stayed at or below 2.6 GB. Protocol and findings, full table.

The base learner here is a strong default, ExtraTrees with balanced class weights. PDL holds its own on small data and is not meant for large data; fit warns from 2,000 rows on.

Continuous benchmark (CI)

Every change to pdll runs tests/e2e: 10 offline datasets (iris, wine, breast cancer, digits, 2D toy problems, synthetic, a Titanic-like mixed-type table and a dataset with a tiny class). Each is scored with 5-fold stratified CV and macro-F1, comparing ExtraTrees, RandomForest and DecisionTree alone versus wrapped in PDC. The build fails if accuracy or speed regresses against the committed baseline.

The largest and most stable gains are for single decision trees (digits, 300 rows: 0.75 → 0.93; across three CV seeds a tree goes from 0.70–0.75 to 0.86–0.91). PDL does not win everywhere: it is slightly behind on the linearly separable toy problem and on breast cancer with forests.

Paper experiment

To reproduce the experiment of the paper, install the research requirements and run run_benchmark.py with a base learner and a dataset number, between 0 and 99. Example:

pip install -r requirements-benchmark.txt
python run_benchmark.py --model DecisionTreeClassifier --data 0

Scores will be stored in ./results/tmp/ directory. pdll 1.x reproduces the predictions of the original implementation (0.3.1) on the paper's inputs; tests/test_golden.py checks this on every change.

Experiment

We use 99 datasets from the OpenML repository. We compare the performance of the PDC algorithm with 7 base learners. We use the macro F1 score as a metric. The search space is inspired from TPOT a state-of-the-art library in optimizing Sklearn pipelines

Score comparison

2D datasets Examples

Here we see the difference in the learned patterns between PDL and the base learner. In case PDL is compatible with the base learner (DecisionTree, RandomForest) then the scores improves. In case the base learner is not compatible with PDL (SVC, AdaBoost, ...) then the scores gets lower.

Citation

If pdll helps your work, please cite the paper (GitHub's "Cite this repository" button gives the same entry, and pdll.__citation__ holds it too):

@inproceedings{belaid2024pairwise,
  title     = {Pairwise Difference Learning for Classification},
  author    = {Belaid, Mohamed Karim and Rabus, Maximilian and H{\"u}llermeier, Eyke},
  booktitle = {Discovery Science},
  pages     = {284--299},
  publisher = {Springer},
  year      = {2024},
  doi       = {10.1007/978-3-031-78980-9_18},
  url       = {https://arxiv.org/abs/2406.20031}
}

Uncertainty quantification with PDL: Belaid, Rabus & Hüllermeier, Uncertainty Quantification in Pairwise Difference Learning for Classification, Machine Learning, 2025, doi:10.1007/s10994-025-06949-0.

The first commit of this repository corresponds to the original implementation of the PDC algorithm.

Contributing

Issues and pull requests are welcome. pip install -e ".[test]" then pytest runs the fast suite; pytest -m e2e runs the performance regression gate (a few minutes) and pytest -m network the OpenML integration tests.

Acknowledgments: We would like to thank Tim Wibiral, Dorra ElMekki, Viktor Bengs, Muhammad Zeeshan Anwer, Muhammad Hossein Shaker, Alireza Javanmardi, Patrick Kolpaczki, and Maximilian Muschalik for their early comments on this work. We also acknowledge LRZ and IDIADA for computational resources.