Probabilistic Scoring Lists

Probabilistic scoring lists are incremental models that evaluate one feature of the dataset at a time. PSLs can be seen as a extension to scoring systems in two ways:

• they can be evaluated at any stage allowing to trade of model complexity and prediction speed.
• they provide probablistic predictions instead of deterministic decisions for each possible score.

Scoring systems are used as decision support systems for human experts e.g. in medical or judical decision making.

This implementation adheres to the sklearn-api.

Install

pip install scikit-psl

Usage

For examples have a look at the examples folder, but here is a simple example

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

from skpsl import ProbabilisticScoringList

# Generating synthetic data with continuous features and a binary target variable
X, y = make_classification(n_informative=10, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=.2, random_state=42)

psl = ProbabilisticScoringList({-1, 1, 2})
psl.fit(X_train, y_train)
print(f"Brier score: {psl.score(X_test, y_test, -1):.4f}")
"""
Brier score: 0.2438  (lower is better)
"""

df = psl.inspect(5)
print(df.to_string(index=False, na_rep="-", justify="center", float_format=lambda x: f"{x:.2f}"))
"""
 Stage Threshold  Score  T = -2  T = -1  T = 0  T = 1  T = 2  T = 3  T = 4  T = 5
  0            -     -       -       -   0.51      -      -      -      -      - 
  1     >-2.4245  2.00       -       -   0.00      -   0.63      -      -      - 
  2     >-0.9625 -1.00       -    0.00   0.00   0.48   1.00      -      -      - 
  3      >0.4368 -1.00    0.00    0.00   0.12   0.79   1.00      -      -      - 
  4     >-0.9133  1.00    0.00    0.00   0.12   0.12   0.93   1.00      -      - 
  5      >2.4648  2.00    0.00    0.00   0.07   0.07   0.92   1.00   1.00   1.00 
"""

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

0.7.2 - 2024-10-11

Fixed

• MaxEntropyBinarizer also uses stricter test to detect binary features

0.7.1 - 2024-10-09

Fixed

• PSL uses stricter test to detect binary features

0.7.0 - 2024-03-27

Added

• PSL classifier
  • probability predictions can return confidence intervals
  • probability calibration using BetaCalibration
  • stages can now be sliced and iterated (_getitem_(), _iter_())
• Metrics
  • Weighted loss metric
  • Rank loss metric

Fixed

• PSL more robust against non-standard class labels like "True"/"False" instead of boolean values

0.6.3 - 2024-03-08

Added

• PSL supports Dataframes as inputs

0.6.2 - 2024-02-01

Added

• PSL supports instance weights

0.6.1 - 2023-12-15

Added

• Extended precision at recall function

0.6.0 - 2023-12-07

Added

• Significantly extended the configuration capabilities with predefined features to limit the PSLs searchspace

Changed

• PSL global loss defaults to sum(cascade)
• rewrote/extracted expected entropy calculation

Fixed

• PSL inspect is now more robust

0.5.1 - 2023-11-24

Fixed

• PSL classifier optimization regarding global loss was incorrect

0.5.0 - 2023-11-16

Added

• _ClassifierAtK
  • Sigmoid calibration additional to isotonic
• PSL classifier
  • Make optimization loss configurable
  • Small searchspace_analyisis(·) function makes lookahead choice more informed

Fixed

• Fixed lookahead search space and considering global loss for model-sequence evaluation

Changed

• Updated dependencies and added black
• Moved Binarizer to different module
• Moved PSL hyperparameters to constructor

0.4.2 - 2023-11-09

Fixed

• _ClassifierAtK
  • Expected entropy for stage 0 now also calculated wrt. base 2
  • Data with only 0 or 1 is now also interpret as binary data

0.4.1 - 2023-10-19

Fixed

• Small import error

0.4.0 - 2023-10-17

Added

• Add brute force threshold optimization method to find the global optimum, bisect optimizer remains default method

Changed

• Restructured source files

0.3.1 - 2023-09-12

Fixed

• PSL is now correctly handles when all instances belong to the negative class
• #1 if the first feature is assigned a negative score, it is now assigned the most negative score

0.3.0 - 2023-08-10

Added

• PSL classifier can now run with continuous data and optimally (wrt. expected entropy) select thresholds to binarize the data

Changed

• Significantly improved optimum calculation for MinEntropyBinarizer (the same optimization algorithm is shared with the psls internal binarization algorithm)

0.2.0 - 2023-08-10

Added

• PSL classifier
  • introduced parallelization
  • implemented l-step lookahead
  • simple inspect(·) method that creates a tabular representation of the model

0.1.0 - 2023-08-08

Added

• Initial implementation of the PSL algorithm