samesame 0.1.3

Classifier two-sample tests and related tests.

samesame

Same, same but different ...

samesame implements classifier two-sample tests (CTSTs) and as a bonus extension, a noninferiority test (NIT).

These were either missing or implemented with some tradeoffs (looking at you, sample-splitting) in existing libraries. And so, samesame fills in the gaps :)

Motivation

What is samesame good for? It is for data (model) validation, performance monitoring, drift detection (dataset shift), statistical process control, covariate balance and so on and so forth.

As an example, this motivating example comes from the related R package dsos.

Installation

To install, run the following command:

python -m pip install samesame

Quick Start

Simulate outlier scores to test for no adverse shift when the null (no shift) holds.

from samesame.ctst import CTST
from samesame.nit import DSOS
from sklearn.metrics import roc_auc_score
import numpy as np

n_size = 600
rng = np.random.default_rng(123_456)
os_train = rng.normal(size=n_size)
os_test = rng.normal(size=n_size)
null_ctst = CTST.from_samples(os_train, os_test, metric=roc_auc_score)
null_dsos = DSOS.from_samples(os_train, os_test)

In this example, we reject the null of equal distribution (i.e. CTST)

print(f"{null_ctst.pvalue=:.4f}")
# null_ctst.pvalue=0.0358

However, we fail to reject the null of no adverse shift (i.e. DSOS), meaning that the test sample (os_test) does not seem to contain disproportionally more outliers than the training sample (os_train).

print(f"{null_dsos.pvalue=:.4f}")
# null_dsos.pvalue=0.9500

This is the type of false alarms that samesame can highlight by comparing tests of equal distribution to noninferiority tests.

Usage

Functionality

Below, you will find an overview of common modules in samesame.

Function	Module
Bayesian inference	samesame.bayes
Classifier two-sample tests (CTSTs)	samesame.ctst
Noninferiority tests (NITs)	samesame.nit

Attributes

When the method is a statistical test, samesame saves (stores) the results of some potentially computationally intensive results in attributes. These attributes, when available, can be accessed as follows.

Attribute	Description
.statistic	The test statistic for the hypothesis.
.null	The null distribution for the hypothesis.
.pvalue	The p-value for the hypothesis.
.posterior	The posterior distribution for the hypothesis.
.bayes_factor	The bayes factor for the hypothesis.

Examples

To get started, please see the examples in the docs.

Dependencies

samesame has few dependencies beyond the standard library. It will probably work with some older Python versions. It is, in short, a lightweight dependency for most machine learning projects.samesame is built on top of, and is compatible with, scikit-learn and numpy.