samesame 0.1.0

Classifier two-sample tests and noninferiority tests.

samesame

Same, same but different ...

samesame implements classifier two-sample tests (CTSTs) and as a bonus extension, a noninferiority test (NIT).

These were either missing or implemented with some tradeoffs (looking at you, sample-splitting) in existing libraries. And so, samesame fills in the gaps :)

Motivation

What is samesame good for? It is for data (model) validation, performance monitoring, drift detection (dataset shift), statistical process control, and on and on.

Want more? Here you go. This motivating example comes from the related R package dsos.

Installation

To install, run the following command:

python -m pip install samesame

Usage

Functionality

Below, you will find an overview of common modules in samesame.

Function	Module
Bayesian inference	samesame.bayes
Classifier two-sample tests (CTSTs)	samesame.ctst
Noninferiority tests (NITs)	samesame.nit

Attributes

When the method is a statistical test, samesame saves (stores) the results of some potentially computationally intensive results in attributes. These attributes, when available, can be accessed as follows.

Attribute	Description
.statistic	The test statistic for the hypothesis.
.null	The null distribution for the hypothesis.
.pvalue	The p-value for the hypothesis.
.posterior	The posterior distribution for the hypothesis.
.bayes_factor	The bayes factor for the hypothesis.

Examples

To get started, please see the documentation and examples.

Dependencies

samesame has few dependencies beyond the standard library. It will probably work with some older Python versions. It is, in short, a lightweight dependency for most machine learning projects.samesame is built on top of, and is compatible with, scikit-learn and numpy.