ML Testing

The goal of this module is to create a flexible and easy to use module for testing machine learning models, specifically those in scikit-learn.

The tests will be readable enough that anyone can extend them to other frameworks and APIs with the major notions kept the same, but more or less the ideas will be extended, no work will be taken in this library to extend passed the scikit-learn API.

You can read the docs for a more detailed explaination.

A video explaining the idea behind the framework

Tests Covered

Testing Against Metrics
- Classification Tests
  - Rule Based Testing:
    - precision lower boundary
    - recall lower boundary
    - f1 score lower boundary
    - AUC lower boundary
    - precision lower boundary per class
    - recall lower boundary per class
    - f1 score lower boundary per class
    - AUC lower boundary per class
  - Decision Based Testing:
    - precision fold below average
    - recall fold below average
    - f1 fold below average
    - AUC fold below average
    - precision fold below average per class
    - recall fold below average per class
    - f1 fold below average per class
    - AUC fold below average per class
  - Against New Predictions
    - proportion of predictions per class
    - class imbalance tests
    - probability distribution similarity tests
    - calibration tests
  - environmental impact tests
    - energyusage upper bound test
- Regression Tests
  - Rule Based Testing:
    - Mean Squared Error upper boundary
    - Median Absolute Error upper boundary
  - Decision Based Testing:
    - Mean Squared Error fold above average
    - Median Absolute Error fold above average
Testing Against Run Time Performance
- prediction run time for simulated samples of size X
Testing Against Input Data
- percentage of correct imputes for any columns requiring imputation
- dataset testing - http://www.vldb.org/pvldb/vol11/p1781-schelter.pdf
Memoryful Tests
- cluster testing - this is about the overall structure of the data If the number of clusters increases or decreases substantially that should be an indicator that the data has changed enough that things should possibly be rerun
- correlation testing - this is about ensuring that the correlation for a given column with previous data collected in the past does not change very much. If the data does change then the model should possibly be rerun.
- shape testing - this is about ensuring the general shape of for the given column does not change much over time. The idea here is the same as the correlation tests.

Possible Issues

Some known issues with this, any machine learning tests are going to require human interaction because of type 1 and type 2 error for statistical tests. Additionally, one simply needs to interrogate models from a lot of angles. It can't be from just one angle. So please use with care!

Future Features

cross validation score testing
add custom loss function
add custom accuracy function
add these tests: https://www.datasciencecentral.com/profiles/blogs/a-plethora-of-original-underused-statistical-tests
clustering for classification
Unsupervised and semi supervised tests
- verify similarity in clusters to similarity in labels
- generate a small representative set of labels and then propagate other labels

References

Metadata

Release files for drifter-ml 0.26

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for drifter-ml 0.26
File	Size	Uploaded	Provenance
drifter_ml-0.26.tar.gz	22.7 kB	Jan 24, 2026	Without provenance	Details

Built distribution (wheel)

Table of built distributions (wheels) for drifter-ml 0.26
File	Interpreter	ABI	Platform	Reset
drifter_ml-0.26-py3-none-any.whl 19.3 kB Jan 24, 2026 Without provenance	Python 3	none	any	Details

Total release size: 42.1 kB

Release files / drifter_ml-0.26.tar.gz

Download URL	drifter_ml-0.26.tar.gz
Size	22.7 kB
Tags	Source
SHA-256 checksum How to use checksums	`e1b36a13425b5dd50c2cfa5b1c5beeea0f3e3392d2c8d4e0d9928202fa1702f1`
BLAKE2b-256 checksum How to use checksums	`77733d79106c0a41ca2557748328f7d11989cc2c1e8173790da7691f7c019d86`
Upload date	Jan 24, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/6.1.0 CPython/3.13.2`

Release files / drifter_ml-0.26-py3-none-any.whl

Download URL	drifter_ml-0.26-py3-none-any.whl
Size	19.3 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`d82b7d13cd225bebb946a0b72366a9f9cefd9464a71f71ffe03b2a3e730f0af5`
BLAKE2b-256 checksum How to use checksums	`8432b71438fcffb7ca9e1328b1a56ed9068c72b6d8f9d8ca527a3758b2809e39`
Upload date	Jan 24, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/6.1.0 CPython/3.13.2`

Security for drifter-ml 0.26

Report project as malware

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.