⚠️ WIP: All 0.1.x releases are unstable. 0.2.0 will be the first stable, production-ready release.

Releases >0.1.3 are functional and ready for testing, but the API is still subject to change.

frameworthy

Lightweight statistical validation library for data changes. Built for engineers and scientists who need to validate data changes with statistical rigor. Works in data validation pipelines, in tests with assertions, or in exploratory settings with easily inspectable results.

Getting Started

Installation

Available on PyPI, use pip or your preferred package manager.

pip install frameworthy
# or
uv add frameworthy

Examples

See scripts/examples.py for a few quick examples.

import frameworthy as fw
import polars as pl # or pandas

# load your data if necessary
before_df = pl.read_csv("before.csv")
after_df = pl.read_csv("after.csv")

# run a check
result = (
    fw.check(after_df, before=before_df)
    .mean("column_name")
    .equivalent(within=0.1)
)

# inspect the results
print(result)
# or raise an exception on failure
result.assert_passed()

Usage

Bring your data: two pandas/polars DataFrames (before and after / pre and post).

A standard frameworthy check looks like this:

fw.check(after_df, before_df).mean("column_name").equivalent(within=0.1)

1. Start a check with fw.check(after_df, before_df)
2. Specify the metric to check, with .mean("column_name")
3. Make a claim about the change, using .equivalent(), .change_greater_than(), or .change_less_than()
4. Examine the results or assert that the check passed with .assert_passed() for testing

Methodology

These are the most important notes to be aware of for correct usage. For further details, see the more extensive docs.

Confidence Intervals

The parameter alpha is a one-sided significance level everywhere; .equivalent()'s displayed CI is (1-2α), not (1-α). This is to maintain consistency with the TOST (Two One-Sided Tests) equivalence testing method.

Examples:

• .equivalent()
  • alpha=0.05 → 90% CI
  • alpha=0.025 → 95% CI
• .change_greater_than() and .change_less_than()
  • alpha=0.1 → 90% CI
  • alpha=0.05 → 95% CI

Development

To run the tests:

make test

To run the examples:

uv run scripts/examples.py

License

MIT, but attribution is appreciated.