Skip to main content

abtest-framework

A production-grade A/B testing library for data scientists. Covers the full experiment lifecycle — from power analysis through Bayesian decision-making — with a Streamlit dashboard for interactive analysis.

Features

Module What it does
power Sample size & MDE calculations for proportions and means
frequentist z-test, Welch's t-test, CUPED variance reduction, Mann-Whitney, Bonferroni/BH correction
bayesian Beta-Binomial & Normal-Normal Bayesian tests with expected loss
sequential SPRT, always-valid p-values, novelty/primacy effect detection, SRM check

Quick Start

from abtest.power import sample_size_proportion
from abtest.frequentist import proportion_test, cuped_test
from abtest.bayesian import bayesian_proportion_test
from abtest.sequential import SPRTMonitor, detect_novelty_effect, check_srm

# 1. Pre-experiment: how many users do I need?
result = sample_size_proportion(baseline_rate=0.10, mde=0.02, alpha=0.05, power=0.80)
print(result)
# Per-group n: 3,841  |  Total n: 7,682

# 2. Sanity check before analysis
srm = check_srm(actual_ctrl=3841, actual_trt=3841)
print(srm["message"])  # ✅ No SRM detected

# 3. Frequentist test
result = proportion_test(388, 3841, 469, 3841, alpha=0.05)
print(result)
# Lift: +20.9%  |  p=0.0033  |  ✅ Significant

# 4. CUPED variance reduction (secondary metric)
cuped_result = cuped_test(ctrl_revenue, trt_revenue, ctrl_pre_revenue, trt_pre_revenue)
print(f"Variance reduced by {cuped_result.variance_reduction_pct}%")

# 5. Bayesian decision
bayes = bayesian_proportion_test(388, 3841, 469, 3841, loss_threshold=0.001)
print(f"P(treatment better): {bayes.prob_treatment_better:.1%}")
print(f"Decision: {bayes.decision}")
# P(treatment better): 99.8%
# Decision: ✅ Ship Treatment

# 6. Sequential monitoring (stop early when evidence is strong)
monitor = SPRTMonitor(p0=0.10, p1=0.12, alpha=0.05, beta=0.20)
for ctrl_obs, trt_obs in data_stream:
    result = monitor.update(ctrl_obs, trt_obs)
    if result.decision != "continue":
        print(result)
        break

# 7. Check for novelty effects
novelty = detect_novelty_effect(ctrl_timeseries, trt_timeseries, n_windows=4)
print(novelty)

Installation

# From PyPI (once published)
pip install abtest-framework

# From source
git clone https://github.com/yourusername/abtest-framework
cd abtest-framework
pip install -e ".[app]"

Streamlit Dashboard

streamlit run app.py

The dashboard has four pages:

  • Power Analysis — interactive sample size calculator with power curves
  • Significance Tests — proportion z-test, CUPED, multiple testing correction
  • Bayesian Analysis — posterior visualisation and expected loss decisions
  • Sequential & Novelty — SPRT monitoring chart and novelty effect detector

Project Structure

abtest-framework/
├── abtest/
│   ├── __init__.py       # Clean public API
│   ├── power.py          # Sample size & MDE calculations
│   ├── frequentist.py    # z-test, t-test, CUPED, BH correction
│   ├── bayesian.py       # Beta-Binomial & Normal-Normal Bayesian tests
│   └── sequential.py     # SPRT, always-valid p-values, novelty detection
├── tests/
│   └── test_abtest.py    # 23 unit tests (pytest)
├── examples/
│   └── full_experiment_walkthrough.py
├── app.py                # Streamlit dashboard
├── pyproject.toml        # Package config (PyPI-ready)
└── README.md

Key Concepts Implemented

CUPED (Controlled-experiment Using Pre-Experiment Data)

Variance reduction technique from Microsoft Research (Deng et al., 2013). Regresses out a pre-experiment covariate to reduce metric noise, giving equivalent power with fewer users.

Expected Loss (Bayesian Decision Rule)

Instead of a binary significant/not-significant decision, compute:

  • E[loss | choose treatment] = average amount lost if treatment turns out worse
  • Ship when expected loss falls below a business-defined threshold

SPRT (Sequential Probability Ratio Test)

Wald's sequential test that lets you stop experiments early when evidence accumulates — without inflating the false positive rate the way repeated classical testing does.

Novelty Effect Detection

Fits a linear regression on per-window treatment lift over time. Flags experiments where lift significantly decays — indicating users were reacting to novelty, not real value.

Sample Ratio Mismatch (SRM)

Chi-square test that checks whether the actual user split matches the intended randomisation ratio. A significant SRM means the experiment has a bug and results are invalid.

Testing

pytest tests/ -v
# 23 passed in 1.55s

Publishing to PyPI

pip install build twine
python -m build
twine upload dist/*

References

  • Deng, A., et al. (2013). Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data. WSDM.
  • Johari, R., et al. (2017). Peeking at A/B Tests: Why it matters, and what to do about it. KDD.
  • Wald, A. (1945). Sequential Tests of Statistical Hypotheses. Annals of Mathematical Statistics.
  • Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy Online Controlled Experiments. Cambridge.

License

MIT

Metadata

Release files for abtest-framework 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for abtest-framework 0.1.0
File Size Uploaded Provenance Actions
abtest_framework-0.1.0.tar.gz 5.3 kB Without provenance Details for abtest_framework-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for abtest-framework 0.1.0
File Interpreter ABI Platform Actions
abtest_framework-0.1.0-py3-none-any.whl Python 3 none any Details for abtest_framework-0.1.0-py3-none-any.whl

Total release size: 10.1 kB

Release files / abtest_framework-0.1.0.tar.gz

Download URL abtest_framework-0.1.0.tar.gz
Size 5.3 kB
Tags Source
SHA-256 checksum
How to use checksums
ca9911afd877909b0a4c70837f5d06c7a142726b485874bff379fbed301516d4
BLAKE2b-256 checksum
How to use checksums
8e4d28b5f4b42f7eaa4d42f9df0cf1c5cb334e8f2eb1bc83b9d491b30e2b7771
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.10.11

Release files / abtest_framework-0.1.0-py3-none-any.whl

Download URL abtest_framework-0.1.0-py3-none-any.whl
Size 4.8 kB
Tags Python 3
SHA-256 checksum
How to use checksums
cc4aafeeaf438e8ce45ddf0c5930e3cd9bf859d8e729a3f56527088648d3eb10
BLAKE2b-256 checksum
How to use checksums
b8d51d388d0b44e1db65ccfa36e3fcc7f9a199afe630c1104f4a1e4b272a146a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.10.11

Security for abtest-framework 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page