ab-stats 0.1.2

A library that provides statistical test results for A/B tests

ab-stats

ab-stats is a Python library that computes the statistics you need for A/B tests. It runs a two-sample proportion z-test for rate (proportion) differences and Welch's t-test for mean differences between control and treatment groups, and returns p-value, confidence intervals, uplift (relative change), and minimum sample size (MSS) in a pandas DataFrame.

Documentation

• README

Installation

Dependencies

ab-stats depends on:

• NumPy (>= 1.20)
• Pandas (>= 1.3)
• SciPy (>= 1.7)

Python 3.8 or newer is required.

User installation

Install from PyPI with pip:

pip install ab-stats

or with conda:

conda install -c conda-forge ab-stats

Quick start

1. Proportion (rate) difference — proportions_ztest

Pass sample sizes and success counts for control and treatment; the function runs a two-sample proportion z-test and returns uplift, confidence intervals, and minimum sample size. MSS is the sample size required for the given α and β under the assumption that the observed effect is true; it is computed post hoc and should be used as a reference only.

from ab_stats import proportions_ztest

# Control: 101 successes out of 998; Treatment: 122 successes out of 1001
df = proportions_ztest(
    control_n=998,
    control_success=101,
    treatment_n=1001,
    treatment_success=122,
    alpha=0.05,
    power=0.8,
)
print(df)

Output:

metric_formula	metric_value	delta_relative	delta_absolute	p_value	CI_relative	CI_absolute	MSS	statistic
122/1001	0.1219	20.45	0.0207	0.14162	[5.12%, 35.78%]	[-0.0069, 0.0483]	152.3% (657)	1.47

2. Mean difference — ttest_ind_welch

Pass lists of values for control and treatment; the function computes means, variances, and sample sizes internally and runs Welch's t-test.

from ab_stats import ttest_ind_welch

# Example: observation lists for control and treatment
control = [10.1, 9.8, 11.2, 10.5, 9.9, 10.8, 10.3, 11.0, 9.7, 10.4, 9.8, 10.1]  # n=12
treatment = [12.0, 11.5, 12.8, 11.9, 12.2, 12.5, 11.7, 12.1, 12.3, 11.8]  # n=10

df = ttest_ind_welch(control, treatment, alpha=0.05, power=0.8)
print(df)

Output:

metric_formula	metric_value	delta_relative	delta_absolute	p_value	CI_relative	CI_absolute	MSS	statistic	df
120/10	12.03	17.14	1.76	0.00273	[8.21%, 26.07%]	[0.65, 2.87]	45.2% (221)	3.45	18.52

3. Using with Pandas

Results are returned as a pandas DataFrame, so you can merge with other columns or filter as usual.

import pandas as pd
from ab_stats import proportions_ztest, ttest_ind_welch

# Proportion test
result_prop = proportions_ztest(1000, 100, 1000, 120)
# Use result_prop["p_value"], result_prop["CI_relative"], etc.

# Mean test (lists → means, variances, n are computed inside the function)
control_vals = [1.0, 2.0, 3.0, 4.0, 5.0]
treatment_vals = [2.0, 3.0, 4.0, 5.0, 6.0]
result_ttest = ttest_ind_welch(control_vals, treatment_vals)
# Use result_ttest["metric_value"], result_ttest["df"], etc.

License

MIT License. See LICENSE for details.