ab-metric-eval 0.1.1

Statistical analysis tools for A/B experiment metric evaluation

ab-metric-eval

Statistical analysis tools for A/B experiment metric evaluation.

ab-metric-eval provides two general-purpose functions for analysing experiment results from any pandas DataFrame — no Spark or platform-specific dependencies required.

Features

• Experiment summaries — group-level enrollment counts, crossover and exclusion rates, and a daily enrollment timeline chart
• Hypothesis testing — Welch's t-test (continuous) and proportions z-test (binary), with configurable significance level
• CUPED variance reduction — optional pre-exposure covariate adjustment to increase statistical power
• Winsorization — symmetric or asymmetric outlier clipping for continuous metrics
• Pre-exposure bias testing — diagnostic check for randomisation imbalance before drawing conclusions
• Flexible imputation — zero-fill or drop-missing strategies for pre/post pivoting

Installation

pip install ab-metric-eval

Quick start

import pandas as pd
from ab_metric_eval import generate_experiment_summary, analyze_experiment_metric

# df is a pandas DataFrame with one or two rows per unit
# (pre-exposure and post-exposure), containing at minimum:
#   experiment_group, analysis_unit_id, analysis_exclusion_flag,
#   crossover_flag, first_exposure_utc_date

# 1. Summarise enrollment and group balance
summary_df, enrollment_df, fig = generate_experiment_summary(df)

# 2. Run a hypothesis test on a continuous metric
summary, results, fig, winsor_summary = analyze_experiment_metric(
    df=df,
    metric_column="revenue",
    pre_exposure_eval_column="pre_revenue",   # optional, enables CUPED + bias test
    metric_type="continuous",
    winsorize_percentile=0.99,
    apply_cuped=True,
    alpha=0.05,
)

API reference

generate_experiment_summary(df)

Generates group-level summary statistics and a daily enrollment timeline.

Parameters

Name	Type	Description
df	pd.DataFrame	Per-unit experiment data. Required columns: experiment_group, name, analysis_unit_id, analysis_exclusion_flag, crossover_flag, first_exposure_utc_date. Optional: metric_time_window.

Returns (summary_df, enrollment_df, fig) — group summary, daily enrollment counts, matplotlib figure.

---

analyze_experiment_metric(df, metric_column, ...)

Runs a full statistical analysis on a single metric, including optional pre-exposure bias testing and CUPED adjustment.

Parameters

Name	Type	Default	Description
df	pd.DataFrame	—	Per-unit data (pre- and/or post-exposure rows).
metric_column	str	—	Post-exposure metric column to test.
pre_exposure_eval_column	str | None	None	Pre-exposure metric column. Required for bias test and CUPED.
metric_type	str	'continuous'	'continuous' or 'binary'.
winsorize_percentile	float | tuple | None	None	Winsorization bounds. A single float (e.g. 0.99) clips symmetrically at the (1%, 99%) percentiles. A tuple (lower, upper) allows asymmetric clipping; use None for either end to skip that tail.
alpha	float	0.05	Significance level for the hypothesis test.
apply_cuped	bool	False	Apply CUPED adjustment (requires pre_exposure_eval_column; not supported for binary metrics).
imputation_strategy	str	'zero'	How to handle units missing pre- or post-exposure rows after pivoting: 'zero' imputes with 0, 'drop' removes the unit.
include_exclusion_flag	bool	False	If False, units where analysis_exclusion_flag is True are removed.
include_crossover_flag	bool	False	If False, units where crossover_flag is True are removed.
bias_test_alpha	float | None	None	Significance threshold for the pre-exposure balance diagnostic. When None, a graduated reporting scheme is used instead of a binary pass/fail.

Returns (summary_df, test_results_df, fig, winsorization_summary_df).

---

Data classes

ExperimentConfig and MetricDefinition are convenience dataclasses for grouping experiment parameters and metric definitions respectively.

from ab_metric_eval import ExperimentConfig, MetricDefinition

config = ExperimentConfig(
    expt_name="my-experiment",
    enrollment_date_start="2025-01-01",
    enrollment_date_end_incl="2025-01-31",
    analysis_date_start="2025-02-01",
    analysis_date_end_incl="2025-02-28",
    unit_type="userId",
)

metric = MetricDefinition(
    name="revenue",
    sql_expression="SUM(revenue)",
    description="Total revenue per user",
)

Source

github.com/five-blue-tiles/general-repo/tree/main/ab-metric-eval

Requirements

• Python 3.8+
• numpy >= 1.19
• pandas >= 1.0
• scipy >= 1.5
• statsmodels >= 0.12
• matplotlib >= 3.3
• seaborn >= 0.11

License

MIT