peekproof

A/B tests you can check early without fooling yourself.

If you check an A/B test every day and stop the first time p < 0.05, your false positive rate is no longer 5%. With daily checks over two weeks it is about 22%. peekproof gives you the standard fix, group sequential testing, plus the tools around it.

pip install peekproof

Plan a test, then check it as often as you planned

import peekproof

test = peekproof.plan(n_looks=5)      # five evenly spaced looks, 5% false positives overall
print(test.table())

 look  info_fraction  z_boundary  nominal_p  alpha_spent
    1            0.2      4.8769     0.0000       0.0000
    2            0.4      3.3570     0.0008       0.0008
    3            0.6      2.6803     0.0074       0.0076
    4            0.8      2.2898     0.0220       0.0244
    5            1.0      2.0310     0.0423       0.0500

The bar is very high early and close to an ordinary test at the end. At each look, pass the counts so far:

test.check(look=2, x_a=300, n_a=2000, x_b=360, n_b=2000)
# {'z': 2.556, 'boundary': 3.357, 'nominal_p': 0.0106, 'stop': False, 'decision': 'keep going', ...}

test.check(look=5, x_a=760, n_a=5000, x_b=880, n_b=5000)
# {'z': 3.241, 'boundary': 2.031, 'stop': True, 'decision': 'stop: B is higher than A', ...}

At look 2 an ordinary test would have called a winner (p = 0.011). The plan says keep going.

Options: spending="obf" (O'Brien-Fleming-type, the default) or spending="pocock", any alpha, and uneven looks with peekproof.plan(info_fracs=[0.25, 0.5, 1.0]).

What else is in it

peekproof.sample_size_per_arm(0.19, -0.01)        # 23665 users per group to detect a 1-point drop
peekproof.srm_test(44700, 45489)                  # sample ratio mismatch check
peekproof.two_prop_ztest(8502, 44700, 8279, 45489)  # difference, z, p-value, confidence interval

# See the peeking problem for yourself: false positives vs number of looks
peekproof.simulate.fpr_vs_looks(0.19, 45000, looks=(1, 5, 14))
#  looks  false_positive_rate
#      1                0.048
#      5                0.141
#     14                0.223

How it is checked

• Boundaries are computed by numerical integration, not simulation, and match the published O'Brien-Fleming values from gsDesign / ldbounds to three decimals.
• Simulations of 300,000 tests with no real effect confirm the overall false positive rate stays at 5% for both spending functions, including with uneven looks.
• 38 tests in the repository cover the statistics against scipy, published values and simulation.

Limits

• Two groups and yes/no outcomes (conversion, retention). No revenue-style metrics yet.
• Looks are planned in advance. If you need to look at any time, use the mSPRT functions in peekproof.sequential, which trade some power for that freedom.

Where it came from

The package grew out of an analysis of the Cookie Cats mobile game A/B test (90,189 players). The full analysis and write-up are in the repository, and there is an interactive app with a peeking simulator.

References

• Lan, DeMets (1983). Discrete sequential boundaries for clinical trials. Biometrika.
• Jennison, Turnbull (2000). Group Sequential Methods with Applications to Clinical Trials.
• Johari, Pekelis, Walsh (2017). Peeking at A/B Tests: Why it matters, and what to do about it. KDD.

MIT License.