Confidence sequences and uniform boundaries
Project description
Confidence sequences and uniform boundaries
This library supports calculation of uniform boundaries, confidence sequences, and always-valid p-values. These constructs are useful in sequential A/B testing, best-arm identification, and other sequential statistical procedures. The main reference is
S. R. Howard, A. Ramdas, J. McAuliffe, J. Sekhon. Uniform, nonparametric, non-asymptotic confidence sequences. Preprint, arXiv:1810.08240.
This library is in early-stage development and should not be considered stable. I have tested it only on Python 3.7.0 on macOS Mojave. The implementation is in C++ and a compiler with C++14 support is required to build the package.
Installing the python package
Run pip3 install confseq
.
Demos
Estimating average treatment effect in a randomized trial
demo/ate_demo.py
illustrates how to compute a confidence sequence for average
treatment effect in a randomized trial with bounded potential outcomes, along
with an always-valid p-value sequence. The method is based on Corollary 2 of the
paper and uses the gamma-exponential mixture boundary. This demo requires
numpy
and pandas
.
Uniform boundaries
The confseq.boundaries
module implements several uniform boundaries from the
paper.
-
There are four mixture boundaries. These are implemented by the functions
<TYPE>_log_mixture()
and<TYPE>_mixture_bound()
, where<TYPE>
is one ofnormal
(Propositions 4 and 5),gamma_exponential
(Proposition 8),gamma_poisson
(Proposition 9), orbeta_binomial
(Propositions 6 and 7).<TYPE>_log_mixture(s, v, ...)
returns the logarithm of the mixture supermartingale when called with S_t, the martingale, and V_t, the intrinsic time process. The reciprocal of the exponential of this value is an always-valid p-value. These functions are denoted log(m(s,v)) in the paper.<TYPE>_mixture_bound(v, alpha, ...)
returns the uniform boundary with crossing probability at most alpha, evaluated at intrinsic time v.
Each function takes another required argument
v_opt
and an optional argumentalpha_opt=0.05
. These arguments are used to set the tuning parameter for each mixture, denoted by rho or r in the paper, optimizing the uniform boundary with crossing probabilityalpha_opt
for intrinsic timev_opt
. Such tuning is discussed in section 3.5 of the paper.The gamma-exponential and gamma-Poisson mixtures also require a scale parameter
c
. The beta-binomial mixture requires range parametersg
andh
. Finally, thenormal_*
andbeta_binomial_*
functions accept an optional boolean parameteris_one_sided
which isTrue
by default. IfFalse
, the two-sided variants of these mixtures are used (Propositions 4 and 6). -
The polynomial stitching boundary (see Theorem 1 and the subsequent example) is implemented by
poly_stitching_bound
. Besidesv
andalpha
, this function requires the tuning parameterv_min
as well as optional parametersc
,s
, andeta
, all documented in the paper.
All functions accept NumPy arrays and perform vectorized operations.
C++ library
The underlying implementation is in a single-file, header-only C++ library in
src/confseq/uniform_boundaries.h
. The top of the file defines a simplified
interface mirroring the Python interface described above. Below that is an
object-oriented interface useful for more involved work. The
confseq.boundaries
Python module is a wrapper generated by
pybind11.
Unit tests
Run make -C /path/to/confseq/tests runtests
to run the C++ unit tests.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for confseq-0.0.2-cp37-cp37m-macosx_10_13_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4c6702dca6d0c97f6cb0f40d4c69824cf7e1fa30ba6bec8f9895e0edf591cefa |
|
MD5 | b49461923e48129ce16287edda58688b |
|
BLAKE2b-256 | a82f15a701b8adcf9f7904e18ef49f0209fd15b86d1cc58fc53e6d3d6d6e833c |