Confidence sequences and uniform boundaries
Project description
Confidence sequences and uniform boundaries
This library supports calculation of uniform boundaries, confidence sequences, and alwaysvalid pvalues. These constructs are useful in sequential A/B testing, bestarm identification, and other sequential statistical procedures. The library is written in C++ and Python with a full Python interface and partial R interface. The main references are

Howard, S. R., Ramdas, A., McAuliffe, J., and Sekhon, J. (2021), Timeuniform, nonparametric, nonasymptotic confidence sequences, The Annals of Statistics, 49(2), 10551080.

Howard, S. R. and Ramdas, A. (2021), Sequential estimation of quantiles with applications to A/Btesting and bestarm identification, Bernoulli, to appear.

WaudbySmith, I. and Ramdas, A. (2021), Estimating means of bounded random variables by betting, preprint, arXiv:2010.09686.

WaudbySmith, I. and Ramdas, A. (2020), Confidence sequences for sampling without replacement, NeurIPS, 33.
This library is in earlystage development and should not be considered stable. Automated tests run on Python 3.7, 3.8, 3.9 and 3.10 on the latest Ubuntu and macOS.
The C++ implementation requires a compiler with C++14 to build the package, as well as the Boost C++ headers.
In the Python package, functions are split across modules by topic, as detailed below. In the R package, all functions mentioned below are exported in a single namespace.
Installing the python package
Run pip install confseq
at the command line.
Installing the R package
Run the following in the R console:
install.packages('devtools')
devtools::install_github('gostevehoward/confseq/r_package')
Demos
Estimating average treatment effect in a randomized trial
demo/ate_demo.py
illustrates how to compute a confidence sequence for average
treatment effect in a randomized trial with bounded potential outcomes, along
with an alwaysvalid pvalue sequence. The method is based on Corollary 2 of the
paper and uses the gammaexponential mixture boundary. This demo requires
numpy
and pandas
.
Quantile confidence sequences
demo/quantiles.py
illustrates how to use some of the included boundaries to
construct confidence sequences for quantiles based on a stream of
i.i.d. samples. The file includes a function to estimate a single, fixed
quantile, as well as a function to estimate all quantiles simultaneously, with
error control uniform over quantiles and time.
Uniform boundaries
The confseq.boundaries
Python module implements several uniform boundaries
from the confidence sequences paper.

There are four mixture boundaries. These are implemented by the functions
<TYPE>_log_mixture()
and<TYPE>_mixture_bound()
, where<TYPE>
is one ofnormal
(Propositions 4 and 5),gamma_exponential
(Proposition 8),gamma_poisson
(Proposition 9), orbeta_binomial
(Propositions 6 and 7).<TYPE>_log_mixture(s, v, ...)
returns the logarithm of the mixture supermartingale when called with S_t, the martingale, and V_t, the intrinsic time process. The reciprocal of the exponential of this value is an alwaysvalid pvalue. These functions are denoted log(m(s,v)) in the paper.<TYPE>_mixture_bound(v, alpha, ...)
returns the uniform boundary with crossing probability at most alpha, evaluated at intrinsic time v.
Each function takes another required argument
v_opt
and an optional argumentalpha_opt=0.05
. These arguments are used to set the tuning parameter for each mixture, denoted by rho or r in the paper, optimizing the uniform boundary with crossing probabilityalpha_opt
for intrinsic timev_opt
. Such tuning is discussed in section 3.5 of the paper.The gammaexponential and gammaPoisson mixtures also require a scale parameter
c
. The betabinomial mixture requires range parametersg
andh
. Finally, thenormal_*
andbeta_binomial_*
functions accept an optional boolean parameteris_one_sided
which isTrue
by default. IfFalse
, the twosided variants of these mixtures are used (Propositions 4 and 6). 
The polynomial stitching boundary (see Theorem 1 and the subsequent example) is implemented by
poly_stitching_bound
. Besidesv
andalpha
, this function requires the tuning parameterv_min
as well as optional parametersc
,s
, andeta
, all documented in the paper. 
This module also includes a
bernoulli_confidence_interval
function which computes confidence sequences for the mean of any distribution with bounded support by making use of the subBernoulli condition. Observations must be scaled so that the support is within the unit interval [0, 1].
All functions accept NumPy arrays in Python or vectors in R and perform vectorized operations.
Quantile bounds
The confseq.quantiles
Python module implements two quantileuniform confidence
sequences from the quantile paper.
empirical_process_lil_bound
is based on Theorem 2, and can be used to construct iteratedlogarithmrate confidence sequences for quantiles in which the confidence radius (in quantile space) is constant for all quantiles. This can also be used run the sequential KolmogorovSmirnov test described in section 7.2.double_stitching_bound
is based on Theorem 3, and can be used to construct confidence sequences for quantiles in which the confidence radius (in quantile space) varies, getting smaller for extreme quantiles close to zero and one.
Finally, quantile_ab_p_value
implements the twosided sequential test of the
hypothesis that two populations have equal values for some quantile, based on
Theorem 5. The theorem covers tests of null hypothesis other than equality, as
well as onesided tests, but these are not yet implemented.
C++ library
The main underlying implementation is in a singlefile, headeronly C++ library in
src/confseq/uniform_boundaries.h
. The top of the file defines a simplified
interface mirroring the Python interface described above. Below that is an
objectoriented interface useful for more involved work. The
confseq.boundaries
Python module is a wrapper generated by
pybind11. The R package uses
Rcpp.
Additional python modules
Some implementations (such as bettingbased or withoutreplacement confidence sequences) are only available in Python at the moment. Specifically, these include the implementations of
src/confseq/betting.py
src/confseq/betting_strategies.py
src/confseq/conjmix_bounded.py
, andsrc/confseq/cs_plots.py
.
If you would like to help create an R interface for these methods, it would be appreciated!
Unit tests
C++
make C /path/to/confseq/test runtests
Python (with random tests)
pytest ignore=test/googletest1.8.1/
Python (without random tests)
pytest m "not random" ignore=test/googletest1.8.1/
Citing this software
Howard, S. R., WaudbySmith, I. and Ramdas, A. (2019), ConfSeq: software for confidence sequences and uniform boundaries, https://github.com/gostevehoward/confseq [Online; accessed <today>].
@Misc{,
author = {Steven R. Howard, Ian WaudbySmith, and Aaditya Ramdas},
title = {{ConfSeq}: software for confidence sequences and uniform boundaries},
year = {2021},
url = "https://github.com/gostevehoward/confseq",
note = {[Online; accessed <today>]}
}
Project details
Release history Release notifications  RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for confseq0.0.11pp38pypy38_pp73manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm  Hash digest  

SHA256  0fbc01b01b3a7a30f79af274002c4ff2d5525392c2fcec62a7e23681d55f595f 

MD5  bffa689e53a5ada2abb94cd1775c00d6 

BLAKE2b256  5a7d71712f84685e1b7f4439f5ebc230d670f737133b86581e268a6903406db9 
Hashes for confseq0.0.11pp38pypy38_pp73macosx_10_9_x86_64.whl
Algorithm  Hash digest  

SHA256  54d4a87505ca7beec07b15d595d1b59b63c474e1a748e476052e035657cd4f0a 

MD5  db27270d7c3feb7b502a96756308712a 

BLAKE2b256  1b264961b802ae68f76dca2fd69a59e151e531ffbf55120f5fbde2351ad7961c 
Hashes for confseq0.0.11pp37pypy37_pp73manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm  Hash digest  

SHA256  09d4c8f933f61305b290b57739c86a1253ee720379347c1094579ccc466fc9c7 

MD5  c57bd27bc63c1c12865be0afba22e0c4 

BLAKE2b256  bca912fdbe8f73c6d5cc3b5c88e0bac5177222bc73a2f8c1cc64a462a2ad6fe0 
Hashes for confseq0.0.11pp37pypy37_pp73macosx_10_9_x86_64.whl
Algorithm  Hash digest  

SHA256  df2db9fc06b5395b370303ce4d73fcf86834d4a848ec745989b671d3a366863c 

MD5  5126dfd03827fd10dd31238a44815f0c 

BLAKE2b256  2d6e6e3978cfbf81c6137bef40b32dc65d00065a5471d9371a4daf57c4f84380 
Hashes for confseq0.0.11cp310cp310manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm  Hash digest  

SHA256  123c4e5b4a3c6f84b2164b382e004499bfaf2e7d879d1cd6cf5098fc64067860 

MD5  7e883a23962530d1564fec7b480a2a70 

BLAKE2b256  241e92c55d59cbbdfcdd3dc4069ec53ad021b7f1125b85a75ae722ce03058e88 
Hashes for confseq0.0.11cp310cp310macosx_10_9_universal2.whl
Algorithm  Hash digest  

SHA256  91e26f12e353948e41cadb4d6f229e9763bfe53732ed729a1d30569b9ce1d702 

MD5  677dc724f2143427025e4053fa208aed 

BLAKE2b256  35eb218239678a20bf793cb1ab101c2cd6c15f3a3f2f97da91fe6b6f06b7e739 
Hashes for confseq0.0.11cp39cp39manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm  Hash digest  

SHA256  25c018b235f2a0c13d2cf6206e74d3cd348e1262f519ebfb3edc541f504829c6 

MD5  2a657b28026cb852dbd7f47a050d7b37 

BLAKE2b256  b70b81941af3b7bad2652f96242dfebac9bbd3911e064490ae11193cfac3cc02 
Hashes for confseq0.0.11cp39cp39macosx_10_9_universal2.whl
Algorithm  Hash digest  

SHA256  2f792fe15e3c188865abf7c0a96bfffa410d7d49df52b92cbb1e3bb6fcc1110c 

MD5  5bdf35e82d53d5ca9c51cc2f4c2b8bda 

BLAKE2b256  66c6c03ea5793ccd57d8fca37535b28c3ef4e05f3e96edaf57542519355d26f2 
Hashes for confseq0.0.11cp38cp38manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm  Hash digest  

SHA256  e297e3f46b95d0b3ee33976ede5bbdd026ade9df4dc1f345e5167ace8d27ca3d 

MD5  2be115daa7a6a9105966bda308313dc8 

BLAKE2b256  6eb887b4ab4f17c78735d3e3406b6e78cbedc27a7f053147e01936cbd667163c 
Hashes for confseq0.0.11cp38cp38macosx_10_9_universal2.whl
Algorithm  Hash digest  

SHA256  aa7131b98ecff842fd046699b6dd1561e5bf3095ca4e486e8ba6efb19948ce78 

MD5  514366a7935dfa87dd3a1edb4954577e 

BLAKE2b256  cc44fe99f807eca7f865efdc970f916936700ef6879375592ad4bd2a7e8a7455 
Hashes for confseq0.0.11cp37cp37mmanylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm  Hash digest  

SHA256  789558065f91b476bf74324a73a60046f2f80c14ea3d01f11da41c94ad4921b5 

MD5  3f0e611b83228b062dfb5ec3124db1e4 

BLAKE2b256  0e1840b2133b8d0a751505d133e13e524b0f913729db552c8cd807c0b9ade4d1 
Hashes for confseq0.0.11cp37cp37mmacosx_10_9_x86_64.whl
Algorithm  Hash digest  

SHA256  f9767933840cacd6eb590c17c522b67d452ddd7b19abce65dfb47757a9f30c98 

MD5  aa3f16d480a459569be241e8aa436863 

BLAKE2b256  528915b84bab45f29ffcbb73cee26c579d6f8dca227a94f8ac5f96fd66edba4a 