Confidence sequences and uniform boundaries
Project description
Confidence sequences and uniform boundaries
This library supports calculation of uniform boundaries, confidence sequences, and alwaysvalid pvalues. These constructs are useful in sequential A/B testing, bestarm identification, and other sequential statistical procedures. The library is written in C++ and Python with a full Python interface and partial R interface. The main references are

Howard, S. R., Ramdas, A., McAuliffe, J., and Sekhon, J. (2021), Timeuniform, nonparametric, nonasymptotic confidence sequences, The Annals of Statistics, 49(2), 10551080.

Howard, S. R. and Ramdas, A. (2021), Sequential estimation of quantiles with applications to A/Btesting and bestarm identification, Bernoulli, to appear.

WaudbySmith, I. and Ramdas, A. (2021), Estimating means of bounded random variables by betting, preprint, arXiv:2010.09686.

WaudbySmith, I. and Ramdas, A. (2020), Confidence sequences for sampling without replacement, NeurIPS, 33.
This library is in earlystage development and should not be considered stable. Automated tests run on Python 3.7, 3.8, 3.9 and 3.10 on the latest Ubuntu and macOS.
The C++ implementation requires a compiler with C++14 to build the package, as well as the Boost C++ headers.
In the Python package, functions are split across modules by topic, as detailed below. In the R package, all functions mentioned below are exported in a single namespace.
Installing the python package
Run pip install confseq
at the command line.
Installing the R package
Run the following in the R console:
install.packages('devtools') devtools::install_github('gostevehoward/confseq/r_package')
Demos
Estimating average treatment effect in a randomized trial
demo/ate_demo.py
illustrates how to compute a confidence sequence for average
treatment effect in a randomized trial with bounded potential outcomes, along
with an alwaysvalid pvalue sequence. The method is based on Corollary 2 of the
paper and uses the gammaexponential mixture boundary. This demo requires
numpy
and pandas
.
Quantile confidence sequences
demo/quantiles.py
illustrates how to use some of the included boundaries to
construct confidence sequences for quantiles based on a stream of
i.i.d. samples. The file includes a function to estimate a single, fixed
quantile, as well as a function to estimate all quantiles simultaneously, with
error control uniform over quantiles and time.
Uniform boundaries
The confseq.boundaries
Python module implements several uniform boundaries
from the confidence sequences paper.

There are four mixture boundaries. These are implemented by the functions
<TYPE>_log_mixture()
and<TYPE>_mixture_bound()
, where<TYPE>
is one ofnormal
(Propositions 4 and 5),gamma_exponential
(Proposition 8),gamma_poisson
(Proposition 9), orbeta_binomial
(Propositions 6 and 7).<TYPE>_log_mixture(s, v, ...)
returns the logarithm of the mixture supermartingale when called with S_t, the martingale, and V_t, the intrinsic time process. The reciprocal of the exponential of this value is an alwaysvalid pvalue. These functions are denoted log(m(s,v)) in the paper.<TYPE>_mixture_bound(v, alpha, ...)
returns the uniform boundary with crossing probability at most alpha, evaluated at intrinsic time v.
Each function takes another required argument
v_opt
and an optional argumentalpha_opt=0.05
. These arguments are used to set the tuning parameter for each mixture, denoted by rho or r in the paper, optimizing the uniform boundary with crossing probabilityalpha_opt
for intrinsic timev_opt
. Such tuning is discussed in section 3.5 of the paper.The gammaexponential and gammaPoisson mixtures also require a scale parameter
c
. The betabinomial mixture requires range parametersg
andh
. Finally, thenormal_*
andbeta_binomial_*
functions accept an optional boolean parameteris_one_sided
which isTrue
by default. IfFalse
, the twosided variants of these mixtures are used (Propositions 4 and 6). 
The polynomial stitching boundary (see Theorem 1 and the subsequent example) is implemented by
poly_stitching_bound
. Besidesv
andalpha
, this function requires the tuning parameterv_min
as well as optional parametersc
,s
, andeta
, all documented in the paper. 
This module also includes a
bernoulli_confidence_interval
function which computes confidence sequences for the mean of any distribution with bounded support by making use of the subBernoulli condition. Observations must be scaled so that the support is within the unit interval [0, 1].
All functions accept NumPy arrays in Python or vectors in R and perform vectorized operations.
Quantile bounds
The confseq.quantiles
Python module implements two quantileuniform confidence
sequences from the quantile paper.
empirical_process_lil_bound
is based on Theorem 2, and can be used to construct iteratedlogarithmrate confidence sequences for quantiles in which the confidence radius (in quantile space) is constant for all quantiles. This can also be used run the sequential KolmogorovSmirnov test described in section 7.2.double_stitching_bound
is based on Theorem 3, and can be used to construct confidence sequences for quantiles in which the confidence radius (in quantile space) varies, getting smaller for extreme quantiles close to zero and one.
Finally, quantile_ab_p_value
implements the twosided sequential test of the
hypothesis that two populations have equal values for some quantile, based on
Theorem 5. The theorem covers tests of null hypothesis other than equality, as
well as onesided tests, but these are not yet implemented.
C++ library
The main underlying implementation is in a singlefile, headeronly C++ library in
src/confseq/uniform_boundaries.h
. The top of the file defines a simplified
interface mirroring the Python interface described above. Below that is an
objectoriented interface useful for more involved work. The
confseq.boundaries
Python module is a wrapper generated by
pybind11. The R package uses
Rcpp.
Additional python modules
Some implementations (such as bettingbased or withoutreplacement confidence sequences) are only available in Python at the moment. Specifically, these include the implementations of
src/confseq/betting.py
src/confseq/betting_strategies.py
src/confseq/conjmix_bounded.py
, andsrc/confseq/cs_plots.py
.
If you would like to help create an R interface for these methods, it would be appreciated!
Unit tests
C++
make C /path/to/confseq/test runtests
Python (with random tests)
pytest ignore=test/googletest1.8.1/
Python (without random tests)
pytest m "not random" ignore=test/googletest1.8.1/
Citing this software
Howard, S. R., WaudbySmith, I. and Ramdas, A. (2019), ConfSeq: software for confidence sequences and uniform boundaries, https://github.com/gostevehoward/confseq [Online; accessed <today>].
@Misc{, author = {Steven R. Howard, Ian WaudbySmith, and Aaditya Ramdas}, title = {{ConfSeq}: software for confidence sequences and uniform boundaries}, year = {2021}, url = "https://github.com/gostevehoward/confseq", note = {[Online; accessed <today>]} }
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for confseq0.0.9pp38pypy38_pp73manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm  Hash digest  

SHA256  dff526b45f94d4d336b023be9bba7e8e1f93b6f31bb333720a89e0055aaaff21 

MD5  f25896973bd8c02aca8ed65bdd371481 

BLAKE2256  ee4dafaed666fafbe49d76c5342d93a4f85d4065a7688d7a119b80b86c435b80 
Hashes for confseq0.0.9pp38pypy38_pp73macosx_10_9_x86_64.whl
Algorithm  Hash digest  

SHA256  44305f30185dd2465bc32000770816926b64badbd52127187700801227e51669 

MD5  6bd7e7cb0331491a46c213b7e31aa782 

BLAKE2256  d816d56efbfd36c82745dace288da5d44090a2b6f98c2472c6a4c69039b6f9d1 
Hashes for confseq0.0.9pp37pypy37_pp73manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm  Hash digest  

SHA256  1392dfd85e462815fe6620b108616abb74e6073774b027b58e0a90553d5082eb 

MD5  8a0c08735e3254393ac0fa7d6225534b 

BLAKE2256  caa1a7553819d29e97b22b42c1dbd8f81f681334e206a46810254cff037f42d0 
Hashes for confseq0.0.9pp37pypy37_pp73macosx_10_9_x86_64.whl
Algorithm  Hash digest  

SHA256  f485c5b464f97e84bb328e18a14e9fa69a13409384a67991f9c2de336650485e 

MD5  fb92b155f5eb6679becc56be5a6900b2 

BLAKE2256  b6fff5c3ffef69f5822d59127ffc1c01bfe97821ec988fa9063e4cb13b638af2 
Hashes for confseq0.0.9cp310cp310manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm  Hash digest  

SHA256  ffea61e720f10989f7eb3aa9d78e7603ab6a2113136db3602e5124e34d900de9 

MD5  bfa86a766a0aa66fb837e6c879f444b9 

BLAKE2256  85b31a84dc6ea1750fecbaf27b1d131d9ccc20201dc971627ee920a38be4160d 
Hashes for confseq0.0.9cp310cp310macosx_10_9_universal2.whl
Algorithm  Hash digest  

SHA256  2f0debaa3629d88a025605324f25c47530e415e94a0ffca22a90ad7cf997c82b 

MD5  d4ca619eff024cfe3e94987bb639c33d 

BLAKE2256  0cf0907b41722772eaab82a7b77780bb41d159317f92efce64263a9b5ab2c4d3 
Hashes for confseq0.0.9cp39cp39manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm  Hash digest  

SHA256  4d2ce94505e66961295d0e1047dcb85ef44a04906b6afd741c0614afe6ec0fc3 

MD5  83cf5629012286477fc4d57dfb27fbdd 

BLAKE2256  1f130e921569b93b5701917133239d70b5f9dfe4313c9b52679db55b6d8b893a 
Hashes for confseq0.0.9cp39cp39macosx_10_9_universal2.whl
Algorithm  Hash digest  

SHA256  0627b5926bf72f016faceb0881d2ef1b9eb54befa4f8cf96ef6f402ed41ecd0c 

MD5  485988e7f569b93e0c3261b9dd483e85 

BLAKE2256  eda8d3099de40411c0698a2444574028b5e695c4f8363182f7c06c7147e9e9f3 
Hashes for confseq0.0.9cp38cp38manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm  Hash digest  

SHA256  adb38cb88f5a30ecb1efaa523a5c130dcf255d995f89659db4da4ca392df5e90 

MD5  3b4105f7b8792f2b6e6279b25c6f315c 

BLAKE2256  f307ba79eb1e28498288648666fe9a769a37d966b997f570aa78e2b22eafd1d4 
Hashes for confseq0.0.9cp38cp38macosx_10_9_universal2.whl
Algorithm  Hash digest  

SHA256  23949305a250f0101214441eb30efdf747d5bf3c0a4192b4e81f58aa91cc0a3a 

MD5  3431a204bad51b7ccc0b78ce115c4eb9 

BLAKE2256  22286857867d9062cee74afb39898c5dcdd85d875e07f375e618530424ba4ad6 
Hashes for confseq0.0.9cp37cp37mmanylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm  Hash digest  

SHA256  2b1529aa31e583e55134121ec4611954355dd91437cda097342780216cf3bbbe 

MD5  ea353764f9b4d4eb596a219a78e3be60 

BLAKE2256  851363e4a7bb37edf2f5f224d80f17722a636edbc66f8e20ed0cf42afceec917 
Hashes for confseq0.0.9cp37cp37mmacosx_10_9_x86_64.whl
Algorithm  Hash digest  

SHA256  81937ef69dbf50f870df0442a6b8ce6517189d9ab72ce531bd628ca7382f54bd 

MD5  f2904582f1ef0344a30c3f46d714613b 

BLAKE2256  aa7f55a10fb4c6d78662c464747a1d3cf27465c1fdfdd62abd7a87d26b532ae3 