Skip to main content

Confidence sequences and uniform boundaries

Project description

Confidence sequences and uniform boundaries

This library supports calculation of uniform boundaries, confidence sequences, and always-valid p-values. These constructs are useful in sequential A/B testing, best-arm identification, and other sequential statistical procedures. The library is written in C++ and Python with a full Python interface and partial R interface. The main references are

This library is in early-stage development and should not be considered stable. Automated tests run on Python 3.7, 3.8, 3.9 and 3.10 on the latest Ubuntu and macOS.

The C++ implementation requires a compiler with C++14 to build the package, as well as the Boost C++ headers.

In the Python package, functions are split across modules by topic, as detailed below. In the R package, all functions mentioned below are exported in a single namespace.

Installing the python package

Run pip install confseq at the command line.

Installing the R package

Run the following in the R console:

install.packages('devtools')
devtools::install_github('gostevehoward/confseq/r_package')

Demos

Estimating average treatment effect in a randomized trial

demo/ate_demo.py illustrates how to compute a confidence sequence for average treatment effect in a randomized trial with bounded potential outcomes, along with an always-valid p-value sequence. The method is based on Corollary 2 of the paper and uses the gamma-exponential mixture boundary. This demo requires numpy and pandas.

Quantile confidence sequences

demo/quantiles.py illustrates how to use some of the included boundaries to construct confidence sequences for quantiles based on a stream of i.i.d. samples. The file includes a function to estimate a single, fixed quantile, as well as a function to estimate all quantiles simultaneously, with error control uniform over quantiles and time.

Uniform boundaries

The confseq.boundaries Python module implements several uniform boundaries from the confidence sequences paper.

  • There are four mixture boundaries. These are implemented by the functions <TYPE>_log_mixture() and <TYPE>_mixture_bound(), where <TYPE> is one of normal (Propositions 4 and 5), gamma_exponential (Proposition 8), gamma_poisson (Proposition 9), or beta_binomial (Propositions 6 and 7).

    • <TYPE>_log_mixture(s, v, ...) returns the logarithm of the mixture supermartingale when called with S_t, the martingale, and V_t, the intrinsic time process. The reciprocal of the exponential of this value is an always-valid p-value. These functions are denoted log(m(s,v)) in the paper.
    • <TYPE>_mixture_bound(v, alpha, ...) returns the uniform boundary with crossing probability at most alpha, evaluated at intrinsic time v.

    Each function takes another required argument v_opt and an optional argument alpha_opt=0.05. These arguments are used to set the tuning parameter for each mixture, denoted by rho or r in the paper, optimizing the uniform boundary with crossing probability alpha_opt for intrinsic time v_opt. Such tuning is discussed in section 3.5 of the paper.

    The gamma-exponential and gamma-Poisson mixtures also require a scale parameter c. The beta-binomial mixture requires range parameters g and h. Finally, the normal_* and beta_binomial_* functions accept an optional boolean parameter is_one_sided which is True by default. If False, the two-sided variants of these mixtures are used (Propositions 4 and 6).

  • The polynomial stitching boundary (see Theorem 1 and the subsequent example) is implemented by poly_stitching_bound. Besides v and alpha, this function requires the tuning parameter v_min as well as optional parameters c, s, and eta, all documented in the paper.

  • This module also includes a bernoulli_confidence_interval function which computes confidence sequences for the mean of any distribution with bounded support by making use of the sub-Bernoulli condition. Observations must be scaled so that the support is within the unit interval [0, 1].

All functions accept NumPy arrays in Python or vectors in R and perform vectorized operations.

Quantile bounds

The confseq.quantiles Python module implements two quantile-uniform confidence sequences from the quantile paper.

  • empirical_process_lil_bound is based on Theorem 2, and can be used to construct iterated-logarithm-rate confidence sequences for quantiles in which the confidence radius (in quantile space) is constant for all quantiles. This can also be used run the sequential Kolmogorov-Smirnov test described in section 7.2.
  • double_stitching_bound is based on Theorem 3, and can be used to construct confidence sequences for quantiles in which the confidence radius (in quantile space) varies, getting smaller for extreme quantiles close to zero and one.

Finally, quantile_ab_p_value implements the two-sided sequential test of the hypothesis that two populations have equal values for some quantile, based on Theorem 5. The theorem covers tests of null hypothesis other than equality, as well as one-sided tests, but these are not yet implemented.

C++ library

The main underlying implementation is in a single-file, header-only C++ library in src/confseq/uniform_boundaries.h. The top of the file defines a simplified interface mirroring the Python interface described above. Below that is an object-oriented interface useful for more involved work. The confseq.boundaries Python module is a wrapper generated by pybind11. The R package uses Rcpp.

Additional python modules

Some implementations (such as betting-based or without-replacement confidence sequences) are only available in Python at the moment. Specifically, these include the implementations of

  • src/confseq/betting.py
  • src/confseq/betting_strategies.py
  • src/confseq/conjmix_bounded.py, and
  • src/confseq/cs_plots.py.

If you would like to help create an R interface for these methods, it would be appreciated!

Unit tests

C++

make -C /path/to/confseq/test runtests

Python (with random tests)

pytest --ignore=test/googletest-1.8.1/

Python (without random tests)

pytest -m "not random" --ignore=test/googletest-1.8.1/

Citing this software

Howard, S. R., Waudby-Smith, I. and Ramdas, A. (2019-), ConfSeq: software for confidence sequences and uniform boundaries, https://github.com/gostevehoward/confseq [Online; accessed ].

@Misc{,
  author = {Steven R. Howard, Ian Waudby-Smith, and Aaditya Ramdas},
  title = {{ConfSeq}: software for confidence sequences and uniform boundaries},
  year = {2021--},
  url = "https://github.com/gostevehoward/confseq",
  note = {[Online; accessed <today>]}
}

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

confseq-0.0.11.tar.gz (686.7 kB view details)

Uploaded Source

Built Distributions

confseq-0.0.11-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (592.0 kB view details)

Uploaded PyPymanylinux: glibc 2.17+ x86-64

confseq-0.0.11-pp38-pypy38_pp73-macosx_10_9_x86_64.whl (246.0 kB view details)

Uploaded PyPymacOS 10.9+ x86-64

confseq-0.0.11-pp37-pypy37_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (595.1 kB view details)

Uploaded PyPymanylinux: glibc 2.17+ x86-64

confseq-0.0.11-pp37-pypy37_pp73-macosx_10_9_x86_64.whl (481.3 kB view details)

Uploaded PyPymacOS 10.9+ x86-64

confseq-0.0.11-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (306.6 kB view details)

Uploaded CPython 3.10manylinux: glibc 2.17+ x86-64

confseq-0.0.11-cp310-cp310-macosx_10_9_universal2.whl (420.3 kB view details)

Uploaded CPython 3.10macOS 10.9+ universal2 (ARM64, x86-64)

confseq-0.0.11-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (306.9 kB view details)

Uploaded CPython 3.9manylinux: glibc 2.17+ x86-64

confseq-0.0.11-cp39-cp39-macosx_10_9_universal2.whl (420.7 kB view details)

Uploaded CPython 3.9macOS 10.9+ universal2 (ARM64, x86-64)

confseq-0.0.11-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (306.4 kB view details)

Uploaded CPython 3.8manylinux: glibc 2.17+ x86-64

confseq-0.0.11-cp38-cp38-macosx_10_9_universal2.whl (420.7 kB view details)

Uploaded CPython 3.8macOS 10.9+ universal2 (ARM64, x86-64)

confseq-0.0.11-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (309.8 kB view details)

Uploaded CPython 3.7mmanylinux: glibc 2.17+ x86-64

confseq-0.0.11-cp37-cp37m-macosx_10_9_x86_64.whl (244.4 kB view details)

Uploaded CPython 3.7mmacOS 10.9+ x86-64

File details

Details for the file confseq-0.0.11.tar.gz.

File metadata

  • Download URL: confseq-0.0.11.tar.gz
  • Upload date:
  • Size: 686.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.9.16

File hashes

Hashes for confseq-0.0.11.tar.gz
Algorithm Hash digest
SHA256 b4f7723089478dda4b011a0bfe4d449f017d7ce0ec0d5039181c3e4a576bc914
MD5 c4f4957f50f603d8bfdd6fb108117920
BLAKE2b-256 683db0ccdd8cd7b3e4ae10f028f6233509cd5fd7d446091cc7785e9ac250d78f

See more details on using hashes here.

File details

Details for the file confseq-0.0.11-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for confseq-0.0.11-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 0fbc01b01b3a7a30f79af274002c4ff2d5525392c2fcec62a7e23681d55f595f
MD5 bffa689e53a5ada2abb94cd1775c00d6
BLAKE2b-256 5a7d71712f84685e1b7f4439f5ebc230d670f737133b86581e268a6903406db9

See more details on using hashes here.

File details

Details for the file confseq-0.0.11-pp38-pypy38_pp73-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for confseq-0.0.11-pp38-pypy38_pp73-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 54d4a87505ca7beec07b15d595d1b59b63c474e1a748e476052e035657cd4f0a
MD5 db27270d7c3feb7b502a96756308712a
BLAKE2b-256 1b264961b802ae68f76dca2fd69a59e151e531ffbf55120f5fbde2351ad7961c

See more details on using hashes here.

File details

Details for the file confseq-0.0.11-pp37-pypy37_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for confseq-0.0.11-pp37-pypy37_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 09d4c8f933f61305b290b57739c86a1253ee720379347c1094579ccc466fc9c7
MD5 c57bd27bc63c1c12865be0afba22e0c4
BLAKE2b-256 bca912fdbe8f73c6d5cc3b5c88e0bac5177222bc73a2f8c1cc64a462a2ad6fe0

See more details on using hashes here.

File details

Details for the file confseq-0.0.11-pp37-pypy37_pp73-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for confseq-0.0.11-pp37-pypy37_pp73-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 df2db9fc06b5395b370303ce4d73fcf86834d4a848ec745989b671d3a366863c
MD5 5126dfd03827fd10dd31238a44815f0c
BLAKE2b-256 2d6e6e3978cfbf81c6137bef40b32dc65d00065a5471d9371a4daf57c4f84380

See more details on using hashes here.

File details

Details for the file confseq-0.0.11-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for confseq-0.0.11-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 123c4e5b4a3c6f84b2164b382e004499bfaf2e7d879d1cd6cf5098fc64067860
MD5 7e883a23962530d1564fec7b480a2a70
BLAKE2b-256 241e92c55d59cbbdfcdd3dc4069ec53ad021b7f1125b85a75ae722ce03058e88

See more details on using hashes here.

File details

Details for the file confseq-0.0.11-cp310-cp310-macosx_10_9_universal2.whl.

File metadata

File hashes

Hashes for confseq-0.0.11-cp310-cp310-macosx_10_9_universal2.whl
Algorithm Hash digest
SHA256 91e26f12e353948e41cadb4d6f229e9763bfe53732ed729a1d30569b9ce1d702
MD5 677dc724f2143427025e4053fa208aed
BLAKE2b-256 35eb218239678a20bf793cb1ab101c2cd6c15f3a3f2f97da91fe6b6f06b7e739

See more details on using hashes here.

File details

Details for the file confseq-0.0.11-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for confseq-0.0.11-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 25c018b235f2a0c13d2cf6206e74d3cd348e1262f519ebfb3edc541f504829c6
MD5 2a657b28026cb852dbd7f47a050d7b37
BLAKE2b-256 b70b81941af3b7bad2652f96242dfebac9bbd3911e064490ae11193cfac3cc02

See more details on using hashes here.

File details

Details for the file confseq-0.0.11-cp39-cp39-macosx_10_9_universal2.whl.

File metadata

File hashes

Hashes for confseq-0.0.11-cp39-cp39-macosx_10_9_universal2.whl
Algorithm Hash digest
SHA256 2f792fe15e3c188865abf7c0a96bfffa410d7d49df52b92cbb1e3bb6fcc1110c
MD5 5bdf35e82d53d5ca9c51cc2f4c2b8bda
BLAKE2b-256 66c6c03ea5793ccd57d8fca37535b28c3ef4e05f3e96edaf57542519355d26f2

See more details on using hashes here.

File details

Details for the file confseq-0.0.11-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for confseq-0.0.11-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 e297e3f46b95d0b3ee33976ede5bbdd026ade9df4dc1f345e5167ace8d27ca3d
MD5 2be115daa7a6a9105966bda308313dc8
BLAKE2b-256 6eb887b4ab4f17c78735d3e3406b6e78cbedc27a7f053147e01936cbd667163c

See more details on using hashes here.

File details

Details for the file confseq-0.0.11-cp38-cp38-macosx_10_9_universal2.whl.

File metadata

File hashes

Hashes for confseq-0.0.11-cp38-cp38-macosx_10_9_universal2.whl
Algorithm Hash digest
SHA256 aa7131b98ecff842fd046699b6dd1561e5bf3095ca4e486e8ba6efb19948ce78
MD5 514366a7935dfa87dd3a1edb4954577e
BLAKE2b-256 cc44fe99f807eca7f865efdc970f916936700ef6879375592ad4bd2a7e8a7455

See more details on using hashes here.

File details

Details for the file confseq-0.0.11-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for confseq-0.0.11-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 789558065f91b476bf74324a73a60046f2f80c14ea3d01f11da41c94ad4921b5
MD5 3f0e611b83228b062dfb5ec3124db1e4
BLAKE2b-256 0e1840b2133b8d0a751505d133e13e524b0f913729db552c8cd807c0b9ade4d1

See more details on using hashes here.

File details

Details for the file confseq-0.0.11-cp37-cp37m-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for confseq-0.0.11-cp37-cp37m-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 f9767933840cacd6eb590c17c522b67d452ddd7b19abce65dfb47757a9f30c98
MD5 aa3f16d480a459569be241e8aa436863
BLAKE2b-256 528915b84bab45f29ffcbb73cee26c579d6f8dca227a94f8ac5f96fd66edba4a

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page