sample-size

This python project is a helper package that uses power analysis to calculate required sample size for any experiment.

Script Usage Guide

Sample size script lets you get the sample size estimation easily by providing metric inputs.

Requirements

Please make sure you have Python 3 installed before using the script.

Verify Python was installed

python -V # python version should >=3.10, <3.13

Verify pip was installed

pip -V 

Install the package

pip install sample-size
pip show sample-size # verify package was installed

Start using the script

run-sample-size will prompt required questions for you to enter the input it needs

run-sample-size

graph TD
    A(Alpha) --> B(Variants)
    B --> C(Metric Type)
    C --> D(Metadata)
    D --> E(MDE)
    E --> G(Alternative)
    G --> F{{Register another metric?}}
    F --> C & H(Sample Size)

Script Constraints

• This package supports

  • Single and multiple metrics per calculation
  • Multiple cohorts, i.e. more than one treatment variant, per calculation
  • Metric types: Boolean, Numeric, and Ratio

• Default statistical power (80%) is used in run-sample-size all the time

• Input constraints

  • alpha: (0, 0.4]
  • probability (Boolean Metric): (0, 1)
  • variance (Numeric and Ratio Metrics): [0, )
  • registered metrics: [1, ]
  • variants: [2, ]

  Please be aware that we are running simulations many times when calculating sample size for multiple metrics or variants. Therefore, too many cohorts or metrics will have extremely long runtime.

Contributing

All contributors and contributions are welcome! Please see the contributing docs for more information.