ep-stats
Statistical package for the experimentation platform.
It provides a general Python package and REST API that can be used to evaluate any metric in an AB test experiment.
Features
- Robust two-tailed t-test implementation with multiple p-value corrections and delta methods applied.
- Sequential evaluations allow experiments to be stopped early.
- Connect it to any data source to get either pre-aggregated or per randomization unit data.
- Simple expression language to define arbitrary metrics.
- Sample size estimation.
- REST API to integrate it as a service in experimentation portal with score cards.
Documentation
We have got a lovely documentation.
Base Example
ep-stats allows for a quick experiment evaluation. We are using sample testing data to evaluate metric Click-through Rate in experiment test-conversion.
from epstats.toolkit import Experiment, Metric, SrmCheck
experiment = Experiment(
'test-conversion',
'a',
[Metric(
1,
'Click-through Rate',
'count(test_unit_type.unit.click)',
'count(test_unit_type.global.exposure)'),
],
[SrmCheck(1, 'SRM', 'count(test_unit_type.global.exposure)')],
unit_type='test_unit_type')
# This gets testing data, use other Dao or get aggregated goals in some other way.
from epstats.toolkit.testing import TestData
goals = TestData.load_goals_agg(experiment.id)
# evaluate experiment
ev = experiment.evaluate_agg(goals)
ev contains evaluations of exposures, metrics, and checks. This will provide the following output.
ev.exposures:
| exp_id | exp_variant_id | exposures |
|---|---|---|
| test-conversion | a | 21 |
| test-conversion | b | 26 |
ev.metrics:
| exp_id | metric_id | metric_name | exp_variant_id | count | mean | std | sum_value | confidence_level | diff | test_stat | p_value | confidence_interval | standard_error | degrees_of_freedom |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| test-conversion | 1 | Click-through Rate | a | 21 | 0.238095 | 0.436436 | 5 | 0.95 | 0 | 0 | 1 | 1.14329 | 0.565685 | 40 |
| test-conversion | 1 | Click-through Rate | b | 26 | 0.269231 | 0.452344 | 7 | 0.95 | 0.130769 | 0.223152 | 0.82446 | 1.18137 | 0.586008 | 43.5401 |
ev.checks:
| exp_id | check_id | check_name | variable_id | value |
|---|---|---|---|---|
| test-conversion | 1 | SRM | p_value | 0.465803 |
| test-conversion | 1 | SRM | test_stat | 0.531915 |
| test-conversion | 1 | SRM | confidence_level | 0.999000 |
Installation
You can install this package via pip.
pip install ep-stats
Running
You can run a testing version of ep-stats via
python -m epstats
Then, see Swagger on http://localhost:8080/docs for API documentation.
Contributing
To get started locally, you can clone the repo and quickly get started using the Makefile.
git clone https://github.com/avast/ep-stats.git
cd ep-stats
make install-dev
It sets a new virtual environment .venv in ./.venv using .venv, installs all development dependencies, and sets pre-commit git hooks to keep the code neatly formatted with ruff.
To run tests, you can use Makefile as well.
uv sync # activate python environment
make check
To run a development version of ep-stats do
uv run python -m epstats
Documentation
To update documentation run
uv run mkdocs gh-deploy
It updates documentation in GitHub pages stored in branch gh-pages.
Inspiration
Software engineering practices of this package have been heavily inspired by marvelous calmcode.io site managed by Vincent D. Warmerdam.
1 maintainer
Metadata
1 maintainer
Release files for ep-stats 2.5.10
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| ep_stats-2.5.10.tar.gz | 1.5 MB | Without provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| ep_stats-2.5.10-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 1.5 MB
Release files / ep_stats-2.5.10.tar.gz
| Download URL | ep_stats-2.5.10.tar.gz |
|---|---|
| Size | 1.5 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
b9429c52ba81baba25967c8b7c15700cf0c3bc2c8283d2fb2b945f9b5aa15d6c
|
|
BLAKE2b-256 checksum How to use checksums |
17526e96b6bdcdd702e8fc22d101efc9d4a77f338b6ed0547b5f452605f09428
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.12.9 {"installer":{"name":"uv","version":"0.12.9","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / ep_stats-2.5.10-py3-none-any.whl
| Download URL | ep_stats-2.5.10-py3-none-any.whl |
|---|---|
| Size | 63.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
7d5699ecd87d9beb9d9b4d835d6e7aa9d682125756cc0ceeafd408c1f5b1553b
|
|
BLAKE2b-256 checksum How to use checksums |
fdc66e9669c2ad2daef2140f6cd5cc6145eac7b2e7bf90639c6ed3d3235cdddf
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.12.9 {"installer":{"name":"uv","version":"0.12.9","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Security for ep-stats 2.5.10
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
