PyPI version

ep-stats

Statistical package for the experimentation platform.

It provides a general Python package and REST API that can be used to evaluate any metric in an AB test experiment.

Features

  • Robust two-tailed t-test implementation with multiple p-value corrections and delta methods applied.
  • Sequential evaluations allow experiments to be stopped early.
  • Connect it to any data source to get either pre-aggregated or per randomization unit data.
  • Simple expression language to define arbitrary metrics.
  • Sample size estimation.
  • REST API to integrate it as a service in experimentation portal with score cards.

Documentation

We have got a lovely documentation.

Base Example

ep-stats allows for a quick experiment evaluation. We are using sample testing data to evaluate metric Click-through Rate in experiment test-conversion.

from epstats.toolkit import Experiment, Metric, SrmCheck
experiment = Experiment(
    'test-conversion',
    'a',
    [Metric(
        1,
        'Click-through Rate',
        'count(test_unit_type.unit.click)',
        'count(test_unit_type.global.exposure)'),
    ],
    [SrmCheck(1, 'SRM', 'count(test_unit_type.global.exposure)')],
    unit_type='test_unit_type')

# This gets testing data, use other Dao or get aggregated goals in some other way.
from epstats.toolkit.testing import TestData
goals = TestData.load_goals_agg(experiment.id)

# evaluate experiment
ev = experiment.evaluate_agg(goals)

ev contains evaluations of exposures, metrics, and checks. This will provide the following output.

ev.exposures:

exp_id exp_variant_id exposures
test-conversion a 21
test-conversion b 26

ev.metrics:

exp_id metric_id metric_name exp_variant_id count mean std sum_value confidence_level diff test_stat p_value confidence_interval standard_error degrees_of_freedom
test-conversion 1 Click-through Rate a 21 0.238095 0.436436 5 0.95 0 0 1 1.14329 0.565685 40
test-conversion 1 Click-through Rate b 26 0.269231 0.452344 7 0.95 0.130769 0.223152 0.82446 1.18137 0.586008 43.5401

ev.checks:

exp_id check_id check_name variable_id value
test-conversion 1 SRM p_value 0.465803
test-conversion 1 SRM test_stat 0.531915
test-conversion 1 SRM confidence_level 0.999000

Installation

You can install this package via pip.

pip install ep-stats

Running

You can run a testing version of ep-stats via

python -m epstats

Then, see Swagger on http://localhost:8080/docs for API documentation.

Contributing

To get started locally, you can clone the repo and quickly get started using the Makefile.

git clone https://github.com/avast/ep-stats.git
cd ep-stats
make install-dev

It sets a new virtual environment .venv in ./.venv using .venv, installs all development dependencies, and sets pre-commit git hooks to keep the code neatly formatted with ruff.

To run tests, you can use Makefile as well.

uv sync  # activate python environment
make check

To run a development version of ep-stats do

uv run python -m epstats

Documentation

To update documentation run

uv run mkdocs gh-deploy

It updates documentation in GitHub pages stored in branch gh-pages.

Inspiration

Software engineering practices of this package have been heavily inspired by marvelous calmcode.io site managed by Vincent D. Warmerdam.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for ondraz from gravatar.com ondraz

Credits

Author: Ondra Zahradník

Requires

Python >=3.11

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

ep_stats-2.5.10.tar.gz (1.5 MB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

ep_stats-2.5.10-py3-none-any.whl (63.1 kB view details)

Uploaded Python 3

File details

Details for the file ep_stats-2.5.10.tar.gz.

File metadata

  • Download URL: ep_stats-2.5.10.tar.gz
  • Upload date:
  • Size: 1.5 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: uv/0.12.9 {"installer":{"name":"uv","version":"0.12.9","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

File hashes

Hashes for ep_stats-2.5.10.tar.gz
Algorithm Hash digest
SHA256 b9429c52ba81baba25967c8b7c15700cf0c3bc2c8283d2fb2b945f9b5aa15d6c
MD5 d915fbce50c3a932a15f9f759ee2a9d5
BLAKE2b-256 17526e96b6bdcdd702e8fc22d101efc9d4a77f338b6ed0547b5f452605f09428

See more details on using hashes here.

File details

Details for the file ep_stats-2.5.10-py3-none-any.whl.

File metadata

  • Download URL: ep_stats-2.5.10-py3-none-any.whl
  • Upload date:
  • Size: 63.1 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: uv/0.12.9 {"installer":{"name":"uv","version":"0.12.9","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

File hashes

Hashes for ep_stats-2.5.10-py3-none-any.whl
Algorithm Hash digest
SHA256 7d5699ecd87d9beb9d9b4d835d6e7aa9d682125756cc0ceeafd408c1f5b1553b
MD5 f5428fa12bd3f878b4465bcd49461be7
BLAKE2b-256 fdc66e9669c2ad2daef2140f6cd5cc6145eac7b2e7bf90639c6ed3d3235cdddf

See more details on using hashes here.

Release history Release notifications | RSS feed

This release

2.5.10 This release

2 files

2.5.9

2 files

2.5.8

2 files

2.5.7

2 files

2.5.6

2 files

2.5.5

2 files

2.5.4

2 files

2.5.3

2 files

2.5.2

2 files

2.5.1

2 files

2.5.0

2 files

2.4.0

2 files

2.3.6

2 files

2.3.5

2 files

2.3.4

2 files

2.3.3

2 files

2.3.2

2 files

2.3.1

2 files

2.3.0

2 files

2.2.7

2 files

2.2.6

2 files

2.2.5

2 files

2.2.4

2 files

2.2.3

2 files

2.2.2

2 files

2.2.1

2 files

2.2.0

2 files

2.1.0

2 files

2.0.0

2 files

1.7.5

2 files

1.7.4

2 files

1.7.3

2 files

1.7.2

2 files

1.7.1

2 files

1.7.0

2 files

1.6.0

2 files

1.5.1

2 files

1.5.0

2 files

1.4.0

2 files

1.3.2

2 files

1.3.1

2 files

1.3.0

2 files

1.2.2

2 files

1.2.1

2 files

1.2.0

2 files

1.1.1

2 files

1.1.0

2 files

1.0.2

2 files

1.0.1

2 files

1.0.0

2 files

0.6.0

2 files

0.5.9

2 files

0.5.8

2 files

0.5.6

2 files

0.5.5

2 files

0.5.4

2 files

0.5.3

2 files

0.5.2

2 files

0.5.1

2 files

0.4.1

2 files

0.3.6

2 files

0.3.5

2 files

0.3.3

2 files

0.3.2

2 files

0.3.1

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page