Skip to main content

RunStats: Computing Statistics and Regression in One Pass

RunStats is an Apache2 licensed Python module that computes statistics and regression in a single pass.

Long running systems often generate numbers summarizing performance. It could be the latency of a response or the time between requests. It’s often useful to use these numbers in summary statistics like the arithmetic mean, minimum, standard deviation, etc. When many values are generated, computing these summaries can be computationally intensive. It may even be infeasible to keep every recorded value. In such cases computing online statistics and online regression is necessary.

In other cases, you may only have one opportunity to observe all the recorded values. Python’s generators work exactly this way. Traditional methods for calculating the variance and other higher moments requires multiple passes over the data. With generators, this is not possible and so computing statistics in a single pass is necessary.

The Python RunStats module was designed for these cases by providing a pair of classes for computing summary statistics and linear regression in a single pass. Summary objects work on series which may be larger than memory or disk space permit. They may also be efficiently combined together to create aggregate measures.

Features

  • Pure-Python

  • Fully Documented

  • 100% Test Coverage

  • Numerically Stable

  • Statistics summary computes mean, variance, standard deviation, skewness, kurtosis, minimum and maximum.

  • Regression summary computes slope, intercept and correlation.

  • Developed on Python 2.7

  • Tested on CPython 2.6, 2.7, 3.2, 3.3, 3.4 and PyPy 2.5+, PyPy3 2.4+

Quickstart

Installing RunStats is simple with pip:

$ pip install runstats

You can access documentation in the interpreter with Python’s built-in help function:

>>> from runstats import Statistics, Regression
>>> help(Statistics)
>>> help(Regression)

Tutorial

The Python runstats module provides two types for computing running Statistics and Regression. The Regression object leverages Statistics internally for its calculations. Each can be initialized without arguments:

>>> from runstats import Statistics, Regression
>>> stats = Statistics()
>>> regr = Regression()

Statistics objects support three methods for modification. Use push to add values to the summary, clear to reset the summary, and sum to combine Statistics summaries:

>>> for num in range(10):
...     stats.push(num)
>>> stats.mean()
4.5
>>> stats.maximum()
9
>>> stats += stats
>>> stats.mean()
4.5
>>> stats.variance()
8.68421052631579
>>> len(stats)
20
>>> stats.clear()
>>> len(stats)
0
>>> stats.minimum() is None
True

Use the Python built-in len for the number of pushed values. Unfortunately the Python min and max built-ins may not be used for the minimum and maximum as sequences are instead expected. There are instead minimum and maximum methods which are provided for that purpose:

>>> import random
>>> random.seed(0)
>>> for __ in range(1000):
...     stats.push(random.random())
>>> len(stats)
1000
>>> min(stats)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: iteration over non-sequence
>>> stats.minimum()
0.00024069652516689466
>>> stats.maximum()
0.9996851255769114

Statistics summaries provide five measures of a series: mean, variance, standard deviation, skewness and kurtosis:

>>> stats = Statistics([1, 2, 5, 12, 5, 2, 1])
>>> stats.mean()
4.0
>>> stats.variance()
15.33333333333333
>>> stats.stddev()
3.915780041490243
>>> stats.skewness()
1.33122127314735
>>> stats.kurtosis()
0.5496219281663506

All internal calculations use Python’s float type.

Like Statistics, the Regression type supports three methods for modification: push, clear and sum:

>>> regr.clear()
>>> len(regr)
0
>>> for num in range(10):
...     regr.push(num, num + 5)
>>> len(regr)
10
>>> regr.slope()
1.0
>>> more = Regression((num, num + 5) for num in range(10, 20))
>>> total = regr + more
>>> len(total)
20
>>> total.slope()
1.0
>>> total.intercept()
5.0
>>> total.correlation()
1.0

Regression summaries provide three measures of a series of pairs: slope, intercept and correlation. Note that, as a regression, the points need not exactly lie on a line:

>>> regr = Regression([(1.2, 1.9), (3, 5.1), (4.9, 8.1), (7, 11)])
>>> regr.slope()
1.5668320150154176
>>> regr.intercept()
0.21850113956294415
>>> regr.correlation()
0.9983810791694997

Both constructors accept an optional iterable that is consumed and pushed into the summary. Note that you may pass a generator as an iterable and the generator will be entirely consumed.

All internal calculations are based entirely on the C++ code by John Cook as posted in a couple of articles:

Reference and Indices

License

Copyright 2015 Grant Jenks

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Metadata

Release files for runstats 1.4.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for runstats 1.4.0
File Size Uploaded Provenance Actions
runstats-1.4.0.tar.gz 169.2 kB Without provenance Details for runstats-1.4.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for runstats 1.4.0
File Interpreter ABI Platform Actions
runstats-1.4.0-cp27-cp27m-macosx_10_11_x86_64.whl CPython 2.7 CPython 2.7 pymalloc macOS 10.11+ x86-64 Details for runstats-1.4.0-cp27-cp27m-macosx_10_11_x86_64.whl

Total release size: 225.4 kB

Release files / runstats-1.4.0.tar.gz

Download URL runstats-1.4.0.tar.gz
Size 169.2 kB
Tags Source
SHA-256 checksum
How to use checksums
333c1da92067e58aa98a33b4f936b0df53db2d4b057483a93152aedbc81fec39
BLAKE2b-256 checksum
How to use checksums
a0ab1adc70b1fc75342577ddfc2e3376601914c2b3da2d501d5a11bbd8287861
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No

Release files / runstats-1.4.0-cp27-cp27m-macosx_10_11_x86_64.whl

Download URL runstats-1.4.0-cp27-cp27m-macosx_10_11_x86_64.whl
Size 56.2 kB
Tags CPython 2.7 CPython 2.7 pymalloc macOS 10.11+ x86-64
SHA-256 checksum
How to use checksums
357f89c6968df918084afe9f78bf22832e47b4e52f83200487b9884b9e716fc3
BLAKE2b-256 checksum
How to use checksums
ee412ad9e83db8323a587606a852bade65185080508b7acafeb9725029f4b7ca
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No

Security for runstats 1.4.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page