Basic example

from omoment import OMeanVar
import numpy as np
import pandas as pd

rng = np.random.default_rng(12354)
g = rng.integers(low=0, high=10, size=1000)
x = g + rng.normal(loc=0, scale=10, size=1000)
w = rng.exponential(scale=1, size=1000)

# calculate overall moments
OMeanVar.compute(x, w)
# should give: OMeanVar(mean=4.6, var=108, weight=1.08e+03)

# or calculate moments for every group
df = pd.DataFrame({'g': g, 'x': x, 'w': w})
omvs = df.groupby('g').apply(OMeanVar.of_frame, x='x', w='w')

# and combine group moments to obtain the same overall results
OMeanVar.combine(omvs)

# addition is also supported
omvs.loc[0] + omvs.loc[1]

At the moment, univariate and bivariate distributions are supported. Bivariate distributions allow for fast linear regression with two variables (and constant) calculation. Even multivariate distributions can be efficiently processed in a similar fashion, so the support for them might be added in the future. Moments of multivariate distributions would also allow for linear regression estimation and other statistical methods (such as PCA or regularized regression) to be calculated in a single pass through large distributed datasets.

Similar packages

OMoment package aims for fast calculation of weighted distribution moments (mean and variance at the moment), great compatibility with numpy and pandas and suitability for distributed datasets (composability of results). I have not found a package that would satisfy this, even though similar packages indeed exist.

import numpy as np
from omoment import OMeanVar
from runstats import Statistics
import time

rng = np.random.Generator(np.random.PCG64(12345))
x = rng.normal(size=1_000_000)

start = time.time()
omv = OMeanVar.compute(x)
end = time.time()
print(f'{end - start:.3g} seconds')
# 0.0146 seconds

start = time.time()
st = Statistics(x)
end = time.time()
print(f'{end - start:.3g} seconds')
# 2.83 seconds

Documentation

• https://protivinsky.github.io/omoment/index.html