spark-bestfit

Modern distribution fitting library with pluggable backends (Spark, Ray, Local)

Efficiently fit ~90 scipy.stats distributions to your data using parallel processing. Supports Apache Spark for production clusters, Ray for ML workflows, or local execution for development.

Features

• Parallel Processing: Spark, Ray, or local thread backends
• ~90 Continuous + 16 Discrete Distributions
• Multiple Metrics: K-S, A-D, SSE, AIC, BIC
• Bounded Fitting: Truncated distributions with natural bounds
• Heavy-Tail Detection: Warns when data may need special handling
• Gaussian Copula: Correlated multi-column sampling
• Model Serialization: Save/load to JSON or pickle
• FitterConfig Builder: Fluent API for complex configurations

Full feature list at spark-bestfit.readthedocs.io

Installation

pip install spark-bestfit              # Core (BYO Spark)
pip install spark-bestfit[spark]       # With PySpark
pip install spark-bestfit[ray]         # With Ray
pip install spark-bestfit[plotting]    # With visualization

Quick Start

from spark_bestfit import DistributionFitter
import numpy as np
from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()
data = np.random.normal(loc=50, scale=10, size=10_000)
df = spark.createDataFrame([(float(x),) for x in data], ["value"])

fitter = DistributionFitter(spark)
results = fitter.fit(df, column="value")

best = results.best(n=1)[0]
print(f"Best: {best.distribution} (KS={best.ks_statistic:.4f})")

Without Spark:

from spark_bestfit import DistributionFitter, LocalBackend
import pandas as pd

df = pd.DataFrame({"value": np.random.normal(50, 10, 1000)})
fitter = DistributionFitter(backend=LocalBackend())
results = fitter.fit(df, column="value")

Backends

See Backend Guide for configuration details.

Compatibility

Ranges are what pip may install; the version in brackets is what pip picked with pyspark[sql]==<patch> for the newest patch of each minor (3.5.9, 4.0.5, 4.1.4, 4.2.1, 4.3.0) in November 2026. pip install spark-bestfit[spark] picks matching versions on its own, except on Spark 3.5: only pyspark[sql] caps NumPy below 2, so pin numpy<2 there. A weekly canary tests every Spark minor from 3.5 up.

Documentation

Full documentation at spark-bestfit.readthedocs.io:

• Quickstart Guide
• Backend Guide
• Features
• Performance & Scaling
• API Reference

Contributing

Contributions welcome! See Contributing Guide.

License

MIT License - see LICENSE for details.