broom-sm

Tidy-style statistical inference for Python with statsmodels

broom-sm brings the ergonomic design of broom and the tidyverse to Python's statsmodels ecosystem. The package centers around three main verbs—stats_tidy(), stats_glance(), and stats_augment()—supplemented by bootstrapping utilities, diagnostic plots, and Bayesian helpers.

import pandas as pd
import statsmodels.api as sm
from broom_sm import stats_report

# Load data
mtcars = sm.datasets.get_rdataset("mtcars").data

# Fit once, get everything
report = mtcars.stats_report(
    formula="mpg ~ wt + hp",
    stat_type="ols"
)

# Tidy coefficient table
print(report["tidy"])
#>       term  estimate  std.error  conf.low  conf.high   statistic    p.value
#> 0  Intercept  37.227270   1.877627  33.343267  41.111273  19.826764  1.27e-17
#> 1         wt  -3.877831   0.714968  -5.355789  -2.399873  -5.423781  1.19e-05
#> 2         hp  -0.031157   0.011436  -0.054812  -0.007501  -2.724389  1.12e-02

# Model-level statistics
print(report["glance"])
#>   stat_type  nobs      llf       aic       bic  df_model  df_resid  rsquared
#> 0       ols    32 -72.54928  153.09856  158.95938       2.0      29.0  0.826783

Installation

# Core package (tidy verbs + diagnostics)
pip install broom-sm

# With visualization dependencies
pip install broom-sm[viz]

# With Bayesian bootstrap support
pip install broom-sm[bayes]

The Tidy Workflow

broom-sm is built around three core verbs that convert statsmodels objects into tidy DataFrames:

Example: Analysis of Variance

Test whether vehicle weight differs by cylinder count:

import pandas as pd
import statsmodels.api as sm

mtcars = sm.datasets.get_rdataset("mtcars").data

# Calculate observed statistic
obs_stat = mtcars.stats_anova_tidy(
    formula="wt ~ factor(cyl)",
    anova_type=2
)

# Bootstrap the null distribution
null_dist = mtcars.boot_tidy(
    formula="wt ~ factor(cyl)",
    stat_type="ols",
    n_boot=1000,
    seed=42
)

# Visualize
from broom_sm import stats_residual_plot
figures = mtcars.stats_residual_plot(["cyl"], y="wt")
figures[0][1].show()

# Calculate p-value
from scipy import stats
f_stat = obs_stat["statistic"].iloc[0]
p_value = 1 - stats.f.cdf(f_stat, obs_stat["df"].iloc[0], obs_stat["df_resid"].iloc[0])

Key Features

🔁 Tidy Verbs

All core verbs work with formulas or pre-fitted statsmodels results:

# Formula interface
df.stats_tidy("y ~ x1 + x2", stat_type="ols")

# Pre-fitted model interface
import statsmodels.formula.api as smf
model = smf.ols("y ~ x1 + x2", data=df).fit()
df.stats_tidy(model=model)

🧱 Extensible Model Registry

Support for OLS, GLMs (Poisson, Gamma, Beta, Negative Binomial), GEE, MixedLM, PHReg/Survival, and Quantile Regression. Register custom models:

from broom_sm.model_registry import ModelSpec, register_model
import statsmodels.formula.api as smf

register_model(
    "tobit",
    ModelSpec(
        fitter=lambda formula, data, **kwargs: smf.tobit(formula, data=data, **kwargs).fit(),
        stat_name="z_stat"
    )
)

📦 Bootstrapping

Built-in resampling with consistent logging:

boot = mtcars.boot_tidy(
    formula="mpg ~ wt",
    stat_type="ols",
    n_boot=500,
    seed=11
)
boot.groupby("term")["estimate"].agg(["mean", "std"])

📊 Diagnostics & Visualization

All plot helpers return Matplotlib figures (no implicit plt.show()):

# Residual diagnostics
figures = df.stats_residual_plot(["x1", "x2"], y="y")

# Influence plot
fig = df.stats_influence_plot("y ~ x1 + x2", stat_type="ols")

# Coefficient forest plot
tidy = df.stats_tidy("y ~ x1 + x2", stat_type="ols")
fig, ax = stats_coef_forest(tidy)

🧪 Robust Standard Errors

Pass cov_type, cov_kwds, family, link, or weights directly:

df.stats_tidy(
    formula="mpg ~ wt",
    stat_type="glm",
    family="binomial",
    weights=df["weights"],
    cov_type="HC3"
)

🛠️ Command-Line Interface

Quick reports from the terminal:

# Single model report
broom-sm report --data data.csv --formula 'y ~ x1 + x2' --stat-type ols

# Compare multiple models
broom-sm compare --data data.csv --stat-type ols \
  --formulas "y ~ x1" "y ~ x1 + x2"

Output defaults to JSON; pass --format csv for tabular output.

🔗 widyr Integration (R parity)

broom-sm now includes a Python port of core widyr verbs for tidy pairwise and wide-matrix workflows:

• pairwise_count, pairwise_cor, pairwise_dist, pairwise_similarity
• pairwise_pmi, pairwise_delta
• widely_svd, widely_kmeans, widely_hclust, widely, squarely
• cor_sparse

from broom_sm import pairwise_cor, widely_kmeans

# Pairwise country similarity by life expectancy trajectories
corr = pairwise_cor(gapminder, "country", "year", "lifeExp", method="pearson")

# Cluster countries in wide feature space
clusters = widely_kmeans(gapminder, "country", "year", "lifeExp", k=3, random_state=0)

All pairwise outputs use tidy columns (item1, item2, metric column), and the module is fully exported from broom_sm.__init__.

Model Coverage

Tidy Diagnostics (broom + broomExtra parity)

The package also exposes tidy wrappers for diagnostic, model-comparison, and inference helpers — many of which are parity work for R's broom and broomExtra:

See docs/audit_vs_r_broom.md for the full parity matrix (which verbs are covered, partial, or out-of-scope due to a missing statsmodels analogue).

Changelog

Version 0.2.0 — 2026-09-02

Added

• New widyr parity module with tidy pairwise/wide verbs: pairwise_count, pairwise_cor, pairwise_dist, pairwise_similarity, pairwise_pmi, pairwise_delta, widely_svd, widely_kmeans, widely_hclust, widely, squarely, and cor_sparse.
• Public exports for the full widyr surface from broom_sm.__init__.
• New integration coverage in tests/test_widyr.py for pairwise outputs, upper-triangle filtering, metric validation, sparse correlation, and clustering/SVD behavior.
• New phreg (Cox PH) support in stats_tidy/stats_glance, including synthesized nobs/aic/bic when they are missing from fitted results.
• New parity helpers: stats_kendall_tidy, stats_coeftest, stats_manova_tidy, stats_rmse, stats_roc_tidy, stats_breusch_pagan, stats_white_test, stats_dffits, stats_cooks_distance, stats_leverage, stats_crossv_kfold, and stats_crossv_mc.
• New AI workflow guide: docs/howto/ai-assistant.md.

Changed

• GitHub Actions CI is now multi-job with:
  • matrix tests on Python 3.10/3.11/3.12
  • explicit extras install (testing,viz,bayes)
  • Sphinx docs build with warnings treated as errors
  • advisory ruff and mypy checks
• Fixed CI dependency installation by removing invalid .[dev].
• stats_tidy and stats_glance now use shared coercion/synthesis helpers so numpy-backed statsmodels results are converted to robust tidy/glance output.

Documentation

• Added docs/audit_vs_r_broom.md, a detailed parity audit against broom, broomExtra, and broom.mixed.
• Updated README.md, docs/index.md, docs/howto/index.md, and CONTRIBUTING.md for the AI assistant playbook and CI expectations.
• Added parity test summary to tests/test_parity.py (34 new tests; 96 passed, coverage 93%).

Version 0.1.3 — 2026-07-13

Quality fixes, visual/plotting diagnostics testing, and coverage expansion to 96%:

• Fixed OLS Weights: Changed the direct Ordinary Least Squares fitter registration to use WLS when weights are supplied, making weights functional rather than silent placebos.
• Fixed stats_augment NaN alignment: Rewrote alignment logic to assign pandas Series directly (relying on index alignment rather than .values), avoiding length mismatches when rows are dropped. Used pre-transformed exog values for predictions.
• Optimized stats_tidy merges: Replaced consecutive DataFrame merges on the "term" column with direct coefficient construction.
• Dependency cleanup: Moved seaborn, matplotlib, and bayesian_bootstrap to optional package extras, adding guarded imports and descriptive import errors.
• Coverage expansion: Created extensive tests for visual diagnostics, CLI parameters (--index-col), fallback paths, and mocked import environments, raising line coverage to 96% with all 62 tests passing.

Version 0.1.2 — 2026-06-20

P0 fixes from the 2026-06-20 code review:

• stats_augment now validates index uniqueness for data and new_data, rejects overlapping indices, and aligns residuals / influence diagnostics position-wise for the in-sample path. The .in_sample flag is now a single boolean rather than a set-based membership test.
• prepare_fit now passes freq_weights to GLM-family fitters (Poisson, Gamma, Negative Binomial, Beta, etc.) and keeps weights for OLS.
• boot_tidy, boot_glance, and boot_augment raise RuntimeError when every bootstrap replication fails, instead of returning an empty DataFrame.
• stats_residual_plot validates that the target column y is numeric before passing it to plotting / probplot.
• stats_vif now emits a clear warning that the intercept is omitted, and handles no-intercept formulas consistently without adding a constant.

Selected P1 fixes in the same release:

• anova_type is validated to be 1, 2, or 3 in stats_anova_tidy.
• stats_kruskal_tidy validates that group_col and value_col exist.
• stats_correlation_tidy validates that requested columns exist and are numeric.
• stats_formula now quotes non-syntactic column names with Q('...').
• bayes_boot validates target_column / n_samples and warns when NaN values are dropped.
• stats_chisquare_plot drops NaN categories before building the contingency table.
• Repository URLs in setup.cfg updated from jcvall/broom-sm to ezraair555/broom-sm.
• Removed the unused src/extra_sm package.

Documentation

Full documentation (API, how-to guides, tutorials, and plot gallery) lives in docs/:

• Tutorials — End-to-end walkthroughs
• How-to Guides — Task-oriented recipes
• AI Assistant Workflow — Deterministic patterns for coding agents
• API Reference — Complete function documentation
• Quick Start — Get started in 5 minutes

The rendered site is at https://ezraair555.github.io/broom-sm/.

Contributing

We welcome contributions! Please review our contributing guidelines and Python Software Foundation code of conduct.

For questions and discussions, please post on GitHub Discussions. If you think you've encountered a bug, please submit an issue.

License

MIT License — see LICENSE.txt for details.

Acknowledgments

broom-sm draws inspiration from:

• broom (R) — Tidy model outputs
• infer (R) — Tidy statistical inference
• pandas_flavor — DataFrame method registration
• statsmodels — Statistical modeling in Python