NullSweep

Diagnose missing data before you impute it. NullSweep reads the shape and the mechanism of the holes in your DataFrame, then gives you a single interface to fill them with a method the evidence actually justifies.

Works with pandas and Polars — both installed by default, no extras to remember.

Why

df.fillna(df.mean()) is a guess. Two questions, answerable straight from the data, turn it into a decision:

Pattern is about tractability; mechanism is about correctness. You need both.

Installation

pip install nullsweep

A single batteries-included install: pandas and Polars support, the KNN/MICE/regression imputers, the MAR/MCAR statistical tests, and the visualizations all come with it.

Quickstart

import nullsweep as ns

shape, details = ns.detect_global_pattern(df)              # "univariate" | "monotone" | "non-monotone"
mech, details = ns.detect_feature_pattern(df, "income")    # "MAR_evidence" | "MCAR_consistent" | "undetermined"

ns.plot_missing_values(df, "matrix")                       # look before you test

df = ns.impute_nulls(df, column="income", strategy="mice") # strategy chosen from the evidence

impute_nulls returns a new frame by default — your original is never mutated.

API

Detecting patterns (where)

pattern, details = ns.detect_global_pattern(df)

For monotone, details["matrix"] shows the nesting: each cell answers "whenever the row variable is missing, is the column variable also missing?"

       A      B      C
A  False   True   True      # A missing ⇒ B and C missing
B  False  False   True      # A ⊆ B ⊆ C — that nesting is monotonicity
C  False  False  False

For univariate, details["column"] names the offending column.

Detecting mechanisms (why)

label, details = ns.detect_feature_pattern(df, "target")

The column's missingness indicator is screened against every other column with logistic regression. If nothing predicts it, Little's (1988) MCAR test runs on the numeric columns to separate "looks random" from "we can't tell."

MCAR_consistent means failed to reject, not proven. details carries mar_predictors (a per-predictor evidence map) and, when it ran, mcar_test (statistic, degrees of freedom, p-value, and a plain-language message).

Useful options:

# Catch confounded MAR with a single joint model instead of marginal screens
ns.detect_feature_pattern(df, "target", multivariate=True)

# The pseudo R-squared gate (default 0.2) is deliberately strong and can hide
# weak-but-real signals. Lower it to increase sensitivity:
ns.detect_feature_pattern(df, "target", pseudo_r_squared_threshold=0.05)

# Numeric predictors only / skip the MCAR step
ns.detect_feature_pattern(df, "target", include_categorical=False, run_mcar_test=False)

Little's test is also available directly: from nullsweep.patterns.mcar.little import little_mcar_test.

Imputing (what)

df = ns.impute_nulls(df, column="age", strategy="mean")
df = ns.impute_nulls(df, column=["age", "income"], strategy="mice")
df = ns.impute_nulls(df)                                    # strategy="auto" on every column with gaps
df = ns.impute_nulls(df, strategy="listwise", threshold=2)  # drop rows with >= 2 missing values
df = ns.impute_nulls(df, column="city", strategy="constant", fill_value="Unknown")

Strategy families

Notes worth knowing:

• knn / mice / regression operate on numeric columns and reject non-numeric targets. When a target column is given, complete numeric predictors are still used as context but only the target is written back. Encode categoricals first, or use a frequency strategy.
• flag adds a <column>_missing indicator per column. With column=None it flags only columns that have gaps; include_all_columns=True flags every column.
• delete_column / listwise take a threshold: floats are missing-value proportions, integers are counts, and equality is deleted.
• Directional fills leave edge gaps — backfill cannot fill a trailing NaN. Check the result rather than assuming it is complete.
• auto uses interpolation for continuous columns only when the index is datetime/timedelta (override with strategy_params={"allow_ordered_interpolation": True}), and treats integer/boolean-coded numeric columns as categorical (disable with {"detect_numeric_categorical": False}).

Parameters

Train/test workflows

impute_nulls recomputes its fill values from whatever frame it is handed — applying it to a test set leaks. Use the transformer instead:

from nullsweep import NullSweepImputer

imputer = NullSweepImputer(column="income", strategy="mean").fit(train)
train_imputed = imputer.transform(train)
test_imputed = imputer.transform(test)   # filled with TRAIN's statistics

It learns means, medians, modes, the KNN/MICE/regression models, and the per-column choice made by auto at fit time, then applies them unchanged. Directional strategies carry no cross-frame state; listwise learns a per-row mask, so use fit_transform for it.

Visualizing

fig = ns.plot_missing_values(df, "heatmap", figsize=(8, 4), cmap="magma")

Each returns a Matplotlib figure and accepts the underlying plot function's keyword arguments. None of them prove anything — they point your eye at the structure so you know which test to run first.

pandas and Polars

Every public function accepts either a pandas or a Polars DataFrame, and the imputers return the same type they were given. Polars frames are immutable, so in_place=True warns and returns a new frame.

Contributing

Contributions are welcome. Please submit pull requests, open issues, or suggest improvements.

License

MIT