AutomatedCleaning

AutomatedCleaning is a Python library for automated data cleaning, now powered by a Rust compute backend. It preprocesses and analyzes datasets — handling missing values, outliers, spelling corrections, text cleaning, PII masking and more — while the CPU-heavy work runs in compiled Rust for speed and true (GIL-free) parallelism.

Why Rust + Python?

The library is a hybrid: a friendly Python API on top, a fast Rust engine underneath.

• 🐍 Python frontend — the API you call, plus everything that leans on the Python ecosystem (Polars I/O, scikit-learn imputation, Plotly dashboards, Presidio PII, Claude AI).
• 🦀 Rust backend (automatedcleaning._rustcore) — the number-crunching and per-row string work: text preprocessing, symbol stripping, IQR outliers, skewness, correlation, multicollinearity and JSON detection.

You install one wheel and write normal Python — the Rust is invisible.

┌─────────────────────────── your code ───────────────────────────┐
│  import automatedcleaning as ac                                  │
│  df = ac.load_data("data.csv"); ac.clean_data(df)                │
└──────────────────────────────┬──────────────────────────────────┘
                               │ calls
┌──────────────────────────────▼──────────────────────────────────┐
│  Python frontend  (automatedcleaning/cleaning.py)                │
│    Polars • scikit-learn • Plotly • Presidio • LangChain         │
└──────────────────────────────┬──────────────────────────────────┘
                               │ delegates hot loops
┌──────────────────────────────▼──────────────────────────────────┐
│  🦀 Rust backend  (_rustcore, built with PyO3 + rayon)           │
│    text cleaning • skewness • IQR • correlation • JSON detect     │
└──────────────────────────────────────────────────────────────────┘

Benchmark — text preprocessing over 50,000 rows: ~3.8× faster than the pure-Python path, with the GIL released so it scales across cores.

Features

• Supports both large (100+ GB) and small datasets
• Detects and handles missing values and duplicate records
• Identifies and corrects spelling errors in categorical values
• Detects and removes outliers (IQR)
• Detects and fixes data imbalance
• Identifies and corrects skewness in numerical data
• Checks for correlation and detects multicollinearity
• Analyzes cardinality in categorical columns
• Identifies and cleans text columns
• Detects JSON-type columns
• Detects and masks PII columns
• Performs univariate, bivariate, and multivariate analysis (interactive dashboard)
• Evaluates models: detailed metrics, SHAP explanations, error analysis, fairness checks and auto-generated model cards

Installation

From PyPI (recommended)

pip install AutomatedCleaning

Prebuilt abi3 wheels are published for Linux, macOS and Windows and work on CPython 3.11+ — no Rust toolchain needed to install. The import name is also automatedcleaning (import automatedcleaning as ac).

From source

Building from source requires the Rust toolchain (because the backend is compiled):

# 1. Install Rust (https://rustup.rs) and maturin
pip install maturin

# 2. Clone and build
git clone https://github.com/DataSpoof/dataspoof-data-cleaning.git
cd dataspoof-data-cleaning

# Build & install into the current environment
pip install .

# --- or, for development (needs an active virtualenv) ---
python -m venv .venv && source .venv/bin/activate   # Windows: .venv\Scripts\activate
maturin develop --release

Quick start

import automatedcleaning as ac

# Load any CSV / TSV / JSON / Parquet file (via Polars)
df = ac.load_data("dataset.csv")

# Run the full interactive cleaning pipeline
df_cleaned = ac.clean_data(df, background_image_path="assets/gradient.png")

clean_data() walks the dataset through the whole pipeline and asks a few interactive questions (column spelling fixes, categorical corrections, target column for imbalance). It writes cleaned_data.csv and an EDA dashboard to output/eda/dashboard.html.

Pipelines: fit once, apply anywhere (recommended for production)

clean_data() is interactive and re-learns everything from whatever data it is given. For scheduled jobs, train/test splits and production scoring, use a pipeline instead. It asks no questions, learns its cleaning rules on training data, and replays exactly those rules on new data, so a test set or a production batch never influences how it is cleaned.

import automatedcleaning as ac

pipe = ac.default_cleaning_pipeline()
train_clean = pipe.fit_transform(train_df)   # learn the rules on training data
test_clean = pipe.transform(test_df)         # same rules, nothing re-learned

print(pipe.report_)                          # what each step did (rows, columns, nulls, time)
pipe.save("cleaning_pipeline.json")          # everything it learned, as plain JSON (no pickle)

# later, in a scheduled job or an API
pipe = ac.Pipeline.load("cleaning_pipeline.json")
batch_clean = pipe.transform(batch_df)

The default pipeline: fix missing-value placeholders → normalise text → repair numbers and dates stored as text → drop duplicate rows (training only) → drop useless columns (mostly missing, constant, ID-like, duplicated) → impute, adding a <col>_was_missing flag for columns ≥5% missing → group rare categories → cap extreme outliers (3×IQR) → drop near-duplicate numeric columns. Change any step by name:

pipe = ac.default_cleaning_pipeline(clip_outliers=False, impute={"numeric": "mean"})

Build your own pipeline

Mix the stateful steps (which learn in fit) with any of the functions above:

pipe = ac.Pipeline([
    "replace_disguised_missing",                       # any library function, by name
    ac.ConvertNumericText(),
    ac.Impute(numeric="median", add_indicators=True),
    ac.EncodeCategorical(method="onehot"),
    ac.Scale(method="standard"),
], name="churn-features")

Stateful steps: drop_columns, drop_collinear, convert_numeric_text, parse_dates, impute, clip_outliers, remove_outliers, log_transform_skewed, scale, group_rare_categories, encode_categorical, drop_duplicates. Run ac.available_steps() to list every step name with a description.

Config files

Keep pipelines in version control as YAML or JSON:

# cleaning.yaml
name: churn-cleaning
on_schema_mismatch: raise        # warn (default) | raise | ignore
steps:
  - step: replace_disguised_missing
    params: {numeric_sentinels: [-999]}
  - step: drop_columns
    params: {max_missing: 0.5}
  - step: impute
    params: {numeric: median, add_indicators: true}
  - step: remove_outliers          # drops rows during fit only

pipe = ac.Pipeline.from_config("cleaning.yaml")

Built for production

• Schema checks: transform compares incoming columns and types with the training data and warns, or raises with on_schema_mismatch="raise".
• Training-only steps: row-removing steps (drop_duplicates, remove_outliers) run during fit and are skipped by transform, so scoring never silently drops records.
• Audit trail: every run produces a RunReport with a run ID, timings, per-step changes, data fingerprints, the config hash and library versions. Pass audit_log="runs.jsonl" to append each run to a file.
• Safe persistence: saved pipelines are plain JSON you can review, diff and load safely. Nothing is pickled.
• Extensible: subclass ac.Step and decorate it with @ac.register_step to add your own steps; they work in configs and in saved pipelines like the built-in ones.

Profiling & data quality (automatedcleaning.quality)

Automated EDA

p = ac.profile(df)
print(p)                 # overview, every column's type and meaning, plain-English insights
p["Fare"].numeric        # min, max, mean, median, std, p01…p99, skew, outliers, …
p.to_frame()             # one row per column, ready to filter or export
p.to_json("profile.json")

Each column gets a semantic type describing what it holds, not just how it's stored: identifier, binary, categorical, numeric, discrete, text, email, phone, url, datetime, numeric_text (numbers stored as text), datetime_text (dates stored as text), constant or empty. Insights are generated for findings such as:

• columns that are mostly empty
• identifier and constant columns
• strong correlations
• heavy skew
• columns dominated by a single value

Quality score

report = ac.check_quality(df)
print(report)                  # score, grade, dimension bars, weakest columns, top issues
report.score, report.grade     # e.g. 97.2, "A"
report.dimensions              # completeness / uniqueness / validity / consistency, each 0-100
report.weakest_columns()       # e.g. [("Cabin", 50.9), ("Age", 89.3)]
report.to_frame()              # every issue with severity, column and cells affected

The score combines four standard data-quality dimensions:

• completeness: real values, counting "NA"/"?" placeholders and -999 codes as missing
• uniqueness: rows that aren't duplicates
• validity: values pass range, email, phone and calendar-date rules
• consistency: no stray whitespace, case variants or mixed date formats

The default weights are 0.35 / 0.15 / 0.30 / 0.20 and can be changed with weights=. Grades run A ≥ 90, B ≥ 80, C ≥ 70, D ≥ 60, otherwise F. The overall score averages over every cell, so one broken column barely moves it; weakest_columns() is there to catch that.

Data contracts

Declare what valid data looks like and check every batch against it:

# customers.contract.yaml
name: customers
extra_columns: warn            # allow | warn | error
no_duplicate_rows: true
unique: [customer_id]
columns:
  customer_id: {type: integer, required: true, unique: true}
  email:       {type: string, format: email, max_null_fraction: 0.01}
  age:         {type: number, min: 0, max: 120, warn_only: [min, max]}
  plan:        {type: string, allowed: [basic, pro, enterprise]}

contract = ac.Contract.from_file("customers.contract.yaml")
result = contract.validate(batch_df)
print(result)                  # PASSED/FAILED with every broken rule and example values
result.raise_if_failed()       # raise ContractViolation in a job or CI

# No contract yet? Infer one from data you trust, review it, then commit it:
ac.Contract.infer(train_df).to_yaml("customers.contract.yaml")

Inferred contracts make structural rules errors (columns, types, required, unique) and statistical rules warnings (ranges, allowed values, lengths, null fractions), because new data legitimately drifts.

Schema checks

diff = ac.compare_schemas(train_df, new_df)
diff.removed, diff.added, diff.changed      # e.g. {"age": ["Float64", "String"]}
diff.compatible                             # False if a column vanished or changed type

Enforce quality inside pipelines

pipe = ac.Pipeline([
    ac.ValidateContract(),                       # infers a contract from training data on fit
    *ac.default_cleaning_pipeline().steps,
    ac.QualityGate(min_score=85, min_dimensions={"completeness": 90}),
])
pipe.fit(train_df)
pipe.save("pipeline.json")                       # the inferred contract is saved inside it
ac.Pipeline.load("pipeline.json").transform(new_batch)   # raises if the batch breaks it

Use on_fail="warn" on either step to log a warning instead of stopping. The EDA dashboard's Overview tab is built on this module, so it shows the same score, weakest columns and issues.

Feature engineering (automatedcleaning.features)

Feature steps work like the cleaning steps: they learn in fit, replay in transform, and save as JSON. Give the pipeline a target so steps that need the label can use it:

pipe = ac.Pipeline(
    [*ac.default_cleaning_pipeline().steps, *ac.default_feature_pipeline("churned").steps],
    target="churned",
)
train_features = pipe.fit_transform(train_df)       # model-ready numeric features + the label
new_features = pipe.transform(new_df)               # the label isn't needed to predict

The default pipeline runs these steps in order:

1. date/time features
2. drop leaky columns
3. text embeddings
4. target encoding for columns with 16+ categories
5. one-hot encoding for the rest
6. feature selection

time_col="..." adds lag features and turns off target encoding.

Target-aware pipelines: with target=, ordinary steps never see the label. That means imputation can't invent labels, outlier clipping can't change them, and encoders can't encode them. Row-removing steps drop the same rows from the label, and the label can be absent at prediction time.

Leakage detection: it scores every column on its own against the target. A column that predicts almost perfectly is usually recorded after the outcome (refund_issued when predicting churn), and such columns are flagged:

report = ac.detect_leakage(df, "churned")
print(report)            # [high] refund_issued: on its own it predicts 'churned' with cv roc_auc = 1.000 ...

AutoML (automatedcleaning.automl)

model = ac.AutoML("churned", time_budget=120).fit(train_df)
print(model.summary())               # task, metric, CV scheme, best model, best score per model
model.leaderboard_                   # every trial with its score, std, time and parameters
model.evaluate(test_df)              # accuracy, balanced accuracy, F1, ROC-AUC, log-loss (RMSE, MAE, R² for regression)
model.predict(new_df)                # labels in their original form
model.predict_proba(new_df)          # one proba_<class> column per class
model.feature_importance()           # permutation importance
model.save("churn_model/")           # directory: manifest.json + model.joblib + preprocess.json
model = ac.AutoML.load("churn_model/")

AutoML(target).fit(df) works through these steps:

1. Detect the task. Two values means binary; text, or up to 10 whole-number values, means multiclass; anything else is regression. Rows sorted by a date column are treated as a time series.
2. Pick a metric: ROC-AUC, macro-F1 or RMSE.
3. Drop leaky columns.
4. Clean and engineer features with the target-aware pipelines above.
5. Cross-validate a baseline and every model. The defaults are logistic/ridge regression, histogram gradient boosting, random forest, extra trees, k-NN, and LightGBM and XGBoost if installed. Folds are stratified, or time-ordered for time series.
6. Spend the rest of time_budget on random hyperparameter search around the best model families.
7. Try a soft-voting ensemble of the top three families.
8. Refit the winner on all rows.

On Titanic, with the defaults and a 40-second budget, AutoML reaches 0.918 ROC-AUC and 83% accuracy on held-out rows.

Saved models: a trained model can't be stored as JSON, so model.joblib uses joblib, the scikit-learn standard. Its SHA-256 is recorded in manifest.json and verified before loading, which catches corrupted or swapped files. Unpickling still runs code, so only load models from people you trust.

Clustering

clusters = ac.AutoCluster(k_range=(2, 8)).fit(df)     # k-means and Gaussian mixtures, chosen by silhouette
clusters.labels_                                      # one label per input row
clusters.cluster_profiles(df)                         # size, share and averages per cluster
ac.automl(df)                                         # no target -> AutoCluster; with target= -> AutoML

Optional extras: pip install "AutomatedCleaning[automl]" (LightGBM, XGBoost) and pip install "AutomatedCleaning[embeddings]" (sentence-transformers).

Evaluation & explainability (automatedcleaning.evaluation)

Every tool takes a fitted AutoML model (or any scikit-learn estimator plus target=...) and labelled data. Each is also a method on AutoML.

model = ac.AutoML("Survived").fit(train)

report = model.report(test)              # == ac.evaluate_model(model, test)
print(report)                            # accuracy, F1, ROC-AUC, PR-AUC, MCC, log loss, Brier, ECE ...
report.confusion_matrix                  # also: per_class, calibration, roc_curve, pr_curve
report.confidence_intervals              # 95% bootstrap intervals for the headline metrics
report.best_threshold("f1")              # threshold table for binary models -> best cut-off
ac.evaluate_model(model, test, threshold=0.3)   # metrics at a different decision threshold

Regression reports give RMSE, MAE, median/max error, R², bias, MAPE/sMAPE, the residual distribution and the error for each range of the actual value.

SHAP explanations. Shapley values are computed by the library itself, with no shap dependency. They work for any model, including AutoML's ensembles and preprocessing. With up to 8 features every coalition is evaluated (exact SHAP); otherwise antithetic permutation sampling is used. For every row base_value + sum(values) equals the model's output exactly.

exp = model.explain(test, rows=100)      # == ac.explain(model, test)
print(exp)                               # global importance (mean |SHAP|) + direction of effect
exp.global_importance()                  # feature, mean_abs_shap, share, direction
exp.explain_row(0)                       # "P(Survived = 1) = 0.27 (average 0.42): Pclass = 3 (-0.174), ..."
exp.for_row(0)                           # contributions for one row, largest first
exp.to_shap()                            # a shap.Explanation for shap's plots: pip install "AutomatedCleaning[explain]"
ac.partial_dependence(model, test, "Fare")   # average prediction as one feature varies

Error analysis finds the worst predictions and the segments of the data where the model fails. A segment is a category, a quantile bin, missing values, or a pair of these. Each segment is tested against the rest of the data, with a Benjamini–Hochberg correction so random noise isn't flagged.

errors = model.error_analysis(test)
errors.problem_slices        # e.g. "region = east": error rate 31% (2.4x), q = 0.001
errors.worst                 # most confident mistakes / largest errors, with their inputs
errors.confusions            # most common actual -> predicted mistakes

Fairness & bias checks. These cover each sensitive attribute, intersections such as sex × age band, and proxy features that encode a sensitive attribute.

fair = model.fairness(test, ["Sex", "Age"])      # numeric attributes are cut into quantile bands
print(fair)            # selection rate, TPR, FPR, precision and accuracy per group, plus flags
fair.passed            # False if any limit is exceeded
fair["Sex"].groups     # the per-group table
fair.proxies           # Cramér's V between every feature and each sensitive attribute

The checks cover demographic parity, disparate impact (four-fifths rule), equal opportunity, equalized odds, predictive parity and accuracy gaps. For regression, they compare error and bias across groups. The default limits are in ac.evaluation.DEFAULT_LIMITS; override them with limits={...}.

Model cards. One call generates a full card in the standard layout (Mitchell et al.):

• model details and training data;
• the model-selection leaderboard;
• held-out metrics with confidence intervals and calibration;
• SHAP importance, error analysis and fairness;
• the quality score of the evaluation data;
• auto-written limitations and ethical considerations;
• versions.

card = model.model_card(test, sensitive=["Sex"], name="Titanic survival",
                        intended_use="Teaching example; not for real decisions.", owners="Data team")
card.save("model_card.html")     # self-contained page (light/dark); also .md and .json
card.limitations                 # e.g. "Performs significantly worse on some segments: ..."

Using individual steps

Every stage is also a standalone function you can call directly — useful for scripting or building your own non-interactive pipeline. These delegate to the Rust backend:

import polars as pl
import automatedcleaning as ac

df = ac.load_data("dataset.csv")

df = ac.detect_column_types_and_process_text(df)  # classify + clean text columns (Rust)
df = ac.handle_negative_values(df)                # negatives -> absolute values (Rust)
df = ac.replace_symbols(df)                       # strip $ ₹ , - •  (Rust)
df = ac.handle_missing_values(df)                 # KNN impute + mode fill (Python/sklearn)
df = ac.handle_duplicates(df)                     # drop duplicate rows (Polars)
df = ac.remove_outliers(df)                       # IQR outlier removal (Rust)
df = ac.fix_skewness(df)                          # log-transform skewed columns (Rust)
df = ac.check_multicollinearity(df, threshold=0.7)# drop correlated features (Rust)
df, cardinality = ac.check_cardinality(df)        # report + drop constant columns
df = ac.fix_json_columns(df)                      # expand JSON columns (Rust detection)
df = ac.detect_and_mask_pii_polars(df)            # detect + mask PII (Presidio)
ac.generate_dashboard(df)                         # Plotly EDA dashboard
ac.save_cleaned_data(df, "cleaned_data.csv")

Text preprocessing on its own

ac.preprocess_text("I can't wait!! Visit https://x.co @bob 😀 it's GR8")
# -> 'cannot wait visit'

Additional cleaning steps (automatedcleaning.steps)

Beyond the core pipeline, ~45 focused, composable steps are available directly on ac.*. String-heavy ones (whitespace, Unicode NFKC, mojibake, disguised-missing, booleans) run in the Rust backend.

df = ac.standardize_column_names(df)          # trim/lowercase/snake_case/de-dupe
df = ac.drop_empty_rows_and_columns(df)
df = ac.drop_high_missing_columns(df, 0.5)
df = ac.remove_duplicate_columns(df)
df = ac.drop_id_like_columns(df)
df = ac.replace_disguised_missing(df, numeric_sentinels=[999, -1])  # "NA","?"… -> null (Rust)
df = ac.add_missing_indicators(df)
df = ac.impute_missing(df, numeric_strategy="mice")   # mean|median|knn|mice|interpolate|ffill|bfill
df = ac.normalize_whitespace(df)              # Rust
df = ac.normalize_unicode(df)                 # NFKC + control-char strip (Rust)
df = ac.fix_mojibake(df)                       # "é" -> "é" (Rust)
df = ac.standardize_casing(df, {"name": "title"})
df = ac.parse_dates(df, ["signup"])           # -> native datetime
df = ac.extract_date_parts(df, ["signup"])    # year/month/day/weekday/quarter
df = ac.normalize_booleans(df, ["active"])    # yes/no/Y/N/1/0 -> bool (Rust)
df = ac.downcast_numeric(df)                  # int64->int32, etc.
df = ac.group_rare_categories(df, 0.01)
df = ac.fuzzy_cluster_categories(df)          # merge "New York"/"newyork" (rapidfuzz)
df = ac.encode_categorical(df, "onehot")      # onehot|label|frequency|ordinal|target
df = ac.remove_outliers_zscore(df)            # + _mad, _isolation_forest, _by_group
df = ac.winsorize(df)                          # cap instead of drop
df = ac.fuzzy_deduplicate(df, ["name","email"])
df = ac.scale_numeric(df, "standard")         # minmax|standard|robust
df = ac.bin_numeric(df, ["age"], bins=5)
df = ac.apply_range_rules(df, {"age": (0, 120)}, action="clip")
df = ac.validate_formats(df, {"email": "email", "phone": "phone"})
df = ac.normalize_phone_numbers(df, ["phone"], region="US")   # -> E.164
df = ac.normalize_emails(df, ["email"]); df = ac.canonicalize_urls(df, ["site"])

Steps that need optional libraries (detect_language, redact_profanity, correct_spelling_text) are installed via the extra:

pip install "AutomatedCleaning[text]"

Supported file formats

load_data() reads .csv, .tsv, .json, and .parquet (Excel must be converted first).

Requirements

• Python ≥ 3.11
• Key runtime deps (installed automatically): polars, pandas, numpy, pyarrow, nltk, scikit-learn, matplotlib, seaborn, missingno, plotly, pyfiglet, langchain-anthropic, presidio-analyzer, presidio-anonymizer
• The automatic categorical spell-correction step optionally uses the Anthropic Claude API (you'll be prompted for a key); you can skip it or correct values manually.

Building & contributing

The project uses maturin as its build backend (pyproject.toml). See BUILD.md for the full architecture and build/publish guide.

Cargo.toml            # Rust crate config
src/
  lib.rs              # PyO3 module — functions exposed to Python
  text.rs             # text preprocessing (contractions, regex, stopwords)
  stats.rs            # skewness, IQR, correlation, multicollinearity
  clean.rs            # whitespace, Unicode, mojibake, booleans, placeholders
automatedcleaning/
  core/               # shared Step/Pipeline core (no imports from the rest of the package)
  cleaners.py         # stateful cleaning steps + default_cleaning_pipeline()
  steps.py            # 43 standalone cleaning functions
  cleaning.py         # interactive clean_data() pipeline and EDA dashboard
tests/                # pytest suite

Run the tests (needs the Rust toolchain to build the extension):

python -m venv .venv && source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install maturin
maturin develop --release --extras test
pytest

Cross-platform wheels are built and published to PyPI automatically via GitHub Actions (.github/workflows/CI.yml) on every tagged release (vX.Y.Z).

License

MIT © Abhishek Kumar Singh / DataSpoof