smartclean-df
Turns a messy table into a tidy one in one call: it finds and fixes missing values, duplicate rows, outliers, numbers/dates/booleans stored as text, and dirty column names, and tells you exactly what it changed.
Install
pip install smartclean-df
Parquet input needs the optional extra: pip install "smartclean-df[parquet]".
Quickstart
import pandas as pd
from smartclean_df import clean
df = pd.DataFrame({"Name ": ["Ann", " Bob", "Ann", None], "Age": ["34", "NA", "34", "41"], "Joined": ["2024-01-05", "2024-02-10", "2024-01-05", "-"]})
result = clean(df)
print(result.summary())
print(result.df)
result.df is the cleaned copy (the input is never modified), result.actions
lists every change in the order it was made, and result.summary() prints it
as text. df, actions = clean(df) also works.
What it does
Every step runs in this order and every change is logged as an Action:
- Column names are stripped, inner whitespace collapsed to
_, and lowercased ("Order ID "becomesorder_id); clashes get a_2suffix. - String cells are stripped of surrounding whitespace, and missing tokens
(
"",NA,N/A,null,None,-,?,nan, any case) becomeNaN. - Numbers stored as text are parsed when at least 90% of the non-missing
values parse:
"1,234","$12.50","(300)","45%"(the%is stripped, the value is kept as written:45.0)," 7 ". Whole-number columns becomeint64. Codes with leading zeros ("00123") are left alone. Columns that mix numbers and text stayobject. - Dates stored as text are parsed when at least 90% parse. Day-first versus month-first is inferred from the data, and no pandas warnings leak.
- Booleans stored as text (
yes/no,true/false,y/n,t/f,1/0strings) becomebool. Numeric columns are never touched by this step. - Exact duplicate rows are dropped, then rows and columns that are entirely empty.
- Missing values are imputed: median for numeric columns, mode for
categorical/boolean columns, forward-fill for datetimes. Integer columns
stay integer when the median is a whole number.
missing="drop"drops any row with a missing value instead;missing="none"leaves them. - Outliers in numeric columns are found with the IQR rule
(
Q1 - k*IQR,Q3 + k*IQR,k = iqr_factor, default 3.0)."clip"winsorizes them to the bounds,"flag"only reports them,"drop"removes the rows,"none"skips the step. Columns with zero IQR (constants, 0/1 indicators) are never clipped.
If the input has a default RangeIndex and rows were dropped, the result is
re-indexed from 0; any other index is kept so rows stay traceable.
API
clean(df_or_path, *, missing="auto", outliers="clip", iqr_factor=3.0, duplicates=True,
normalize_columns=True, parse_numbers=True, parse_dates=True, parse_booleans=True,
dry_run=False) -> CleanResult
df_or_path is a pandas.DataFrame or a path to a .csv, .tsv or
.parquet file. missing is "auto", "drop" or "none"; outliers is
"clip", "flag", "drop" or "none". With dry_run=True the actions are
computed and reported but result.df is an unchanged copy of the input.
CleanResult
.df- the cleanedDataFrame(unchanged copy whendry_run=True).actions-list[Action], every change made, in order.input_shape,.output_shape-(rows, columns)before and after.summary()- human-readable text.to_dict()- JSON-safe dict (shapes,actions, outputdtypes)
Action(column, kind, detail, rows_affected) - column is None for
table-wide actions (dropping duplicate rows, dropping rows with missing
values). kind is one of rename_column, strip_whitespace,
missing_tokens, parse_numeric, parse_datetime, parse_boolean,
drop_duplicates, drop_empty_rows, drop_empty_column, impute,
drop_missing_rows, clip_outliers, flag_outliers, drop_outliers.
Cleaner(**same options as clean, except dry_run)
Cleaner.fit(df_or_path) -> Cleaner
Cleaner.transform(df_or_path, *, dry_run=False) -> CleanResult
Cleaner.fit_transform(df_or_path, *, dry_run=False) -> CleanResult
fit learns which columns to parse (and how), the imputation value of every
column, the outlier bounds, and which all-empty columns to drop. transform
applies exactly that to new data, so production batches get the same
treatment as the data you fitted on: learned medians and modes fill new gaps,
learned bounds clip new outliers, and a column that was parsed as a date is
parsed as a date again even if the batch is too small to pass the 90% rule.
Columns not seen during fit only get the stateless whitespace / missing
token cleanup.
The library never prints; it logs through logging.getLogger("smartclean_df").
CLI
smartclean-df data.csv # print the summary of what would change
smartclean-df data.csv --json # print to_dict() as JSON
smartclean-df data.csv --output clean.csv # also write the cleaned table (.csv, .tsv, .parquet)
smartclean-df data.csv --missing drop --outliers flag --iqr-factor 1.5
smartclean-df data.csv --dry-run # report only, never write changed data
smartclean-df --help
Flags mirror the Python options: --missing, --outliers, --iqr-factor,
--keep-duplicates, --keep-column-names, --no-parse-numbers,
--no-parse-dates, --no-parse-booleans, --dry-run.
License
MIT
Metadata
Release files for smartclean-df 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| smartclean_df-0.1.0.tar.gz | 28.3 kB | Without provenance | Details for smartclean_df-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| smartclean_df-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for smartclean_df-0.1.0-py3-none-any.whl |
Total release size: 51.2 kB
Release files / smartclean_df-0.1.0.tar.gz
| Download URL | smartclean_df-0.1.0.tar.gz |
|---|---|
| Size | 28.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
3293e25314fb44a72dc1f83eb72dec9ea6746dcbabd682284dffdd580f84e974
|
|
BLAKE2b-256 checksum How to use checksums |
4538ec12d8557fbe58f9f73e5c40e401add0694df802277877afb21a4a798017
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.10.11
|
Release files / smartclean_df-0.1.0-py3-none-any.whl
| Download URL | smartclean_df-0.1.0-py3-none-any.whl |
|---|---|
| Size | 23.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
66b4b79e44c4a9679b59a4199ff47636a58221eee0324c737342d192f6583416
|
|
BLAKE2b-256 checksum How to use checksums |
4fba4d45ff576fa37ccb7c68bb5e7d0af4187991cb753270c00104452f72c657
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.10.11
|
Security for smartclean-df 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.