dstoolkit

Data Science Toolkit: a small, typed Python package for the first stages of a data project. It profiles a pandas DataFrame and suggests how to clean it, cleans it (duplicates, missing values, outliers, encoding, scaling), validates it against rules you write, explores it (summary statistics, distribution plots, line fits), and prepares it for models with scikit-learn transformers that learn on training data only. Every cleaning function returns a new DataFrame, so your raw data is never changed, and every function copes with missing values on its own. There's also a drag-and-drop browser version that needs no install.

Install

pip install pandas-dstoolkit

Requires Python 3.10+, with pandas, NumPy, matplotlib and SciPy. The scikit-learn transformers and k-nearest-neighbours filling also need scikit-learn, which is optional:

pip install "pandas-dstoolkit[sklearn]"

It installs as pandas-dstoolkit (PyPI treats dstoolkit as too close to an unrelated ds-toolkit) and imports as dstoolkit. Releases are published to PyPI by GitHub Actions through trusted publishing, so no upload token exists to leak. To install the latest code from GitHub instead: pip install git+https://github.com/Dito-84/dstoolkit.

What it's for, and what it isn't

dstoolkit is for getting to know a dataset and cleaning it once: profiling a new file, fixing types, checking it against rules, and preparing a clean copy for analysis or a dashboard. It's built to be safe by default: when a guess could damage data (reading 1/0 as True/False, swapping day and month, filling an ID column with its mean), it leaves the data alone or asks you to be explicit.

The cleaning functions (fill_missing, normalize, encode_categorical, handle_outliers) learn their numbers (means, ranges, categories, outlier bounds) from the DataFrame you pass in and apply them in the same call. That's right for exploring data, but not before a model: run them on a full dataset before splitting it, and information from the test rows leaks into training. For machine learning, use the transformers instead. They learn on the training rows only and work inside a scikit-learn Pipeline, so cross-validation refits them on each fold.

Design decisions

A few choices shape everything else:

• Never change the input, never guess. Functions return a new DataFrame. When a guess could damage data (1/0 as True/False, swapping day and month, filling an ID with its mean), dstoolkit leaves the data alone or asks you to be explicit.
• Learn from training rows only. The cleaning functions are for exploring. For models, the transformers fit on training data inside a scikit-learn Pipeline, so no test data leaks into training.
• Report leaks, don't hide them. profile(target=...) flags columns that give the answer away, and the generated script leaves them out with a comment. On the Titanic data, the lifeboat column alone lifts accuracy from 0.785 to 0.976.
• Validate honestly. Repeated patients get grouped folds, time-ordered data gets a past-to-future split, and every score is shown next to a no-skill baseline.
• Judge drift by size, not significance. compare() rates change by the population stability index, so thousands of rows don't make a trivial shift look alarming.

The reasons, alternatives and costs for these and more are in docs/design-decisions.md.

Profile a dataset in one call

import pandas as pd
import dstoolkit as dst

raw = pd.read_csv("data/titanic3.csv", na_values="?")
report = dst.profile(raw, title="Titanic passengers")
report                       # shows inline in Jupyter
report.save("report.html")   # one self-contained file to share
print(report.code())         # the suggested cleaning steps as runnable code

The report covers the whole dataset (rows, duplicates, missing cells), every column (type, missing values, statistics, outliers and a chart), the correlations between numeric columns, and related columns of any type (numbers, categories, or one of each, from associations(), with p-values corrected for testing many pairs). It also reads the data and suggests cleaning steps, with ready-to-run dstoolkit code: which columns are too sparse to keep, which to fill and how, long tails that suit a log scale, numbers stored as text, categories to encode, labels written more than one way ("Male", "male "), identifier columns to leave out of models, column names with hidden spaces or line breaks, and notes rows at the end of an export ("Source: ..." lines that aren't data). Those notes rows are left out of the column analysis, since they would make number columns look like text, and the code drops them first.

Download the full example report (open it in a browser), or see the Profile_Titanic notebook, which profiles the raw data, applies the suggested code and profiles the result.

Prepare for a model

Name the column a model will predict, and the report checks the things to settle before training:

report = dst.profile(raw, target="survived")
report.target.leaks        # ['boat']: whether a lifeboat is listed predicts survival almost perfectly
print(report.pipeline())   # a leak-free scikit-learn starter, ready to run

• Class balance, with a warning when the smallest class is under 10% of rows. The task is worked out from the target (text, True/False, or a few whole numbers mean classification); task= overrides it.
• How strongly each feature is related to the target, from associations().
• Possible leaks: features almost perfectly related to the target, by value or by whether they're blank. On the Titanic, boat is blank for 99% of the people who died and filled in for 95% of those who survived: it was recorded after the sinking.
• Blanks related to the target, such as missing ages, where filling the gap would hide information unless a flag column keeps it.
• The target itself gets no fill, encoding or outlier advice; rows without a target are dropped.
• Repeated identifiers: an ID column whose values repeat (several admissions per patient, several bookings per customer) gets a warning. With profile(..., group="patient_id"), pipeline() splits by that column (StratifiedGroupKFold or GroupKFold), so the same patient is never in both training and test folds. A random split lets the model recognise patients it has already seen, and scores look better than they will be on new patients.

report.pipeline() turns the suggestions into a starter script. Steps that learn nothing from the data (type fixes, log transforms) run before the split. Steps that learn from it (fill values, outlier bounds, categories) go inside a scikit-learn Pipeline of dstoolkit transformers, so cross-validation refits them on each training fold. Date columns become month and day-of-week features (not year: new data comes from years the model never saw), and a date of birth becomes an age on another date in the data. Leaks and identifiers are left out of the features, with a comment saying why:

# 2. Features and target
# Left out (possible leaks: check they were known before the outcome): boat
# Left out (suggested for dropping): cabin, body
# Left out (identifiers or free text): name, ticket, home.dest
X = df[['pclass', 'sex', 'age', 'sibsp', 'parch', 'fare', 'embarked']]
y = df['survived']

# 3. Steps that learn from the data: inside the pipeline, refitted on each training fold
model = make_pipeline(
    dst.MissingFiller(numeric='median', categorical=None, columns=['age', 'fare']),
    dst.MissingFiller(categorical='most_frequent', numeric=None, columns=['embarked']),
    dst.CategoryEncoder({'sex': 'label', 'embarked': 'onehot'}, drop_original=True),
    HistGradientBoostingClassifier(learning_rate=0.03, max_iter=200, max_depth=3, min_samples_leaf=40, random_state=0),
)

It ends by scoring a no-skill baseline on the same folds, so the result has something to beat: on the Titanic, ROC AUC 0.859 against 0.500 for always guessing the most common class. The model accepts blanks and needs no scaling. Its trees are small and learn slowly: with scikit-learn's default settings it fits small tables too closely, and on the hospital demo its risks were overconfident (the riskiest tenth predicted 63% against 48% actually readmitted). The gentler settings fix that and rank better too; the script says what to add before swapping in one that doesn't, such as logistic regression.

It's a starting point to review, not a finished model: whether a flagged column really leaks, and which model to use, are your decisions. Two parameters record those decisions:

• profile(..., after_outcome=["body"]) marks columns you know were recorded after the outcome. The statistics can miss them: on the Titanic, body (a body identification number) only exists for people who died, but so few bodies were recovered that its blanks look only moderately related to survival.
• report.pipeline(missing_flags=["age"]) keeps a 0/1 flag for blanks that carry signal. Flags are never added unless you ask, because a blank recorded after the outcome leaks the answer.
• report.pipeline(time_column="admitted") splits by time: rows are sorted by that date, and each fold trains on earlier rows and tests on later ones. Random and grouped folds mix past and future; a time split is the honest estimate for a model that will predict next month's cases.

Dates are checked for leaks too: each date, and the days between each pair of dates, are tested against the target. Predicting length of stay, discharged minus admitted is the answer, so the report flags discharged (the later date) as a possible leak. Dates written both day-first and month-first are never guessed at: the report counts how many can only be read one way and how many could be either, so they can be fixed at the source.

Dataset size

Timed with 16 columns (numbers, categories and an ID), charts and a target: about 3 seconds for 10,000 rows, 6 seconds for 100,000, and 50 seconds for a million. For large data, profile(df, sample=20_000) profiles a fixed random 20,000 rows (about 5 seconds at a million rows); the row, duplicate and missing counts and any validation still use every row, and the report says what was sampled. The related-columns section is skipped above 30 columns.

Clean in one call

df = dst.clean(
    raw,
    missing={"numeric": "median", "columns": ["age", "fare", "embarked"]},   # every step is opt-in
    outliers={"columns": ["fare"], "action": "cap"},
    encode={"columns": {"sex": "label", "embarked": "onehot"}},
)
# No duplicates found.
# Filled missing values: age (263), fare (1), embarked (2)
# Capped outliers: fare (171)
# sex labels: female=0, male=1
# Encoded sex (label), embarked (onehot).

dst.show_distribution_profile(df["age"], title="Passenger Age")

Each cleaning step prints a one-line summary of what it changed; pass verbose=False to silence it. Apart from removing exact duplicate rows, clean() changes nothing you don't ask for, and rows keep their original labels (the index), so any row can be traced back to the raw data.

Messages and logging

The summaries go through Python's standard logging (logger name "dstoolkit", INFO level). By default they're simply printed, so nothing changes in notebooks. In a pipeline or a scheduled job, send them to your own logging setup instead, so they get timestamps and land wherever your logs go (a file, AWS CloudWatch from a Lambda function):

import logging
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(name)s %(message)s")
dst.set_message_output("logging")   # or "print" (default), or "off" to silence everything

Validate against your own rules

rules = {
    "age": {"type": "number", "min": 0, "max": 100, "required": True},
    "sex": {"allowed": ["male", "female"]},
    "fare": {"min": 0.01},
    "ticket": {"required": True},
}
check = dst.validate(raw, rules)
# Validated 1,309 rows against 7 rule(s): 2 failed (271 row(s)).
#   age: required (no blanks) failed for 263 row(s), e.g. row 15 = missing; row 37 = missing; row 40 = missing
#   fare: min (≥ 0.01) failed for 17 row(s), e.g. row 7 = 0; row 70 = 0; row 125 = 0

check.ok                       # False
check.summary()                # one row per rule: checked, failed, examples
check.failures()               # every failing value, with the reason
raw.loc[check.failing_index]   # the rows themselves
dst.validate(dst.clean(raw), rules).raise_if_failed()   # stop a pipeline on bad data

checks compare two columns in each row, with ==, !=, <, <=, > or >=:

dst.validate(stays, checks={
    "discharged on or after admission": ("discharge_date", ">=", "admission_date"),
})
# Validated 2 rows against 1 rule(s): 1 failed (1 row(s)).
#   discharged on or after admission: (discharge_date >= admission_date) failed for 1 row(s), e.g. row 1 = 2024-01-30 vs 2024-02-01

validate() reports problems; it never changes or converts the data. The text "12" is not a number, 1 is not True, and date text is not read as a date. Missing values are skipped by every rule except required, a column that doesn't exist is reported (with the closest name), and a mistake in the rules themselves raises before any data is read. Pass the same rules= and checks= to profile() to add a Validation section to the report.

Why not pandera or Great Expectations? They're the right tools for enforcing data contracts in production pipelines. validate() is for the exploration stage: it reports every failing row with examples instead of stopping at the first error, uses the same no-guessing rules as the rest of the toolkit, shows up in the same HTML report as the profile, and adds no dependencies. The rule format is experimental until 1.0.

Check new data before scoring it: dst.compare()

A model only works on data like the data it learned from. compare() checks every column of a new batch against the reference data and returns one row per column, biggest changes first:

changes = dst.compare(train, new_batch)
changes[changes["status"] != "stable"]
# Compared 13 column(s): 4 major; 9 stable.
#   admission_type  major  distribution shifted (PSI 9.48); 1 new value(s), 53.2% of new rows: ER
#   hba1c           major  distribution shifted (PSI 7.96)
#   diagnosis_code  major  7 new value(s), 8.9% of new rows: U07, ...
#   prior_admissions_12m  major  blanks 0.0% → 15.0%

A p-value (Kolmogorov-Smirnov or chi-square, adjusted for testing many columns) is reported but doesn't set the status: with thousands of rows, tiny harmless shifts are "significant". Dates are expected to move forward, so their distribution isn't compared, and identifiers (many values, mostly distinct) only get the type and blank checks. Nothing is changed. The Hospital_Readmissions notebook shows it catching a lab unit change, a relabelled admission type, a new diagnosis code and missing counts.

Transformers for machine learning: dstoolkit.transformers

The same cleaning steps, split into fit (learn from training data) and transform (apply to any data), following scikit-learn's rules. Needs the sklearn extra.

from sklearn.pipeline import make_pipeline
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_val_score

model = make_pipeline(
    dst.MissingFiller(numeric="median", categorical="most_frequent"),
    dst.OutlierCapper(columns=["fare"]),
    dst.CategoryEncoder({"sex": "label", "embarked": "onehot"}, drop_original=True),
    dst.Scaler({"age": "zscore", "fare": "zscore"}),
    LogisticRegression(max_iter=1000),
)
folds = StratifiedKFold(5, shuffle=True, random_state=0)   # shuffle: titanic3.csv is sorted by class
cross_val_score(model, X, y, cv=folds)   # cleaning is refitted on each training fold: no leakage

On the data they were fitted on, they give the same results as fill_missing, handle_outliers(action="cap"), encode_categorical and normalize. They take and return DataFrames (they need column names; inside a ColumnTransformer, call .set_output(transform="pandas")), support get_feature_names_out(), and can be cloned and pickled like any scikit-learn transformer.

Browser app

app/ is a drag-and-drop page: drop a CSV or Excel file to get the profile and a list of suggested cleaning steps, untick any you don't want, and download the cleaned CSV, the report and a Python script that repeats the same steps. Python runs inside the browser (Pyodide), so the file is never uploaded. A Content-Security-Policy makes the browser enforce that: the page may only talk to its own site and one pinned Pyodide release. It's tested end to end in a real browser in CI. It's live at dito-84.github.io/dstoolkit; see app/README.md to run it locally.

Cleaning: dstoolkit.cleaning

Profiling: dstoolkit.profiling

Validation: dstoolkit.validation

Exploration: dstoolkit.exploration

Every chart function returns its matplotlib figure, so it can be saved with fig.savefig("chart.png"). In Jupyter, end the line with ; (or assign the result) so the returned figure isn't shown twice.

Every function is also available directly from dstoolkit, as in the quick start.

Which columns are related?

dst.associations(raw)
# Tested 43 pair(s) of columns; 16 significant (adjusted p < 0.05, strength at least 0.3). Left out: name (1,307 distinct values), ...
#   column_1  column_2  kind     test      measure  strength  p_value  p_adjusted  n     note
#   pclass    fare      numeric  Spearman  rho      -0.71     ...      ...         1308
#   survived  sex       mixed    ANOVA     eta       0.53     ...      ...         1309

Each pair is tested on the rows where both values are present, so missing values never pair one row's value with another's. With many pairs, some look significant by chance, so p_adjusted corrects for the number of tests. significant_only=False returns every pair.

Notebooks

Worked examples on real data:

notebooks/learning/ keeps the tutorial notebooks the package grew out of: NumPy and pandas basics, matplotlib practice, and building the distribution functions step by step.

Project layout

src/dstoolkit/
    __init__.py       top-level API and version
    cleaning.py       duplicates, missing values, outliers, encoding, scaling, age
    profiling.py      profile() report and cleaning suggestions
    validation.py     validate() rules and checks
    monitoring.py     compare(): has new data drifted from the reference data?
    transformers.py   scikit-learn transformers (fit on training data, transform anything)
    _messages.py      where the one-line summaries go (print, logging or off)
    exploration.py    summaries, distribution plots, line fits
    py.typed          marks the package as typed
tests/                pytest suite
app/                  drag-and-drop browser app (page, Python glue, build script, tests)
notebooks/            worked examples on real data
notebooks/learning/   tutorial notebooks
docs/                 design decisions, README screenshot and an example report
data/                 StudentData.csv, titanic3.csv
.github/workflows/    CI (tests, minimum versions, browser app), the GitHub Pages deploy and the PyPI release

Development

git clone https://github.com/Dito-84/dstoolkit
cd dstoolkit
pip install -e ".[dev]"
pytest --cov   # the test suite, with coverage (CI requires at least 95%)
mypy           # type check
ruff check .   # lint: style, import order, likely bugs

The tests check every function, including that inputs are never modified, that every plot draws when values are missing, that report HTML escapes column names, that the code profile() suggests actually runs, and that the transformers learn nothing from test data. Every bug fix has a test that reproduces the bug. The browser app is tested end to end in a real Chromium, and fails on any security-policy violation. GitHub Actions runs the linter, the type check, the tests with coverage and every notebook on Python 3.10, 3.13 and 3.14 for each push and pull request, and runs the tests again with the oldest versions of pandas, NumPy, matplotlib, SciPy and scikit-learn that dstoolkit says it supports. See CHANGELOG.md for version history.

Changes go through a pull request and squash-merge. The branch rules for main are kept in .github/rulesets/protect-main.json: a pull request is required, all five CI checks must pass on an up-to-date branch, merges are squash-only, and main can't be force-pushed or deleted. A test keeps the required checks in step with the CI jobs.

Used in

• Hip & Knee Replacement Outcomes Dashboard (built on synthetic data): scripts/check_data.py profiles a discharge export on the analyst's own computer before a dashboard is built from it, using missing_ok because a blank complication or readmission means "didn't happen"
• Titanic Survival Prediction: a data-quality check before modeling. Its suggestions inform the choices, but missing ages are still filled inside each cross-validation fold so no test data leaks into training

Both use dstoolkit as an optional extra, pinned to a tested commit, so changes here can't break them.

Data

• data/StudentData.csv: 17 students' study hours and grades (two rows deliberately missing)
• data/titanic3.csv: Titanic passenger list from the Vanderbilt Biostatistics dataset, with ? for missing values
• data/synthetic_admissions.csv: made-up hospital admissions generated by data/make_admissions.py from a fixed seed; no real patients

License

MIT © 2026 Rolando Rodriguez Jr.

Credits

The cleaning steps were inspired by AutoClean by Elise Landman (MIT License), an automated preprocessing pipeline for pandas. The functions here are independent implementations.