DataMedicine

Validate, diagnose, and treat tabular datasets in seconds.

DataMedicine is a focused data-quality library for machine-learning engineers, data scientists, analysts, Kaggle users, and AI researchers. Give it a CSV, Parquet, Excel file, or pandas DataFrame; receive one report that explains what is wrong, how serious it is, and how to address it.

from datamedicine import validate

report = validate("train.csv", target="churn")
report.summary()

Why DataMedicine exists

Dataset quality issues are routinely discovered after a model has been trained, deployed, or shared. Generic DataFrame operations are powerful but require knowing exactly what to look for. DataMedicine makes a quality review a normal, repeatable part of a data workflow.

CSV / Parquet / Excel / DataFrame
                │
                ▼
           validate(...)
                │
     ┌──────────┼──────────┐
     ▼          ▼          ▼
 health      diagnosis   exports
     │          │          │
     └──────► treat() ◄───┘

DataMedicine does not hide pandas. It uses pandas efficiently, keeps the returned cleaned data as a DataFrame, and adds a consistent quality layer above it.

Features

• One-line validation for CSV, Parquet, Excel, and pandas DataFrames.
• Dataset overview: shape, memory use, missingness, duplicates, schema, unique values, empty strings, constant columns, and file size.
• Quality checks: mixed data types, infinity, NaN, text whitespace, hidden Unicode, replacement characters, possible identifiers, and duplicate columns.
• ML checks: target distribution, imbalance warning, high cardinality details, correlation pairs, likely leakage, and possible IDs.
• Numeric statistics: min, max, mean, median, standard deviation, skewness, kurtosis, IQR outliers, and Z-score outliers.
• Categorical, datetime, and text-quality profiling.
• Human-readable warnings and a 0–100 health score.
• Automatic, type-aware cleaning with autofix() or report.treat().
• HTML, JSON, Markdown, and optional PDF exports.
• Fingerprints and before/after comparisons for repeatable pipelines.

Installation

DataMedicine requires Python 3.10 or newer.

pip install datamedicine

Install optional support only when needed:

pip install "datamedicine[excel]"    # XLS and XLSX
pip install "datamedicine[parquet]"  # Parquet engines
pip install "datamedicine[pdf]"      # PDF reports
pip install "datamedicine[viz]"      # matplotlib plots

For contributors:

git clone https://github.com/datamedicine/datamedicine.git
cd datamedicine
pip install -e ".[dev]"
pytest
ruff check .
mypy src

Quick start

from datamedicine import validate

report = validate("customers.csv", target="segment")
print(report.health)
report.summary()
print(report.diagnosis())

Expected output resembles:

{'score': 91, 'status': 'Good', 'severity': 'Low'}

🏥 DataMedicine Diagnosis
=========================
Overall Health: 91/100 (Good; Low)

Detected Issues
- Missing Values — 34 cells
- Outliers — 8 IQR candidates

Best practices

• Validate immediately after loading external data and again before training.
• Pass target= for supervised learning datasets; imbalance and numeric leakage checks need it.
• Treat automatic fixes as recommendations: compare before and after reports.
• Store JSON exports or fingerprints alongside model artifacts.

Common mistakes

• Do not call autofix(..., outliers="auto") blindly on a domain-critical quantity such as medical dosage. Review clipping bounds first.
• Do not assume a health score proves a dataset is fit for a particular task. It reports measurable quality, not business validity or fairness.

Validate datasets

validate(source, *, target=None, correlation_threshold=0.95, rare_threshold=0.01, zscore_threshold=3.0, **read_options)

Purpose. Load and inspect a tabular dataset, returning a ValidationReport.

Parameters. source is a pandas DataFrame or a CSV, Parquet, XLS, or XLSX path. target names an optional supervised-learning target. The thresholds tune correlation, rare category, and Z-score findings. read_options is forwarded to the relevant pandas reader, so encoding="utf-8" and sheet_name="Data" work.

Returns. A ValidationReport; validation does not mutate a supplied frame.

CSV example.

from datamedicine import validate

report = validate("data/train.csv", target="label", encoding="utf-8")

Excel example.

report = validate("sales.xlsx", sheet_name="January")

Parquet example.

report = validate("events.parquet")

DataFrame example.

import pandas as pd
from datamedicine import validate

frame = pd.DataFrame({"age": [21, None, 500], "plan": ["Free", "Pro", "Pro"]})
report = validate(frame)

Performance. Most checks are vectorized. Numeric correlation is quadratic in the number of numeric columns, so it is the dominant operation for extremely wide frames. Consider a narrower feature set for those inputs.

Read the report

ValidationReport is the central API. It contains the source-independent data needed for auditing and exporting, plus a private live DataFrame used by treat() and certain plots.

Property	Meaning
report.health_score	Integer quality score from 0 through 100.
report.health_label	Excellent, Good, Fair, Poor, or Critical.
report.health	Mapping with score, status, and action severity.
report.overview	Shape, memory, schema, duplicate, missingness, uniqueness, cardinality, and variance metrics.
report.missing	Total, percent, and missing cells by column.
report.numeric / report.outliers	Numeric statistics and IQR/Z-score findings.
report.categorical	Category frequencies, rare labels, dominance, and cardinality.
report.datetime	Missing, invalid, and duplicate timestamp findings.
report.string_quality	Whitespace, hidden Unicode, and encoding replacement findings.
report.warnings	Typed, human-readable warnings.
report.correlations	Numeric feature pairs above the configured threshold.
report.target_distribution	Class counts when a target was supplied.
report.fingerprint	Dataset identity, row/column counts, and schema.

Health score

Score	Status	Severity
95–100	Excellent	None
85–94	Good	Low
70–84	Fair	Moderate
50–69	Poor	High
0–49	Critical	Critical

The score combines missing values, duplicate rows, detected IQR outliers, and the number of warnings. It is designed for trend tracking and triage, not as a replacement for domain review.

report.summary()

Purpose. Print the concise Markdown overview.

Parameters and return value. None. It writes to standard output.

report.summary()

Use it in notebooks and CI logs. For programmatic inspection, use report properties instead of parsing console output.

report.diagnosis()

Purpose. Produce a doctor-style quality diagnosis: overall health, detected issues, recommended treatments, and an estimated manual triage time saving.

Returns. A formatted string.

print(report.diagnosis())

The estimate is a communication aid, not a service-level guarantee.

report.treat(**options)

Purpose. Clean the live validated DataFrame using autofix() options.

Parameters. Any autofix() option. With no options it removes duplicate rows, uses type-aware missing-value treatment, chooses an outlier strategy, trims spaces, and replaces infinity.

Returns. A new pandas DataFrame; the original data is not mutated.

clean = report.treat(missing="auto", outliers="auto")

This method is unavailable on a report reconstructed from JSON because JSON does not contain a DataFrame. Call autofix() directly in that case.

Clean datasets

autofix(source, **options)

Purpose. Apply selected, non-destructive transformations to a supported source. It returns a pandas DataFrame.

from datamedicine import autofix

clean = autofix(
    "train.csv",
    duplicates=True,
    missing="auto",
    outliers="auto",
    trim_spaces=True,
    normalize_text="lower",
    replace_infinity=True,
    convert_numeric=True,
    remove_constant_columns=True,
    remove_duplicate_columns=True,
)

Autofix options

Option	Purpose	Values
duplicates	Remove repeated records.	True / False
missing	Fill missing cells.	auto, mean, median, mode, zero, forward fill, backward fill, or a scalar
outliers	Clip numeric outliers.	auto, iqr, zscore
trim_spaces	Strip surrounding text whitespace.	Boolean
normalize_text	Normalize string case.	lower, upper, title, or True for lower
replace_infinity	Convert positive/negative infinity to missing values.	Boolean
convert_numeric	Convert wholly numeric text columns.	Boolean
remove_constant_columns	Remove one-value columns.	Boolean
remove_duplicate_columns	Remove content-identical columns.	Boolean
copy	Copy an in-memory frame before cleaning.	Boolean, default True

missing="auto" uses median for numeric values, forward fill for datetime values, and mode for categorical/boolean values. outliers="auto" chooses IQR for skewed numeric data and Z-score clipping otherwise.

Common mistake. copy=False intentionally permits mutation of a DataFrame. Use it only when memory constraints are understood and the caller owns the data.

Compare dataset versions

before = validate("raw.csv", target="label")
clean = before.treat()
after = validate(clean, target="label")

comparison = before.compare(after)
print(comparison)

report.compare(other_report)

Purpose. Quantify quality movement between two reports.

Returns. A ReportComparison with health-score, missing-value, duplicate, and IQR-outlier deltas, plus schema changes. Negative issue deltas indicate an improvement. Call comparison.to_dict() to persist it.

Dataset fingerprints

Fingerprints detect silent dataset changes in scheduled pipelines.

baseline = validate("january.csv")
latest = validate("february.csv")

print(latest.fingerprint)
changes = latest.compare_fingerprint(baseline.fingerprint)
print(changes["added_columns"])

The fingerprint combines schema with a stable serialization of the first 1,000 rows. It is a practical change detector, not a cryptographic integrity proof for an entire source file.

Export reports

report.export_html(path="datamedicine-report.html")

Writes a self-contained responsive HTML report. Open it locally or attach it to a pull request, experiment run, or data-quality incident.

report.export_html("artifacts/quality.html")

report.export_json(path="datamedicine-report.json")

Writes a JSON report suitable for CI artifacts and dashboards. The live source DataFrame is intentionally excluded.

report.export_json("artifacts/quality.json")

report.export_markdown(path="datamedicine-report.md")

Writes a compact Markdown summary for issues, tickets, and model cards.

report.export_markdown("artifacts/quality.md")

report.export_pdf(path="datamedicine-report.pdf")

Writes a PDF through the optional weasyprint dependency.

report.export_pdf("artifacts/quality.pdf")

Install the PDF extra first. PDF export also creates the corresponding HTML file beside the PDF as a useful diagnostic artifact.

Visualizations

Install datamedicine[viz] and use the following methods. Each returns a matplotlib Axes, so standard matplotlib customization remains available.

report.plot_missing()
report.plot_outliers()
report.plot_correlation()
report.plot_class_distribution()  # requires validate(..., target="label")

If matplotlib is absent, missing/outlier/correlation plots print a clear installation instruction and return None. Class-distribution plots also require an explicit target column.

Command line interface

Validate

datamedicine validate train.csv --target label --html quality.html --json quality.json

This prints a summary and optionally writes HTML/JSON artifacts. Use shell exit code and exported JSON in CI; do not scrape human-readable terminal output.

Fix

datamedicine fix train.csv \
  --missing auto \
  --outliers auto \
  --duplicates \
  --trim-spaces \
  --output cleaned.csv

The CLI writes CSV by default, or Parquet when --output ends in .parquet. Validate the result and compare reports before replacing a canonical dataset.

Examples

Runnable examples live in examples/. They cover CSV, Excel, Parquet, DataFrames, all export formats, diagnosis, automatic fixing, comparison, large datasets, imbalanced targets, missing values, and outliers.

Performance and scalability

DataMedicine uses pandas reductions and column-level vectorized operations for the common checks. Memory use necessarily includes the input DataFrame; file loading is delegated to pandas. For very large data:

• Prefer Parquet over CSV when possible.
• Select model-relevant columns before validation if the source is extremely wide.
• Validate a representative sample for exploratory work, then the full dataset before release.
• Use copy=False only after measuring the memory and mutation trade-off.
• Correlation analysis is capped at 200 numeric columns by default. Adjust max_correlation_columns only after considering the quadratic memory cost.

Public API reference

from datamedicine import ReportComparison, ValidationReport, autofix, validate

validate, autofix, ValidationReport, and ReportComparison are the stable public API for v1.0. Internal analyzer modules are implementation details and may evolve in minor releases.

Contributing and release quality

Run formatting, linting, typing, and tests before opening a pull request. Changes to validation rules should include focused regression tests and explain how they affect existing health scores. DataMedicine follows semantic versioning: breaking changes require a major release; new backwards-compatible checks and methods are minor releases; bug fixes are patch releases.

Maintainers should follow the repository release checklist before publishing any artifact.

License

DataMedicine is released under the MIT License.