data-detective-toolkit 0.4.0

Automated data-quality profiling: CLI, web app, and REST API for CSV intelligence reports.

🕵️ Data Detective

Automated data quality profiling in seconds. Point Data Detective at a CSV and get back an intelligent report: shape, data types, missing values, duplicates, outliers, correlations, distribution skew, ID-like columns, and actionable insights. Choose your interface: fast CLI for scripts and CI, browser-based web app for exploration, or REST API for integration.

Why Data Detective?

• Fast: profiles 1000+ rows in under a second
• Thorough: detects 20+ data quality issues (outliers, duplicates, high cardinality, mixed types, skew, etc.)
• Private: your data never leaves your machine; everything runs locally
• Flexible: use as a CLI tool, web app, Python library, or REST API
• Lightweight: no external ML services or cloud dependencies

Two ways to use it

1. CLI: data-detective analyze myfile.csv --html for scripts and CI pipelines.
2. Web app: docker compose up for interactive exploration with live charts.

Both interfaces use the same profiling engine and produce identical reports.

---

Quick start

Web app (interactive, no command line needed)

The easiest way to explore a CSV:

git clone https://github.com/murs5l/data-detective.git
cd data-detective
docker compose up --build

Then open http://localhost:8000, drag and drop a CSV, and explore. The report includes:

• Data shape and types
• Missing value heatmap
• Column Explorer: drill down into any numeric column to see histogram, boxplot, and 5-number summary
• Outlier detection (IQR and MAD methods)
• Correlation heatmap for numeric column pairs
• Insights highlighted in plain English

The browser-based UI is designed for non-technical and technical users alike: actionable insights appear first, and granular technical details are tucked behind a "Full technical report" toggle.

Insights lead the report, color-coded by severity.

Column Explorer: drill into one column at a time.

Running without Docker: if you have Python and FastAPI installed:

pip install -e ".[api]"
uvicorn backend.app.main:app --reload

Then visit http://localhost:8000.

CLI (scripting and CI/CD)

Install locally for use in scripts and automated pipelines:

pip install data-detective-toolkit
data-detective analyze myfile.csv

The PyPI package is data-detective-toolkit; the installed command is data-detective.

See CLI docs below for all flags and output formats.

CLI reference

Installation

pip install data-detective-toolkit

macOS tip: if data-detective comes back as command not found right after installing, pip likely fell back to a --user install (common on macOS's system Python, since its site-packages isn't writeable) and the resulting bin/ folder isn't on your PATH. Two fixes:

• Recommended: install with pipx instead, which installs CLI tools in an isolated environment and wires up PATH for you. Run each line separately, then restart your terminal before the last step:

  brew install pipx
  pipx ensurepath
  

  pipx install data-detective-toolkit
  

  (No Homebrew? Use python3 -m pip install --user pipx instead of the first line.)
• Or find where pip actually put it and add that to your PATH:

  python3 -m site --user-base
  

  This prints something like /Users/you/Library/Python/3.9. Add <that path>/bin to PATH in ~/.zshrc, then restart your terminal.

Basic usage

# Print a text summary to stdout
data-detective analyze myfile.csv

# Generate an interactive HTML report
data-detective analyze myfile.csv --html

# Output JSON for programmatic use
data-detective analyze myfile.csv --json | jq '.insights'

# Write to specific files
data-detective analyze myfile.csv --output-html reports/q1_data.html

Options

Flag	Description	Example
--json	Print full report as JSON to stdout	data-detective analyze data.csv --json
--html	Generate report.html in current directory	data-detective analyze data.csv --html
--output-json	Write JSON report to a specific file path	--output-json reports/profile.json
--output-html	Write HTML report to a specific file path	--output-html reports/profile.html
--outlier-method	Choose outlier detection: iqr or mad (default)	--outlier-method iqr
--quiet	Suppress non-error progress messages	--quiet

Examples

CI/CD pipeline: fail if data quality issues exceed a threshold

data-detective analyze incoming.csv --json | jq '.outliers_mad | length' | xargs -I {} bash -c 'exit {}'

Batch processing: profile all CSVs in a directory

for f in data/*.csv; do
  data-detective analyze "$f" --output-html "reports/$(basename $f .csv).html"
done

Data validation: check for specific issues before ETL

data-detective analyze input.csv --quiet --json | jq '.high_cardinality_columns'

REST API (for integration)

When running the web app or backend API server, you can integrate Data Detective into your own applications.

Endpoints

POST /api/analyze – Analyze a CSV file, return JSON report

curl -F "file=@data.csv" "http://localhost:8000/api/analyze?outlier_method=mad"

Response: JSON object with all profiling data (see example below).

POST /api/analyze/html – Analyze a CSV file, return standalone HTML report

curl -F "file=@data.csv" "http://localhost:8000/api/analyze/html" > report.html

Response: A self-contained HTML file (no external dependencies).

GET /api/health – Health check

curl http://localhost:8000/api/health

Response: {"status": "ok", "version": "0.3.0"}

Python example

import requests

with open("mydata.csv", "rb") as f:
    response = requests.post(
        "http://localhost:8000/api/analyze",
        files={"file": f},
        params={"outlier_method": "mad"}
    )

report = response.json()
print(f"Shape: {report['shape']}")
print(f"Insights: {report['insights']}")
print(f"Outliers detected: {report['outliers_mad']}")

JavaScript example

const formData = new FormData();
formData.append("file", csvFile); // File object from input[type=file]

const response = await fetch("/api/analyze?outlier_method=mad", {
  method: "POST",
  body: formData
});

const report = await response.json();
console.log(report.insights);

Full API documentation is available at /docs (Swagger UI) when the backend is running.

What it detects

Data Detective automatically flags 20+ data quality issues:

Structure & Completeness

• Shape (rows × columns) and column data types
• Missing values (count and percentage per column)
• Duplicate rows
• Duplicate columns (identical data, different names)
• Constant columns (no variation; e.g., all 1s)
• Near-constant columns (one value dominates 95%+ of rows; easy to miss since they "look" like they vary)
• Possible modeling target column (name-based, e.g. target, label, churn, y)

Statistical Issues

• Outliers: IQR method (classic box-and-whisker) or MAD (robust for skewed data)
• Highly correlated numeric pairs
• Skewed distributions (heavy left/right tails)
• Kurtosis (fat tails or sharp peaks)

Data Type & Value Issues

• High-cardinality columns (likely IDs or keys)
• Mixed-type columns (e.g., numbers and strings in the same column)
• Unexpected negative values (e.g., negative age, price, quantity)
• Date-like values not parsed as datetime

Text Columns

• Length statistics (min, max, average)
• Blank or whitespace-only values

Aggregates

• Full numeric correlation matrix
• Histograms with bin edges and counts for numeric columns
• Five-number summary (min, Q1, median, Q3, max) per numeric column
• Per-column memory footprint (KB)

Output

• A 0-100 data health score with a documented, inspectable breakdown (not a black box)
• Plain-English actionable insights highlighting the most important issues

Architecture

The CLI, web app, and REST API are three interfaces over one profiling engine, not three separate implementations. DataProfiler in src/data_detective/profiler.py holds every detector (missing values, outliers, correlations, the health score, etc.) and is the only place that logic lives. The CLI calls it directly; the FastAPI backend wraps the exact same class and exposes it over HTTP for both the web app and the REST API. Add a detector once, and it's instantly available everywhere the tool is used, no duplicated logic to keep in sync across surfaces.

data-detective/
├── src/data_detective/      # Core engine + CLI - the shared source of truth
│   ├── profiler.py          # DataProfiler: every detector, the health score
│   ├── cli.py               # `data-detective` command-line entry point
│   ├── loader.py            # CSV loading, with encoding fallback for messy files
│   ├── html_report.py       # Static HTML report renderer
│   └── report.py            # Plain-text report printer (CLI default output)
├── backend/app/             # FastAPI wrapper around the same DataProfiler
│   ├── main.py              # /api/analyze, /api/analyze/html endpoints
│   └── quick_scan.py        # Optional Go fastscan integration
├── frontend/                # Dependency-free HTML/CSS/vanilla JS web app
├── tools/fastscan/          # Optional Go speed layer (instant CSV pre-scan)
├── tests/, backend/tests/   # Core engine tests, API tests
├── docs/                    # README images/GIF, generated coverage badge
├── scripts/                 # Small maintenance scripts (e.g. coverage badge)
└── .github/workflows/       # CI (tests, lint, typecheck, coverage) + release automation

The Go speed layer is a pure optimization, not a dependency: quick_scan.py checks whether the fastscan binary is present and returns None cleanly if it isn't, so the app works identically (just without the instant pre-scan stat) on a plain pip install with no Go toolchain.

Development

# core engine + CLI
pip install -e ".[dev]"
pytest tests

# backend API
pip install -e ".[api]"
pip install -r backend/requirements-dev.txt
pytest backend/tests

# fastscan (Go speed layer)
cd tools/fastscan && go test ./...

CI runs all three (.github/workflows/tests.yml) on every push and PR.

Roadmap

• GitHub Action to run Data Detective as a data-quality gate in CI
• Support for additional sources (Parquet, JSON, database connections)
• Scheduled/hosted profiling for pipelines beyond ad-hoc uploads

License

See LICENSE.