psiwatch 0.12.0

Dataset drift detection for ML pipelines.

psiwatch

Zero-dependency Python library for dataset drift detection in ML pipelines.

Detect covariate drift, distribution shift, and data quality degradation between two datasets — using PSI, Chi-Square, Mean Shift, and Standard Deviation analysis. Pure Python. No numpy. No scipy. No pandas required.

---

The Problem

You train a model on historical data. Weeks later, predictions go wrong — silently. No errors. No alerts.

The cause is data drift. Production data no longer looks like training data:

• Customer ages shifted
• New cities or categories appeared
• Salary distributions changed
• Credit scores dropped

Most teams discover this after the model has already failed.

---

The Solution

pip install psiwatch
psiwatch compare train.csv production.csv

══════════════════════════════════════════════════════════════
  PSIWATCH REPORT
  baseline: train.csv  →  new: production.csv
  Generated: 2026-06-14 09:00:00
══════════════════════════════════════════════════════════════

  [!!] credit_score  [numeric]  — HIGH DRIFT
     → Mean shifted by 2.20 std devs (752.00 → 624.00)  ↓
     → PSI = 11.1200 (significant drift)
     ┌ Mean:    752.00 → 624.00  ↓
     ├ Std:     48.00 → 71.00
     ├ PSI:     11.1200
     ├ Min:     600.00 → 420.00
     ├ P25:     720.00 → 560.00
     ├ Median:  750.00 → 620.00
     ├ P75:     790.00 → 690.00
     └ Max:     850.00 → 800.00

  [!!] loan_type  [categorical]  — HIGH DRIFT
     → New categories found: ['BNPL', 'Crypto']
     → Categories vanished from new data: ['Personal']
     → PSI = 4.1900
     ┌ PSI:          4.1900
     ├ Chi-square:   3.8200
     ├ New cats:     ['BNPL', 'Crypto']
     └ Vanished:     ['Personal']

  [OK] customer_id  [categorical]  — STABLE
     → No drift detected

──────────────────────────────────────────────────────────────
  HIGH: 2   MEDIUM: 0   PASS: 1

  [!!] Drift Health Score: 11/100  (Significant Drift)
══════════════════════════════════════════════════════════════

---

Why psiwatch?

Most drift detection tools require heavy dependencies and are built for Jupyter notebooks — not pipelines or minimal environments.

Feature	psiwatch	evidently	alibi-detect
Dependencies	Zero	Heavy	Heavy
Install size	~15KB	~50MB+	~100MB+
Works on Termux/Android	Yes	No	No
CLI tool	Yes	No	No
CI/CD --fail-on-drift flag	Yes	No	No
Self-upgrade via CLI	Yes	No	No
pandas DataFrame support	Yes	Yes	Yes
Pure Python	Yes	No	No
Auto update check	Yes	No	No
HTML reports	Yes	Yes	No
Trend direction (↑↓→)	Yes	No	No
Vanished category detection	Yes	No	No

---

Install

pip install psiwatch

Works on Windows, Mac, Linux, VPS, Google Colab, Jupyter, and Termux on Android.

Upgrade

# via CLI (easiest)
psiwatch update

# or standard pip
pip install --upgrade psiwatch

---

Quickstart

git clone https://github.com/tharunstryker/psiwatch
cd psiwatch
pip install -e .
psiwatch compare samples/train.csv samples/new.csv

---

CLI

# Compare two CSV files
psiwatch compare old.csv new.csv

# Save as HTML report
psiwatch compare old.csv new.csv --output report.html

# Save as JSON for pipelines
psiwatch compare old.csv new.csv --output report.json

# Save as plain text
psiwatch compare old.csv new.csv --output report.txt

# Compare specific columns only
psiwatch compare old.csv new.csv --columns age,score,city

# Skip columns (IDs, timestamps, row numbers)
psiwatch compare old.csv new.csv --ignore-columns id,timestamp,row_num

# Set custom PSI threshold
psiwatch compare old.csv new.csv --psi-threshold 0.15

# Fail with exit code 1 if drift detected (for CI/CD)
psiwatch compare old.csv new.csv --fail-on-drift

# Suppress update banner (useful in scripts)
psiwatch compare old.csv new.csv --silent

# Send a Slack/Discord/webhook alert on drift
psiwatch compare old.csv new.csv --webhook https://hooks.slack.com/services/XXX/YYY/ZZZ

# Use a config file instead of passing flags every time
psiwatch compare old.csv new.csv --config myconfig.toml

# One-line health summary (no full report — ideal for shell scripts)
psiwatch summary train.csv new.csv

# Lock training data as a statistical baseline
psiwatch lock train.csv                          # creates psiwatch.lock.json
psiwatch lock train.csv --output model.lock.json

# Check new data against the lock
psiwatch check new.csv
psiwatch check new.csv --lock model.lock.json --fail-on-drift

# Show what's stored in a lock file
psiwatch lock-info

# Track drift across a sequence of datasets over time
psiwatch trend day1.csv day2.csv day3.csv day4.csv
psiwatch trend day1.csv day2.csv day3.csv --baseline first
psiwatch trend day1.csv day2.csv day3.csv --output trend.json

# Watch a directory and check new CSV files as they arrive
psiwatch watch data/
psiwatch watch data/ --once               # one pass, exit — good for cron and CI
psiwatch watch data/ --webhook https://hooks.slack.com/services/XXX --fail-on-drift

# Upgrade to latest version
psiwatch update

# Show installed version
psiwatch version

---

Python Library

CSV files

import psiwatch

psiwatch.compare("old.csv", "new.csv")
psiwatch.compare("old.csv", "new.csv", output="report.html")
psiwatch.compare("old.csv", "new.csv", columns=["age", "score"])

pandas DataFrames

import pandas as pd
import psiwatch

old_df = pd.read_csv("train.csv")
new_df = pd.read_csv("production.csv")

psiwatch.compare(old_df, new_df)
psiwatch.compare(old_df, new_df, output="report.html")

pandas is optional — psiwatch works without it. Only imported when a DataFrame is passed.

Python dicts

psiwatch.compare_data(
    old={"age": [22, 23, 21], "city": ["Chennai", "Delhi", "Mumbai"]},
    new={"age": [28, 30, 29], "city": ["Chennai", "Bangalore", "Hyderabad"]}
)

List of dicts (JSON records)

old_records = [{"age": 22, "city": "Chennai"}, {"age": 23, "city": "Delhi"}]
new_records = [{"age": 28, "city": "Mumbai"}, {"age": 30, "city": "Pune"}]

psiwatch.compare(old_records, new_records)

Single list (one column)

psiwatch.compare_columns([22, 23, 21], [28, 30, 29], name="age")

Raw results (no print)

result = psiwatch.analyze("old.csv", "new.csv")

print(result["health_score"])         # 0-100

for col, data in result["columns"].items():
    print(col, data["severity"])      # HIGH / MEDIUM / PASS
    print(col, data["metrics"])       # PSI, mean, std, chi-square, percentiles, trend_direction
    print(col, data.get("warnings"))  # mixed-type or schema warnings

---

CI/CD — Fail on Drift

Block deployments when data drifts. psiwatch exits with code 1 if health_score < 80.

GitHub Actions

- name: Check data drift
  run: psiwatch compare train.csv production.csv --fail-on-drift

Python

import psiwatch
from psiwatch import DriftDetected

try:
    psiwatch.compare("train.csv", "new.csv", fail_on_drift=True)
except DriftDetected as e:
    print(f"Drift detected: {e}")
    # send alert, stop deploy, log to monitoring

---

Self-Upgrade

psiwatch update

Runs pip install --upgrade psiwatch under the hood — same Python environment, no extra steps. Works on Termux too.

---

Custom Thresholds

# Shortcut — set HIGH boundary, medium auto-scales to 40%
psiwatch.compare("old.csv", "new.csv", psi_threshold=0.15)

# Full control
psiwatch.compare("old.csv", "new.csv", thresholds={
    "psi_medium": 0.05,
    "psi_high": 0.15,
    "mean_shift_medium": 0.2,
    "mean_shift_high": 0.5,
    "std_shift_medium": 0.2,
    "std_shift_high": 0.5,
    "category_share_shift": 0.10,
    "chi_square_medium": 0.5,
})

---

Auto Update Check

The psiwatch CLI checks PyPI for newer versions when you run a command — never on import psiwatch. The check is cached for 24 hours (so it's not a PyPI request on every run, just every CLI invocation within the cache window) and is automatically silent in CI environments (CI=true, GITHUB_ACTIONS=true, PSIWATCH_SILENT=1).

  ╔════════════════════════════════════════════════════╗
  ║  psiwatch update available: 0.9.0 → 0.10.0        ║
  ║  Run: pip install --upgrade psiwatch               ║
  ╚════════════════════════════════════════════════════╝

To suppress from the CLI:

psiwatch compare old.csv new.csv --silent

import psiwatch and library calls like psiwatch.compare(...) never trigger this check or make any network call — it's CLI-only. If you want the check inside your own script, opt in explicitly:

from psiwatch.updater import check_for_update
import psiwatch
check_for_update(psiwatch.__version__)

---

Dataset Warnings

psiwatch warns instead of failing silently when your datasets have schema mismatches.

  [WARN]
     ⚠  Columns only in baseline (skipped): ['old_feature', 'legacy_col']
     ⚠  Columns only in new data (skipped): ['new_feature']
     ⚠  Column 'income' is 72% numeric — treated as categorical. Cast to float if intended as numeric.

---

Trend Direction

Numeric columns include a trend direction — which way the mean moved:

Symbol	Meaning
↑	Mean increased in new data
↓	Mean decreased in new data
→	Mean stable

Available in terminal output, HTML report, and in result["metrics"]["trend_direction"].

---

Vanished Category Detection

Categorical columns now detect categories that existed in the baseline but are completely absent from new data — not just new categories appearing.

  → Categories vanished from new data: ['Personal', 'Auto']

---

Trend Analysis

Track how your data drifts across a sequence of files over time.

psiwatch trend monday.csv tuesday.csv wednesday.csv thursday.csv
psiwatch trend day1.csv day2.csv day3.csv --baseline first --output trend.json

--baseline previous (default) compares each file to the one before it. --baseline first compares every file back to the first (cumulative drift from training). The report shows health score per step, per-column severity and PSI over time, and flags any column that steadily worsened across the sequence.

from psiwatch import analyze_trend

result = analyze_trend(["day1.csv", "day2.csv", "day3.csv"])
print(result["overall_health_history"])   # [97, 68, 21]
print(result["worsening_columns"])        # ["age"]

---

Watch Mode

Poll a directory for new CSV files and check each one against a baseline lock as it arrives.

psiwatch lock train.csv
psiwatch watch data/ --webhook https://hooks.slack.com/services/XXX

--once is designed for cron jobs and CI. Checks current directory contents and exits. psiwatch persists which files it has already checked (mtime-based, stored in <lock>.seen.json), so repeated runs only process new or modified files.

# In a cron job or CI step:
psiwatch watch data/ --once --fail-on-drift

from psiwatch import watch_directory

result = watch_directory("data/", once=True)
print(result["drifted_files"])

---

Webhook Alerts

Send a drift notification to Slack, Discord, or any JSON endpoint when drift is detected. The alert is skipped automatically when health score >= 80.

psiwatch compare train.csv new.csv --webhook https://hooks.slack.com/services/T/B/xxx
psiwatch check new.csv --webhook https://discord.com/api/webhooks/123/abc
psiwatch watch data/ --once --webhook https://example.com/psiwatch-alert

Format auto-detected from URL host: Slack → {"text": "..."}, Discord → {"content": "..."}, anything else → full JSON payload with health_score, summary, message.

from psiwatch import compare, send_webhook

result = compare("train.csv", "new.csv")
send_webhook("https://hooks.slack.com/services/XXX/YYY/ZZZ", result)

---

Config File

Store default settings in psiwatch.toml or .psiwatchrc (JSON) in your project directory. CLI flags always win over the config file.

psiwatch.toml:

psi_threshold = 0.2
ignore_columns = ["id", "timestamp"]
fail_on_drift = true
webhook = "https://hooks.slack.com/services/XXX/YYY/ZZZ"

[thresholds]
mean_shift_high = 0.6

.psiwatchrc (JSON):

{
  "psi_threshold": 0.2,
  "ignore_columns": ["id", "timestamp"],
  "fail_on_drift": true
}

psiwatch auto-detects these files in the current directory. Use --config path/to/config.toml for a different path.

---

Output Formats

Format	Command	Use case
Terminal	default	Quick checks during development
HTML	--output report.html	Sharing with team, presentations
JSON	--output report.json	CI/CD pipelines, automation, dashboards
TXT	--output report.txt	Server logs, plain text reports

All outputs include: timestamp, source file names, per-column metrics, health score.

---

Input Modes

Input	Works with
CSV file path "old.csv"	compare()
pandas DataFrame	compare(), compare_data()
Python dict {"col": [values]}	compare(), compare_data()
List of dicts [{"col": val}, ...]	compare()
Plain Python list	compare_columns()

---

Detection Methods

Numeric columns — age, score, salary, credit score

Method	What it detects
Mean Shift	Average moved significantly
Std Deviation Shift	Spread of values changed
PSI	Overall distribution shape changed
Percentiles	Min, P25, Median, P75, Max compared
Trend Direction	Which way the mean moved (↑ ↓ →)

Categorical columns — city, grade, status, loan type

Method	What it detects
New Category Detection	Values that never existed in training data
Vanished Category Detection	Values gone from new data
Frequency Distribution Shift	Category proportions changed
PSI	Overall distribution changed
Chi-Square	Frequency mismatch is statistically significant

---

PSI Reference

PSI (Population Stability Index) is the industry standard metric for monitoring production data drift.

PSI	Status	Action
< 0.10	Stable	Model is fine
0.10 – 0.25	Moderate Drift	Monitor closely, investigate
> 0.25	Significant Drift	Retrain your model

---

Drift Health Score

Every report includes a single 0–100 score.

Score	Status	Meaning
80–100	Stable	Data is stable, model likely fine
50–79	Moderate Drift	Some columns changed — investigate
0–49	Significant Drift	Major shifts — retrain

Important: if any column is HIGH severity, the score is hard-capped at ≤50 — one bad column in a 20-column dataset does not average away into "Healthy".

---

Real World Example — Banking Data

psiwatch compare bank_2023.csv bank_2026.csv

What psiwatch caught:

• Credit scores dropped from 752 → 624 — riskier customers ↓
• Salaries dropped from 63k → 45k — lower income applicants ↓
• Loan amounts jumped from 500k → 800k — borrowing more, earning less ↑
• New loan types appeared — BNPL, Crypto (never in training data)
• Categories vanished — Personal, Auto no longer in new data
• New statuses appeared — Defaulted, Frozen
• Branches completely changed — 5 old cities gone, 5 new cities added

Health Score: 11/100 — a model trained on 2023 data would be completely blind to all of this.

---

Project Structure

psiwatch/
├── .github/workflows/
│   └── ci.yml            ← pytest on Python 3.8–3.13 + build/version check
├── src/psiwatch/
│   ├── __init__.py      ← public API + DriftDetected exception
│   ├── loader.py        ← CSV, dict, list, DataFrame input
│   ├── analyzer.py      ← PSI, mean/std, chi-square, percentiles, trend, baseline summaries
│   ├── reporter.py      ← terminal, HTML, JSON, TXT output (HTML-escaped)
│   ├── updater.py       ← PyPI version check (24h cached) + self-upgrade — CLI-triggered only
│   ├── locker.py        ← baseline locking (lock / check / lock-info) — stores fingerprints, not raw data
│   ├── trend.py         ← multi-file drift trend analysis (HTML-escaped)
│   ├── watcher.py       ← directory polling with mtime-based state
│   ├── webhook.py       ← Slack/Discord/generic webhook alerts
│   ├── config.py        ← psiwatch.toml / .psiwatchrc config loader
│   └── cli.py           ← psiwatch CLI
├── samples/
│   ├── train.csv        ← example baseline dataset
│   └── new.csv          ← example drifted dataset
├── tests/
│   ├── test_analyzer.py
│   ├── test_locker.py
│   ├── test_reporter.py
│   ├── test_trend.py
│   ├── test_updater.py
│   ├── test_webhook.py
│   └── test_config.py
├── pyproject.toml
└── README.md

---

Run Tests

pip install -e ".[dev]"
pytest

Running psiwatch tests...

PASS numeric high drift detected
PASS numeric no drift — PASS
PASS categorical new category detected
PASS categorical no drift — PASS
PASS full analyze — health score: 0/100
PASS health score clean data: 100/100
PASS column filter works
PASS list of dicts input works
PASS missing column warning fires
PASS health score hard cap: HIGH column in large dataset → 50/100

All tests passed.

---

Zero Dependencies

psiwatch uses only Python's standard library:

Module	Used for
csv	File reading
math	Statistical calculations
json	JSON output + version cache
os	File operations
argparse	CLI interface
urllib	PyPI version check
subprocess	Self-upgrade (psiwatch update)
datetime	Report timestamps

No pip conflicts. No install failures. If Python runs, psiwatch runs.

---

Changelog

v0.12.0 — security & bug-fix release (no new features)

• Fixed: psiwatch lock was storing the entire raw baseline dataset inside the lock file (under values_sample) instead of a statistical fingerprint — a 10,000-row baseline produced a multi-MB lock file containing your original training data. Lock files now store mean/std/percentiles plus a 10-bin histogram (numeric) or category frequencies (categorical) — bounded size regardless of dataset size, and no raw rows. Lock files created before this fix are detected and rejected with a message to re-run psiwatch lock.
• Fixed: HTML reports (to_html(), to_html_trend()) interpolated column names, category values, and source filenames directly into the page — and into an inline <script> block for the trend chart — with no escaping. A column name or category value containing <script>...</script> would execute when the report was opened in a browser. All interpolated content is now HTML-escaped, with an additional guard against </script> breakout in the chart's JSON payload.
• Fixed: import psiwatch made a network call to PyPI on every import (the update-check banner), even inside training pipelines, notebooks, or CI steps that never touch the CLI. The check now only runs from the psiwatch CLI itself; plain import psiwatch makes zero network calls. compare()'s silent_update parameter is now a documented no-op (kept so existing calls don't break) since the check it used to suppress no longer happens at that call site.
• Test suite converted from a standalone script with a hand-rolled pass/fail counter (no real asserts, never run by CI) into a real pytest suite across 6 files, including dedicated regression tests for all three fixes above. Added .github/workflows/ci.yml running the suite on Python 3.8–3.13 plus a package-build and version-consistency check on every push and pull request.

v0.11.0

• psiwatch trend — track drift across a sequence of datasets over time; detect worsening columns
• psiwatch watch — poll a directory for new CSV files and check each against a lock baseline; persists seen-file state across --once runs (cron/CI-safe)
• --webhook URL — send Slack, Discord, or generic JSON alert on any drift detection (compare, check, summary, watch)
• Config file support — drop a psiwatch.toml or .psiwatchrc (JSON) in your project directory to set default thresholds, columns, webhook, etc.; CLI flags always override
• analyze_trend() Python API — full programmatic access to trend result dict including worsening_columns and column_history
• watch_directory() Python API — embed directory watching in your own scripts
• send_webhook() Python API — post drift alerts to any endpoint from Python
• load_config() Python API — load and apply config files programmatically
• Webhook skips automatically when health score ≥ 80 (drift-only alerting by default)
• --once flag on watch — single-pass mode for cron jobs and CI pipelines

v0.10.1

• result["summary"] in analyze() — high_count, medium_count, pass_count, drifted_columns, stable_columns, total_columns
• Sample size warning — fires when baseline and new data differ by more than 10x (PSI unreliable at extreme size ratios)
• --ignore-columns / -x flag — skip columns by name (IDs, timestamps, row numbers)
• psiwatch summary command — one-line health score for shell scripts without a full report
• psiwatch lock / check / lock-info — baseline locking: save a statistical fingerprint of training data, ship it with your model, check against it in CI without the original CSV

v0.10.0

• psiwatch update CLI command — self-upgrade without leaving the terminal
• Trend direction (↑ ↓ →) — numeric columns now show which way the mean moved
• Vanished category detection — categories missing from new data flagged explicitly
• Version banner fixed — fixed-width box, never misaligns on any version string length
• CI detection — banner auto-suppressed when CI=true or GITHUB_ACTIONS=true
• --silent CLI flag — suppress update banner in scripts
• silent_update param in compare() — same for programmatic use
• JSON output now includes source_info field
• pyproject.toml classifiers expanded — Python 3.8–3.13, better discoverability
• vanished_categories in all output formats (terminal, HTML, TXT, JSON)

v0.9.0 (previous)

• pandas DataFrame support — pass DataFrames directly to compare()
• List of dicts input — [{"age": 22, "city": "Chennai"}, ...] supported
• --fail-on-drift CLI flag — exit code 1 when drift detected, for CI/CD pipelines
• DriftDetected exception — catch in Python for custom alerting logic
• Auto version check against PyPI, 24h cached
• Health score hard-cap — any HIGH column caps score at ≤50
• Missing column warnings — schema mismatches shown explicitly
• Mixed-type column warnings — columns 50-80% numeric now warn
• Timestamp + source in all reports
• Chi-square O(n²) → O(n)

v0.2.0

• Custom threshold configuration (psi_threshold, thresholds dict)
• Column filtering (columns parameter)
• HTML report output
• analyze() function for programmatic access

v0.1.0

• Initial release
• CSV comparison via CLI and Python API
• PSI, Mean Shift, Std Shift, Chi-Square, New Category detection
• Terminal, JSON, TXT output
• Zero dependencies

---

Related Tools

If you need heavier drift detection with statistical testing frameworks:

• evidently — full ML monitoring platform
• alibi-detect — advanced drift detection with deep learning support
• scipy.stats — statistical tests

Use psiwatch when you need something lightweight, fast, and dependency-free.

---

License

MIT © 2026 Tharun · Naeris

Built entirely on Android using Termux. No laptop. No PC. No IDE.