Skip to main content

PyDataQuality

Know whether a data file is safe to use, and what changed since the last one. One command, nothing to configure.

Someone sends you a spreadsheet or a CSV. Before you build a report on it, you want to know two things: is anything wrong with it, and is it like the one they sent last time? PyDataQuality answers both in plain language, on your own computer.

$ pdq check orders_november.csv

✖ NOT SAFE TO USE   3 problems, 5 warnings
  orders_november.csv: 1,200 rows, 8 columns. Compared with orders_september.csv.

PROBLEMS
  1  1 column is missing: discount_code. It was in orders_september.csv.
  2  order_id had no repeats last time. Now 1 value appears more than once, e.g. "7100".
  3  order_date has 15 dates in the future (latest: 2031-01-01).

WARNINGS
  4  The file has 1,200 rows. Last time it had 3,000 (60% fewer).
  5  amount values have shifted. The typical value was 54.725, now it is 119.58.
  6  country has 1 value never seen before: "Togo" (10 rows).
  7  status has 1 value never seen before: "chargeback" (10 rows).
  8  1 new column not seen before: channel.

UNCHANGED  customer_id, email

Next:  pdq rows orders_november.csv     save the affected rows with reasons
       pdq accept orders_november.csv   this is fine, make it the new normal

Your data never leaves your machine. There is no account and no server.

Who it is for

Anyone who is sent data files by other people and has to trust them: analysts with a monthly export, finance and operations teams with supplier or bank files, researchers with survey downloads, data scientists with this week's extract. You do not need to know what the rules for the data should be. PyDataQuality learns them from a file you approve.

Three ways to use it

For Start with
A page in your browser People who would rather not use a terminal pdq ui
A command Quick checks, scheduled jobs and pipelines pdq check file.csv
A Python library Notebooks and scripts pdq.check(df)

All three run the same checks and give the same answer.

Install

pip install pydataquality

Python 3.8 or newer. Reads CSV, TSV, Excel, JSON and Parquet (Parquet needs pip install pyarrow).

Without a terminal

There is a desktop app for people who would rather not install Python. It carries its own copy of Python, puts PyDataQuality on the Start menu, and opens the same app as pdq ui.

  • Windows: download PyDataQuality-Setup.exe from the latest release and run it. It installs for your user only and does not ask for an administrator password.
  • Mac (Apple Silicon): download PyDataQuality-macOS.zip from the same page, unzip it and move PyDataQuality to Applications.

The downloads are attached to releases made after version 0.3.0. They are not code-signed yet, so Windows shows a "Windows protected your PC" notice (choose More info, then Run anyway) and macOS asks you to confirm in System Settings > Privacy & Security. The desktop app reads CSV, TSV, Excel and JSON. For Parquet, use the pip install.

To build them yourself, see packaging/README.md.

How it works

1. Check a file

pdq check customers.csv

The first time, there is nothing to compare with, so PyDataQuality looks for what is wrong in any file and what is wrong for the kind of column it is looking at:

It finds Example message
Exact duplicate rows 150 rows are exact copies of another row.
IDs that repeat order_id should be unique, but 2 values appear more than once.
Impossible values age has 21 impossible values: "999", "-5". An age should be between 0 and 120.
Dates in the future signup_date has 15 dates in the future (latest: 2031-01-01).
Mixed or broken dates visit_date mixes 2 date formats. Most look like 2024-01-31, but 3 rows look like "05/01/2026".
Invalid emails email has 31 values that are not email addresses, e.g. "not-an-email".
One value spelled several ways country has 3 spellings of the same value: "Nigeria", "NIGERIA", "nigeria ".
Numbers stored as text amount is mostly numbers, but 1 value is not: "TBC".
Wildly out-of-range numbers y has 2 values wildly outside the usual range (most rows are between 4.72 and 6.54): "58.9", "31.8".
Empty or mostly empty columns deck is empty in 688 rows (77%).

Real files are rarely tidy, so it copes with the usual mess: a title block above the table is skipped (and it tells you which row the table starts on), --, - and N/A in a number or date column count as empty, a pair of columns where each row fills in only one (debit and credit) is not reported as half empty, and long reference or account numbers are treated as labels, not amounts. If it cannot find the column headings at all, it says so and stops.

It works out what each column is (an identifier, a date, an email, a category, a number) before judging it, so it does not, for example, run outlier detection on ID numbers. It is deliberately quiet: a clean file gets NOTHING OBVIOUSLY WRONG.

Without an earlier file it never says "safe", because it cannot know what normal looks like yet.

2. Tell it what normal looks like

pdq accept customers.csv

This saves a small, readable file (.pdq/customers.yml, next to your data) describing the columns, their types, allowed values, ranges and how often they are empty. You write no rules.

3. Check the next one

pdq check customers_november.csv

The baseline is found automatically: dates, month names and words like "final" or "copy" in the file name are ignored, so customers_november.csv is compared with what was learned from customers_october.csv. Now it also reports:

  • columns that disappeared, appeared or changed type
  • far fewer or far more rows than last time
  • a column that used to be unique and now repeats
  • category values never seen before
  • numbers outside the range seen before, or whose distribution has shifted
  • a column that is suddenly much emptier
  • the very same file sent again

When a difference is fine, run pdq accept on the new file to make it the new normal.

4. Act on it

pdq rows customers_november.csv

Saves the affected rows to customers_november_problem_rows.csv with a _pdq_reason column saying why each row is there. Send that back to whoever produced the file.

Prefer not to use a terminal?

pdq ui

An app opens in your browser. Drop a file on it and you get the same verdict. From there you can:

  • open any finding to see the rows behind it, search them and download them
  • ask "Why?" to see how a finding was decided
  • review what was learned, column by column, before saving a file as normal
  • mark a finding as fine so it stops being reported for that dataset
  • see every dataset you have saved, with its history of checks
  • try it on a built-in sample file before using your own data

The app is served from your own computer and only answers to it. It loads nothing from the internet. Close the terminal window to stop it.

To send someone the result:

pdq report customers_november.csv

This writes a single HTML file you can email. In a notebook, pdq.check(df) displays the same summary.

Your own rules

Open the baseline file and add a rules section. Rules are written the way you would say them:

rules:
  price: [greater than 0]
  order_id: [unique, not empty]
  status: [one of paid, pending, refunded]
  ship_date: [on or after order_date]
  email: [is email]
Rule Meaning
unique no value appears twice
not empty no missing values
greater than N, at least N, less than N, at most N numeric limits
between A and B inclusive range
one of a, b, c allowed values (capital letters ignored)
is email, is date format checks
on or after X, on or before X, after X, before X X is a date, another column, or today
not in the future dates up to now
matches REGEX full regular-expression match

A rule that cannot be understood is reported as a problem. It is never skipped silently.

Try a rule without editing anything:

pdq check orders.csv --rule "price: greater than 0"

To silence a finding you have decided is fine, add its key (shown by --why) to an ignore list in the baseline:

ignore:
  - spelling:country

Your rules and ignore sections are kept when you run pdq accept again.

Command reference

pdq check FILE      check a file ("pdq FILE" does the same)
    --why               explain how each finding was decided
    --column NAME       only show findings for this column
    --only KIND         only show one kind of finding, e.g. duplicates
    --rule "COL: RULE"  add a check for this run
    --baseline PATH     compare with a specific baseline
    --no-baseline       do not compare with any earlier file
    --json              machine-readable output
pdq accept FILE     remember this file as normal
    --as NAME           dataset name, if the file name is not a good guide
pdq rows FILE       save the affected rows, with reasons
    -o PATH             where to save
pdq report FILE     write a one-page HTML summary to share
    -o PATH             where to save
pdq ui              open the drag-and-drop page in your browser
    --port N            use a specific port
    --no-browser        print the address without opening it

In a pipeline or scheduled job

pdq check exits with 0 when nothing was found, 1 for warnings, 2 for problems and 3 if the file or its baseline could not be read:

pdq check incoming/orders.csv || exit 1

In Python

import pydataquality as pdq

result = pdq.check("orders_november.csv")   # or pdq.check(df)

result.verdict        # 'ok', 'warnings' or 'problems'
result.headline       # 'NOT SAFE TO USE'
print(result)         # the same text as the command line

for finding in result.problems:
    print(finding.column, finding.message, finding.count)

bad = result.bad_rows()                     # DataFrame with a _pdq_reason column
html = result.to_html()                     # the shareable summary page
pdq.accept("orders_november.csv")           # make it the new normal

# With a DataFrame, say where the baseline lives
pdq.accept(df, path="baselines/orders.yml")
result = pdq.check(new_df, baseline="baselines/orders.yml",
                   rules={"price": ["greater than 0"]})

Reliability and privacy

  • Private. No network requests, no telemetry, no account. See SECURITY.md.
  • The browser page is local. It is served on 127.0.0.1, only answers your own computer, and deletes each dropped file after reading it.
  • It tells you when it cannot do something. An unreadable file, a damaged baseline or a mistyped rule produces a plain message saying what to do next. Nothing is skipped silently and your files are never modified.
  • Quiet by default. It is tuned to avoid false alarms, because a tool that cries wolf gets ignored. A clean file says so.
  • Tested. 100 automated tests run on Python 3.8 to 3.13.

Status

Version 0.3. What works today, and what is planned next:

Works today Planned
pdq check, accept, rows, report Building rules from menus in the browser app
Baselines and comparison with the last file Re-opening a past result after the app is restarted
Your own rules in plain sentences A note field on the summary you share
Browser app: verdict, affected rows, review before saving, datasets, history, sheet chooser, sample file Sharing baselines between the app and the command line
Python library and notebook display Checking files larger than memory

The browser app and the command line keep their baselines in different places for now: the app in ~/.pydataquality, the command in a .pdq folder beside your data.

The full plan for the browser app, screen by screen, is in docs/ui-plan.md.

What it is not

PyDataQuality checks files and DataFrames that fit in memory. It is not a warehouse monitoring platform, and it does not replace a full validation framework. If you need a managed suite of expectations across a data platform, look at Great Expectations, Soda or Pandera. If you want an exhaustive statistical profile, look at ydata-profiling (now fg-data-profiling). Use PyDataQuality when you have a file and need a quick, trustworthy answer.

Profiling API

The original profiling interface is still included for a detailed look at a single DataFrame.

analyzer = pdq.analyze_dataframe(df, name="My Dataset")

analyzer.issues                                   # QualityIssue objects with severity
analyzer.get_problematic_rows("age", "outliers")  # the rows behind an issue
pdq.generate_report(analyzer, output_path="quality_report.html")
pdq.create_visual_report(analyzer, save_path="charts/")

# Compare two datasets (Population Stability Index and KS test)
drift = pdq.compare_drift(pdq.analyze_dataframe(train_df), pdq.analyze_dataframe(new_df))

# A prompt to paste into an AI assistant. Raw values from columns that look
# like personal data are left out unless you pass include_values=True.
from pydataquality import QualityReportGenerator
prompt = QualityReportGenerator(analyzer).generate_ai_remediation_prompt()
pydataquality data.csv --report html --visualize
pydataquality data.csv --rules rules.yaml --fail-on critical

Documentation

Research paper

The profiling kernel and drift statistics are described in a preprint:

PyDataQuality: An Actionable, Lightweight Data Profiling and Distribution Drift Detection Framework for Production Machine Learning Pipelines. Dominion Akinrotimi, 2026.

Read the paper (PDF) · Technical study guide

Contributing

git clone https://github.com/DominionAkinrotimi/pydataquality.git
cd pydataquality
pip install -e ".[dev]"
pytest

The browser app is TypeScript and React in frontend/. Its built files are committed to pydataquality/static/app, so Python users never need Node. To change it:

pdq ui --port 8765 --no-browser     # the Python side
cd frontend && npm install
npm run dev                         # live reload, talks to port 8765
npm run build                       # writes pydataquality/static/app

Bug reports are most useful with a small file that shows the problem, especially a false alarm on data that is actually fine.

License

MIT. See LICENSE.

Metadata

Release files for pydataquality 0.3.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for pydataquality 0.3.1
File Size Uploaded Provenance Actions
pydataquality-0.3.1.tar.gz 272.8 kB Without provenance Details for pydataquality-0.3.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for pydataquality 0.3.1
File Interpreter ABI Platform Actions
pydataquality-0.3.1-py3-none-any.whl Python 3 none any Details for pydataquality-0.3.1-py3-none-any.whl

Total release size: 526.2 kB

Release files / pydataquality-0.3.1.tar.gz

Download URL pydataquality-0.3.1.tar.gz
Size 272.8 kB
Tags Source
SHA-256 checksum
How to use checksums
5a46faa9da79a4e91fb0181fe4163a6311dd4c59dc8e7e147c6f1f8a4799aace
BLAKE2b-256 checksum
How to use checksums
3efae45c98d97e9a07806877f9931157fa905655bf1236a3f10a3c53763a6774
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.14

Release files / pydataquality-0.3.1-py3-none-any.whl

Download URL pydataquality-0.3.1-py3-none-any.whl
Size 253.4 kB
Tags Python 3
SHA-256 checksum
How to use checksums
e4f057ba6dfc542ed99c11c7e20cc029b526d5b72ceb52b93157099faaf2d8dd
BLAKE2b-256 checksum
How to use checksums
36e31a701c08603cfcbe0f5b9617b5f668979e88ff357fb4480a97b3db2fbf6a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.14

Security for pydataquality 0.3.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page