pii-sweep

Scan dataset files for personally identifiable information, with a confidence per column and a CI gate, before the data leaves your hands.

Before a dataset is shared, copied to a notebook, or pushed to a bucket, it is worth knowing whether a column quietly holds emails, card numbers or national IDs. pii-sweep samples each column, runs a set of detectors, and reports which columns look like PII and how strongly.

$ pii-sweep scan customers.parquet
severity  column        type         confidence
high      card_number   credit_card  100%
high      tax_id        ssn          98%
medium    contact       email        91%

Install

$ pip install pii-sweep                 # from PyPI, once released
$ pip install git+https://github.com/jmweb-org/pii-sweep   # latest, available now

Reads CSV, Parquet and JSON Lines through polars.

Usage

$ pii-sweep scan data.csv                 # human-readable table
$ pii-sweep scan data.parquet --json      # machine-readable findings
$ pii-sweep scan data.csv --sample 5000   # cap values scanned per column
$ pii-sweep scan data.csv --threshold 0.3 # flag at a lower match fraction
$ pii-sweep scan data.csv --check         # exit non-zero if PII is found

In CI

Stop a dataset with PII from being committed or published:

- run: pii-sweep scan data/export.parquet --check --fail-on medium

What it detects

Detectors with a checksum (cards, IBAN) are strict, so a column of random 13-digit numbers is not flagged as cards. The confidence is the fraction of sampled non-null values a detector matched; --threshold sets how high that must be to flag a column.

Exit codes

Scope

pii-sweep finds structured PII with clear patterns. It does not detect free-text names or addresses, and a clean report is not a compliance guarantee. Treat it as a fast guardrail, not a substitute for review.

License

MIT. See LICENSE.