pii-sweep
Scan dataset files for personally identifiable information, with a confidence per column and a CI gate, before the data leaves your hands.
Before a dataset is shared, copied to a notebook, or pushed to a bucket, it is
worth knowing whether a column quietly holds emails, card numbers or national
IDs. pii-sweep samples each column, runs a set of detectors, and reports which
columns look like PII and how strongly.
$ pii-sweep scan customers.parquet
severity column type confidence
high card_number credit_card 100%
high tax_id ssn 98%
medium contact email 91%
Install
$ pip install pii-sweep # from PyPI, once released
$ pip install git+https://github.com/jmweb-org/pii-sweep # latest, available now
Reads CSV, Parquet and JSON Lines through polars.
Usage
$ pii-sweep scan data.csv # human-readable table
$ pii-sweep scan data.parquet --json # machine-readable findings
$ pii-sweep scan data.csv --sample 5000 # cap values scanned per column
$ pii-sweep scan data.csv --threshold 0.3 # flag at a lower match fraction
$ pii-sweep scan data.csv --check # exit non-zero if PII is found
In CI
Stop a dataset with PII from being committed or published:
- run: pii-sweep scan data/export.parquet --check --fail-on medium
What it detects
| Type | Severity | How |
|---|---|---|
credit_card |
high | 13-19 digits passing the Luhn checksum |
iban |
high | Country format plus the mod-97 checksum |
ssn |
high | US social-security format with valid ranges |
email |
medium | Standard address pattern |
phone |
medium | International or grouped number, 9-15 digits |
ipv4 |
low | Dotted-quad address |
Detectors with a checksum (cards, IBAN) are strict, so a column of random
13-digit numbers is not flagged as cards. The confidence is the fraction of
sampled non-null values a detector matched; --threshold sets how high that
must be to flag a column.
Exit codes
| Code | Meaning |
|---|---|
| 0 | Scanned; nothing at or above the fail severity (or --check not set) |
| 1 | --check found PII at or above --fail-on |
| 2 | The file was missing or in an unsupported format |
Scope
pii-sweep finds structured PII with clear patterns. It does not detect free-text
names or addresses, and a clean report is not a compliance guarantee. Treat it
as a fast guardrail, not a substitute for review.
License
MIT. See LICENSE.
Metadata
Release files for pii-sweep 0.2.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| pii_sweep-0.2.0.tar.gz | 9.9 kB | Without provenance | Details for pii_sweep-0.2.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| pii_sweep-0.2.0-py3-none-any.whl | Python 3 | none | any | Details for pii_sweep-0.2.0-py3-none-any.whl |
Total release size: 20.0 kB
Release files / pii_sweep-0.2.0.tar.gz
| Download URL | pii_sweep-0.2.0.tar.gz |
|---|---|
| Size | 9.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
2cd46c52ea8f608224f2681374c5e66647af6c20454d35654e5f3424db3092a5
|
|
BLAKE2b-256 checksum How to use checksums |
3e1426a8be7130d7ca0041225207c791c0ea57a26bc4b8b046e5bb37e67afc08
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.11.20 {"installer":{"name":"uv","version":"0.11.20","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Release files / pii_sweep-0.2.0-py3-none-any.whl
| Download URL | pii_sweep-0.2.0-py3-none-any.whl |
|---|---|
| Size | 10.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
7532228930777050a8602247d22c7123a459ca7d6d64c002fd187cce72792c1f
|
|
BLAKE2b-256 checksum How to use checksums |
0210d787e03d563c26c5aae30869a4772c9670e6d68a9dfeb738f4262b461a03
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.11.20 {"installer":{"name":"uv","version":"0.11.20","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Security for pii-sweep 0.2.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.