PII Catcher for Databases and Data Warehouses
Overview
PIICatcher is a scanner for PII and PHI information. It finds PII data in your databases and file systems and tracks critical data. PIICatcher uses two techniques to detect PII:
- Match regular expressions with column names
- Match regular expressions and using NLP libraries to match sample data in columns.
Read more in the blog post on both these strategies.
PIICatcher is batteries-included with a growing set of plugins to scan column metadata as well as metadata. For example, piicatcher_spacy uses Spacy to detect PII in column data.
PIICatcher supports incremental scans and will only scan new or not-yet scanned columns. Incremental scans allow easy scheduling of scans. It also provides powerful options to include or exclude schema and tables to manage compute resources.
There are ingestion functions for both Datahub and Amundsen which will tag columns and tables with PII and the type of PII tags.
Resources
- AWS Glue & Lake Formation Privilege Analyzer for an example of how piicatcher is used in production.
- Two strategies to scan data warehouses
Quick Start
PIICatcher is available as a docker image or command-line application.
Installation
Docker:
alias piicatcher='docker run -v ${HOME}/.config/tokern:/config -u $(id -u ${USER}):$(id -g ${USER}) -it --add-host=host.docker.internal:host-gateway tokern/piicatcher:latest'
Pypi: # Install development libraries for compiling dependencies. # On Amazon Linux sudo yum install mysql-devel gcc gcc-devel python-devel
python3 -m venv .env
source .env/bin/activate
pip install piicatcher
# Install Spacy plugin
pip install piicatcher_spacy
Command Line Usage
# add a sqlite source
piicatcher catalog add-sqlite --name sqldb --path '/db/sqldb/test.db'
# run piicatcher on a sqlite db and print report to console
piicatcher detect --source-name sqldb
╭─────────────┬─────────────┬─────────────┬─────────────╮
│ schema │ table │ column │ has_pii │
├─────────────┼─────────────┼─────────────┼─────────────┤
│ main │ full_pii │ a │ 1 │
│ main │ full_pii │ b │ 1 │
│ main │ no_pii │ a │ 0 │
│ main │ no_pii │ b │ 0 │
│ main │ partial_pii │ a │ 1 │
│ main │ partial_pii │ b │ 0 │
╰─────────────┴─────────────┴─────────────┴─────────────╯
API Usage
Code Snippet:
from dbcat.api import open_catalog, add_postgresql_source
from piicatcher.api import scan_database
# PIICatcher uses a catalog to store its state.
# The easiest option is to use a sqlite memory database.
# For production usage check, https://tokern.io/docs/data-catalog
catalog = open_catalog(app_dir='/tmp/.config/piicatcher', path=':memory:', secret='my_secret')
with catalog.managed_session:
# Add a postgresql source
source = add_postgresql_source(catalog=catalog, name="pg_db", uri="127.0.0.1", username="piiuser",
password="p11secret", database="piidb")
output = scan_database(catalog=catalog, source=source)
print(output)
# Example Output
[
['public', 'sample', 'gender', 'PiiTypes.GENDER'],
['public', 'sample', 'maiden_name', 'PiiTypes.PERSON'],
['public', 'sample', 'lname', 'PiiTypes.PERSON'],
['public', 'sample', 'fname', 'PiiTypes.PERSON'],
['public', 'sample', 'address', 'PiiTypes.ADDRESS'],
['public', 'sample', 'city', 'PiiTypes.ADDRESS'],
['public', 'sample', 'state', 'PiiTypes.ADDRESS'],
['public', 'sample', 'email', 'PiiTypes.EMAIL']
]
Plugins
PIICatcher can be extended by creating new detectors. PIICatcher supports two scanning techniques:
- Metadata
- Data
Plugins can be created for either of these two techniques. Plugins are then registered using an API or using Python Entry Points.
To create a new detector, simply create a new class that inherits from MetadataDetector
or DatumDetector.
In the new class, define a function detect that will return a PIIType
If you are detecting a new PII type, then you can define a new class that inherits from PIIType.
For detailed documentation, check piicatcher plugin docs.
Supported Databases
PIICatcher supports the following databases:
- Sqlite3 v3.24.0 or greater
- MySQL 5.6 or greater
- PostgreSQL 9.4 or greater
- AWS Redshift
- AWS Athena
- Snowflake
- BigQuery
Documentation
For advanced usage refer documentation PIICatcher Documentation.
Survey
Please take this survey if you are a user or considering using PIICatcher. The responses will help to prioritize improvements to the project.
Stats Collection
We use cookies to a analyse our traffic and features usage. We may share information about your use of our product for our social media and marketing purposes. These cookies don't collect your sensitive and/or confidential information. If you would like to opt out of these cookies, run
piicatcher --disable-stats
To Enable:
piicatcher --enable-stats
Contributing
For Contribution guidelines, PIICatcher Developer documentation.
Release files for piicatcher 0.21.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| piicatcher-0.21.2.tar.gz | 19.1 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| piicatcher-0.21.2-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 38.7 kB
Release files / piicatcher-0.21.2.tar.gz
| Download URL | piicatcher-0.21.2.tar.gz |
|---|---|
| Size | 19.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
b12ab887d53e9d411f29657d877c4451367903dd10077af21f45f062518005e6
|
|
BLAKE2b-256 checksum How to use checksums |
a3a9d6901c0027fd88229fbee4e83f3edd202a07f71bf886450985c586b7409f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.5.1 CPython/3.8.17 Linux/5.15.0-1040-azure
|
Release files / piicatcher-0.21.2-py3-none-any.whl
| Download URL | piicatcher-0.21.2-py3-none-any.whl |
|---|---|
| Size | 19.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
4f431baf43afc09340148affdbf2f6f194a4f52a451a51fbd265677c21d89ef2
|
|
BLAKE2b-256 checksum How to use checksums |
f22e81ac36cd26ec5651f2a04e551fd5f87fe19bf3341304f573a6e318665a84
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.5.1 CPython/3.8.17 Linux/5.15.0-1040-azure
|
