Iki_PII_Masker

Do one thing well: mask PII data.

A production-grade, pipe-friendly CLI tool and Python library for data engineers and analysts who need to sanitize datasets fast — without wrestling with config files or heavyweight frameworks.

pii_masker mask data.csv --auto --strategy fake -o clean.csv

---

Features

Feature	Details
19 masking strategies	fake, redact, hash, pbkdf2, salted_hash, hmac, null, partial, truncate, keep, tokenize, pseudonymize, shuffle, anonymize, perturb, bucketize, generalize, mask_format, ner_redact
Expanded reversible crypto	AES-256-GCM, ChaCha20-Poly1305, XChaCha20-Poly1305, XSalsa20-Poly1305, AES-CCM, AES-SIV, AES-CBC-HMAC, Ascon, RSA-OAEP, ECIES, FF1/FF3-1, or KMS envelope — restore originals with --key, env var, or ~/.pii_masker/config.toml
Verification	--verify re-scans masked output for leftover PII after writing
Composite strategy	Chain multiple strategies in Python with optional reversible encryption
Dual PII detection	Column-name heuristics + cell-value scanning (detect_pii_by_value) plus optional NER scanning (detect_pii_by_ner)
Multi-engine	Polars, Pandas, DuckDB, SQLAlchemy (live DB), XML, JSONPath
18 file formats	CSV, Parquet, JSON, NDJSON, Excel, XML, Feather/Arrow IPC, ORC, Pickle, HTML, Fixed-Width (fwf), HDF5, Stata, SPSS, SAS, Avro, Delta Lake, ODS, clipboard — see Supported File Formats for per-engine coverage
Pipe-friendly	stdin → stdout, zero config required
Reproducible fakes	--seed for deterministic output in CI/testing
Dry run + report	Preview masking plan before touching any data
Token vault	tokenize / pseudonymize can use a persistent vault for cross-run consistency (--vault)
Per-column keys	--key-provider local derives a unique reversible key per column, while still using one master secret
PII detector	detect subcommand scans columns, cell values, and optionally NER entities (--ner), then prints sample values. NER is probabilistic — review results and verify before relying on entity-based masking.
Profile-driven config	ProfileConfig + ColumnRuleMap — load masking rules from YAML or Python dict
Public package API	Import directly from the package root with from Iki_PII_Masker import ... or from the façade module for a feature-oriented import surface

---

Installation

# from PyPI
pip install iki-pii-masker

Requirements: Python 3.9+

Core dependencies: rich, polars, pandas, faker, cryptography, pyarrow, openpyxl, duckdb

Optional extras:

pip install sqlalchemy psycopg2-binary   # SQLAlchemy adapter (live database)
pip install jsonpath-ng                  # JSONPath adapter (nested JSON)
pip install pyyaml                       # ProfileConfig YAML support
pip install iki-pii-masker[db]            # optional SQLAlchemy vault and database-backend features
pip install spacy>=3.7                    # Optional NER detection / redaction
python -m spacy download en_core_web_sm    # Required spaCy model for NER features
pip install boto3>=1.30.0                 # AWS KMS envelope support
pip install eciespy pyffx ff3              # Optional reversible cipher support
# XML uses stdlib xml.etree — no install needed (lxml optional for speed)

CLI framework: argparse (stdlib — no extra install needed)

---

Supported File Formats

I/O is handled per-engine by the adapters in adapters/pandas_adapter.py and adapters/polars_adapter.py. Both wrap their underlying library's native readers/writers rather than re-implementing parsing, so behavior matches:

• pandas: https://pandas.pydata.org/docs/user_guide/io.html
• Polars: https://docs.pola.rs/api/python/stable/reference/io.html

Pass --engine pandas or --engine polars (-e) to pick which one handles a given format. --format (-f) is auto-detected from the file extension where possible; a few formats (fwf, delta, clipboard) have no reliable extension and must be passed explicitly.

FileFormat	Extension(s)	Pandas engine	Polars engine
csv	.csv	read/write	read/write
parquet	.parquet	read/write	read/write
json	.json	read/write	read/write
ndjson	.ndjson, .jsonl	read/write	read/write
excel	.xlsx, .xls	read/write	read/write
xml	.xml	—	—
feather (Arrow IPC)	.feather, .ftr, .arrow	read/write	read/write (read/write_ipc)
orc	.orc	read/write	not supported
pickle	.pkl, .pickle	read/write	not supported
html	.html, .htm	read/write	not supported
fwf (fixed-width)	(pass --format fwf)	read-only	not supported
hdf5	.h5, .hdf5	read/write (key="data")	not supported
stata	.dta	read/write	not supported
spss	.sav	read-only	not supported
sas	.sas7bdat	read-only	not supported
avro	.avro	not supported	read/write
delta (Delta Lake)	(pass --format delta)	not supported	read/write
ods (OpenDocument)	.ods	not supported	read-only
clipboard	(pass --format clipboard)	read/write	read/write

Notes:

• XML goes through the dedicated create_xml_adapter (XPath-based), not Engine.polars/Engine.pandas — see the XML adapter section below.
• Picking a format your chosen engine doesn't support raises a NotImplementedError that names the other engine to use instead of failing silently or partially.
• fwf, spss, and sas are read-only because pandas itself has no writer for them.
• avro and delta are Polars-only — there's no pandas equivalent, so --engine pandas with either of those formats is rejected.
• orc, pickle, html, fwf, hdf5, stata, spss, and sas are pandas-only — Polars has no native reader/writer for them.
• Some optional formats need extra dependencies pandas/Polars already document (e.g. pyarrow for feather/ORC, tables/PyTables for HDF5, pyreadstat for SPSS/SAS, fastavro for Avro, deltalake for Delta Lake) — install them the same way you'd install them for pandas/Polars directly.

---

Subcommands

Command	Purpose
mask	Apply a masking strategy to one or more columns
unmask	Decrypt reversible masked columns back to originals
detect	Scan a file and suggest which columns contain PII
validate-profile	Validate a masking profile YAML file
examples	Print a cheat-sheet of usage patterns

---

Quick Start

Step 0 — Detect PII first

Before masking anything, run detect to see what the tool finds and review sample values:

pii_masker detect data.csv

┌─────────────┬─────────────┬──────────────────────────────────────────────────┐
│ Column      │ PII Type    │ Sample Values                                    │
├─────────────┼─────────────┼──────────────────────────────────────────────────┤
│ id          │ —           │ 1, 2, 3                                          │
│ full_name   │ name        │ Alice Smith, Bob Jones, Carol White              │
│ email       │ email       │ alice@example.com, bob@corp.org, carol@test.net  │
│ phone       │ phone       │ +1-555-0100, +1-555-0101, +1-555-0102            │
│ credit_card │ credit_card │ 4111111111111234, 5500005555555559               │
│ revenue     │ —           │ 1200.50, 980.00, 750.00                          │
└─────────────┴─────────────┴──────────────────────────────────────────────────┘

Suggested: pii_masker mask data.csv --columns full_name:email:phone:credit_card --strategy fake

Mask with realistic fake data

pii_masker mask data.csv --columns email:full_name:phone --strategy fake -o masked.csv

Auto-detect and redact (Parquet, Polars engine)

pii_masker mask data.parquet --auto --strategy redact --engine polars -o clean.parquet

Reversible masking

Encrypt columns so they can be restored later with the same key. AES-GCM is the built-in default. You can also choose from additional reversible ciphers with --reversible-cipher.

Supported reversible cipher names:

• aesgcm / aes-256-gcm / aes-192-gcm / aes-128-gcm (default)
• chacha20-poly1305
• xchacha20-poly1305
• xsalsa20-poly1305
• aes-ccm
• aes-siv
• aes-cbc-hmac
• ascon-128 / ascon-128a
• rsa-oaep
• ecies (optional dependency)
• ff1 (optional dependency)
• ff3-1 (optional dependency)
• kms-envelope (optional AWS KMS envelope integration via boto3)

# Mask
pii_masker mask data.csv \
  --columns user_id:email \
  --reversible \
  --key "my-secret-key-2024" \
  -o masked.csv

# Restore
pii_masker unmask masked.csv \
  --columns user_id:email \
  --key "my-secret-key-2024" \
  -o restored.csv

# Persist tokens across runs for tokenize / pseudonymize
pii_masker mask data.csv \
  --columns email \
  --strategy tokenize \
  --vault \
  --vault-path ~/.pii_masker/vault.db \
  --key "vault-secret" \
  -o masked.csv

Vault originals are encrypted at rest using AES-GCM with a master key derived from the resolved secret and salt. Re-running the same masking operation with the same vault file returns the same token for the same input.

The vault secret can be supplied from `--key`, `PII_MASKER_KEY`, `--key-provider-config`, or `~/.pii_masker/config.toml`.

# Use SQLAlchemy-backed token vault with a connection URL
pii_masker mask data.csv \
  --columns email \
  --strategy tokenize \
  --vault \
  --vault-backend sqlalchemy \
  --vault-url sqlite:////tmp/vault.db \
  --key "vault-secret" \
  -o masked.csv

# Use per-column reversible keys with a single master secret
pii_masker mask data.csv \
  --columns email:phone \
  --reversible \
  --key-provider local \
  --key "secret123" \
  -o masked.csv

```bash
# AWS KMS envelope masking
pii_masker mask data.csv \
  --columns user_id:email \
  --reversible \
  --reversible-cipher kms-envelope \
  --kms-provider aws \
  --kms-key-id alias/my-key \
  --kms-region us-east-1 \
  --kms-encryption-context purpose=pii-mask \
  -o masked.csv

# AWS KMS envelope restore
pii_masker unmask masked.csv \
  --columns user_id:email \
  --kms-provider aws \
  --kms-region us-east-1 \
  --kms-encryption-context purpose=pii-mask \
  -o restored.csv

Install optional KMS support with:

pip install .[kms]

# Use ChaCha20-Poly1305 instead of the default AES-GCM
pii_masker mask data.csv \
  --columns user_id:email \
  --reversible \
  --reversible-cipher chacha20-poly1305 \
  --key "my-secret-key-2024" \
  -o masked.csv

You can also supply the same secret from an environment variable or a user config file:

export PII_MASKER_KEY=my-secret-key-2024
pii_masker mask data.csv --columns user_id:email --reversible -o masked.csv

Secret resolution order:

• --key CLI option
• PII_MASKER_KEY environment variable
• --key-provider-config <path> CLI option
• ~/.pii_masker/config.toml

Vault-backed token reversal also resolves the same secret, so --vault uses the same key order when deriving the vault master key or per-column keys.

Encrypted values are stored as ENC:<base64-token> — safe to round-trip through CSV, Parquet, and JSON.

pii_masker unmask also resolves the same secret from the same sources.

Profile-driven masking

Load a masking profile from YAML instead of passing columns and strategy on the command line:

pii_masker mask data.csv \
  --profile masking_profile.yaml \
  --verify \
  -o masked.csv

Use validate-profile to ensure a profile file is valid before running it in production:

pii_masker validate-profile masking_profile.yaml

Pipe-friendly

cat raw.csv | pii_masker mask --format csv --strategy fake > clean.csv

cat data.csv \
  | pii_masker mask --format csv --columns email --strategy redact \
  | gzip > masked.csv.gz

Partial masking

Keep the last N characters, mask the rest with *:

pii_masker mask data.csv \
  --columns credit_card:phone \
  --strategy partial \
  --partial-keep 4 \
  --partial-side right \
  -o masked.csv

4111111111111234  →  ************1234
+1-555-867-5309   →  *************309

Dry run with report

Preview exactly what would be masked before writing anything:

pii_masker mask data.csv --auto --strategy fake --dry-run --report

Reproducible fake data (CI / snapshot tests)

pii_masker mask data.csv --columns email:name --strategy fake --seed 42 -o masked.csv

Hash with salt

pii_masker mask data.csv \
  --columns user_id \
  --strategy hash \
  --salt "pepper_$(date +%Y)" \
  -o hashed.csv

PBKDF2 hashing with secret key

pii_masker mask data.csv \
  --columns user_id:email \
  --strategy pbkdf2 \
  --key "super_secret_pbkdf2_key_2024" \
  -o pbkdf2_hashed.csv

Null out sensitive columns

pii_masker mask report.xlsx \
  --columns ssn:dob \
  --strategy null \
  --engine pandas \
  -o clean.xlsx

---

All Strategies

Strategy	Output example	Reversible?	Best for
fake	alice@fake.com	No	Realistic test/dev data
redact	[EMAIL]	With --reversible	Audit logs, shared reports
hash	SHA:3d7a2c1e9b4f	With --reversible	Join keys, deduplication
pbkdf2	PBKDF2:3d7a2c1e...	No	Strong one-way hashing with a secret key or salt
salted_hash	SALT:3d7a2c1e...	No	Salted deterministic hashing
hmac	HMAC:3d7a2c1e...	No	HMAC-SHA256 keyed deterministic hashing
null	null	No	Dropping PII for analytics
partial	****1234	No	Card numbers, phone numbers
truncate	john...	No	Preserve prefix, discard the rest
keep	original value	N/A	Whitelisting non-PII columns
tokenize	TOK-3d7a2c1e	Via token table	Stable opaque tokens; cross-run lookup possible
pseudonymize	Barbara Clark	Via mapping dict	Consistent fakes — same input → same fake output
shuffle	random order	No	Randomize values within one column
anonymize	ANON-001	No	Generic anonymous placeholders
perturb	34.2 / 987.0	No	Slight noise for analytics-safe numeric values
bucketize	20-30	No	Coarse numeric bucketing
generalize	30-40 / 1990	No	Analytics bucketing — ages, dates, zip codes
mask_format	xxxx@xxxxxxx.xxx	No	Format-preserving masking; keeps separators
ner_redact	Met [PERSON] in [GPE]	No	NER-based redaction inside free-text values

New strategy details

tokenize — replaces each value with a stable TOK-<hex> token. The same input always maps to the same token within a run. Access the lookup table via TokenizeStrategy.token_table or reverse a token with .detokenize(token).

pseudonymize — like fake but consistent: the same real name always becomes the same fake name. This preserves referential integrity across tables — a user_id that appears in five tables will map to the same fake ID in all five after masking.

generalize — coarsens precise values into broader ranges. Numerics become range buckets (34 → 30-40), dates are truncated to year or month (1990-07-15 → 1990), and strings are prefix-masked (SW1A2AA → SW1****).

mask_format — replaces alphanumeric characters with * while keeping structural separators (., -, @, spaces, brackets) in place. An email like john@corp.com becomes xxxx@xxxx.xxx — the shape is preserved so format-sensitive downstream systems still parse it correctly.

ner_redact — detects named entities inside free-text values and replaces them with token labels like [PERSON], [GPE], [ORG]. This is useful for unstructured notes, comments, or addresses where column names alone are not enough.

Note: NER detection is probabilistic and may produce false positives or false negatives. Use --verify for structured PII and review suggested results before masking.

---

Full Option Reference

pii_masker mask

Arguments:
  [INPUT_FILE]              Input file path. Omit to read from stdin.

Options:
  -o, --output PATH         Output file path. Omit to write to stdout.
  -c, --columns TEXT        Colon-separated column names. e.g. email:name:phone
  -s, --strategy STRATEGY   fake|redact|hash|pbkdf2|salted_hash|hmac|null|partial|truncate|keep|
                            tokenize|pseudonymize|shuffle|anonymize|perturb|bucketize|
                            generalize|mask_format [default: redact]
  -e, --engine ENGINE       polars|pandas|duckdb  [default: polars]
  -f, --format FORMAT       csv|parquet|json|ndjson|excel|xml|feather|orc|
                            pickle|html|fwf|hdf5|stata|spss|sas|avro|delta|
                            ods|clipboard (auto-detected from extension where
                            possible — see Supported File Formats)
      --auto                Auto-detect PII columns by name heuristics
      --reversible          Use reversible encryption for masked values
      --reversible-cipher [aesgcm|chacha20-poly1305|aes-ccm|aes-siv|aes-cbc-hmac|rsa-oaep|ecies|ff1|ff3-1|kms-envelope]
                            Choose the reversible cipher for --reversible
      --key TEXT            Secret key for reversible masking
      --kms-provider TEXT   KMS provider used by kms-envelope (default: aws)
      --kms-region TEXT     KMS region for kms-envelope operations
      --kms-key-id TEXT     KMS key identifier required for kms-envelope
      --kms-encryption-context TEXT
                            KMS encryption context entries for kms-envelope
      --profile PATH        Load masking columns/rules from a YAML profile
      --verify              Verify masked output for remaining PII after write
      --salt TEXT           Salt prepended before hashing  [default: ""]
      --seed INTEGER        RNG seed for reproducible fake data
      --partial-keep INT    Number of characters to keep  [default: 4]
      --partial-side TEXT   Which side to keep: right|left  [default: right]
      --dry-run             Preview masking plan without writing output
      --report              Print a masking summary table after processing
      --no-progress         Disable the progress bar

pii_masker unmask

Arguments:
  [INPUT_FILE]              Input file path. Omit to read from stdin.

Options:
  -o, --output PATH         Output file path. Omit to write to stdout.
  -c, --columns TEXT        Colon-separated columns to decrypt  [required]
      --key TEXT            Secret key used during masking  [required]
  -e, --engine ENGINE       polars|pandas|duckdb  [default: polars]
  -f, --format FORMAT       csv|parquet|json|ndjson|excel|feather|orc|pickle|
                            html|fwf|hdf5|stata|spss|sas|avro|delta|ods|
                            clipboard (see Supported File Formats)

--key can also be omitted when PII_MASKER_KEY is set or ~/.pii_masker/config.toml

pii_masker detect

Arguments:
  [INPUT_FILE]              Input file path. Omit to read from stdin.

Options:
  -f, --format FORMAT       csv|parquet|json|ndjson|excel|feather|orc|pickle|
                            html|fwf|hdf5|stata|spss|sas|avro|delta|ods|
                            clipboard (see Supported File Formats)
  -e, --engine ENGINE       polars|pandas|duckdb  [default: polars]
      --samples INTEGER     Sample values to show per column  [default: 3]

pii_masker validate-profile

Arguments:
  PROFILE_FILE             YAML profile path to validate

Validate a YAML masking profile before using it in production.

---

Python API

Every feature is accessible from the package root or the façade module. The package-root imports below are the recommended public API for day-to-day use.

from Iki_PII_Masker import (
    detect_pii,
    detect_pii_by_value,
    detect_pii_by_ner,
    mask_dataframe,
    unmask_dataframe,
    load_data,
    save_data,
    make_context,
    make_reversible_context,
    derive_encryption_key,
    create_adapter,
    create_sql_adapter,
    create_xml_adapter,
    create_jsonpath_adapter,
    report_detection,
    report_masking,
    ProfileConfig,
    ColumnRuleMap,
    Strategy,
    Engine,
    FileFormat,
    encrypt_value,
    decrypt_value,
)

If you prefer the feature-oriented façade surface, the same helpers are still available from Iki_PII_Masker.facade.

Façade feature reference

Feature	What it does
detect_pii(columns)	Scan column names → {col: PIIType} for every PII match
detect_pii_by_value(adapter, sample_rows, threshold)	Scan actual cell values — catches generic column names like col_7
detect_pii_by_ner(adapter, sample_rows, threshold, model)	Scan free-text-like columns with spaCy NER and merge the results with the other detectors
mask_dataframe(adapter, columns, strategy, context, auto=False, dry_run=False, progress=False)	Apply any masking strategy to named columns; returns elapsed seconds
unmask_dataframe(adapter, columns, key, ...)	Reverse reversible masking in-place; supports KMS envelope options
load_data(adapter, source, fmt)	Load a file, path, BytesIO, or None (stdin) into an adapter
save_data(adapter, dest, fmt)	Write adapter data to a file, BytesIO, or None (stdout)
make_context(**kwargs)	Build a plain MaskingContext (salt, seed, partial options, cipher choice)
make_reversible_context(secret, salt=b"", **kwargs)	Build a reversible masking context with configurable cipher support
derive_encryption_key(secret)	Derive a 32-byte key from a secret string for reversible masking
create_adapter(engine)	Instantiate a Polars, Pandas, or DuckDB adapter
create_sql_adapter(url, table)	Mask a live database table via SQLAlchemy
create_xml_adapter(xpath, fields)	Mask XML documents by XPath row selector
create_jsonpath_adapter(paths)	Mask nested JSON by JSONPath expressions
encrypt_value(value, key, cipher="aesgcm", ...)	Direct helper for reversible encryption
decrypt_value(token, key, cipher="aesgcm", ...)	Direct helper for reversible decryption
ProfileConfig.from_yaml(path)	Load masking rules from a YAML file
ProfileConfig.from_dict(data)	Build masking rules from a Python dict
CompositeStrategy(strategies)	Chain multiple masking strategies together in Python
ColumnRuleMap({col: Strategy})	Per-column strategy map with a single .apply(adapter) call
report_detection(adapter, detected, file)	Print Rich PII detection table with sample values
report_masking(adapter, col_map, strategy, elapsed)	Print Rich masking summary table

Detection

Column-name detection (fast, zero I/O):

from Iki_PII_Masker.facade import detect_pii, report_detection
from Iki_PII_Masker.facade import create_adapter, load_data, Engine
from pathlib import Path

adapter  = create_adapter(Engine.polars)
load_data(adapter, Path("data.csv"))

detected = detect_pii(adapter.columns)
report_detection(adapter, detected, Path("data.csv"), samples=3)

Cell-value detection (catches generic column names like col_7):

from Iki_PII_Masker.facade import detect_pii, detect_pii_by_value

name_hits  = detect_pii(adapter.columns)
value_hits = detect_pii_by_value(adapter, sample_rows=100, existing=name_hits)
all_found  = {**name_hits, **value_hits}

Masking strategies

Fake data (reproducible):

from Iki_PII_Masker.facade import mask_dataframe, make_context, Strategy

mask_dataframe(adapter, "email:full_name:phone", Strategy.fake, make_context(seed=42))

Pseudonymize — consistent fakes (preserves referential integrity):

# Same "Alice Smith" in every table → same fake name everywhere
mask_dataframe(adapter, "full_name:email", Strategy.pseudonymize, make_context(seed=1))

Tokenize — stable opaque tokens:

# user_id → TOK-3d7a2c1e  (same input = same token within the run)
mask_dataframe(adapter, "user_id", Strategy.tokenize)

Generalize — coarsen to ranges / year buckets:

# 34 → "30-40",  1990-07-15 → "1990",  SW1A2AA → "SW1****"
mask_dataframe(adapter, "age:dob:zip", Strategy.generalize)

MaskFormat — preserve structural separators:

# john@corp.com → xxxx@xxxx.xxx,  4111-1234-5678-9000 → ****-****-****-****
mask_dataframe(adapter, "email:credit_card", Strategy.mask_format)

NER redaction — redact named entities inside free text:

# "Met Alice in Boston" → "Met [PERSON] in [GPE]"
mask_dataframe(adapter, "notes", Strategy.ner_redact)

Hash with salt:

mask_dataframe(adapter, "user_id:email", Strategy.hash, make_context(salt="pepper_2024"))

Partial masking — keep last 4 digits:

mask_dataframe(adapter, "credit_card:phone", Strategy.partial,
               make_context(partial_keep=4, partial_side="right"))

Null out sensitive columns:

mask_dataframe(adapter, "ssn:dob:password", Strategy.null)

Reversible masking — mask then restore:

from Iki_PII_Masker.facade import (
    mask_dataframe, unmask_dataframe,
    make_reversible_context, derive_encryption_key, Strategy,
)

SECRET = "my-production-secret-2024"

mask_dataframe(adapter, "email:user_id", Strategy.redact,
               make_reversible_context(SECRET))
save_data(adapter, Path("masked.csv"))

# Restore
key = derive_encryption_key(SECRET)
load_data(adapter2, Path("masked.csv"))
unmask_dataframe(adapter2, ["email", "user_id"], key)

Direct crypto helpers — encrypt/decrypt values without a dataframe:

from Iki_PII_Masker import derive_encryption_key, encrypt_value, decrypt_value

key = derive_encryption_key("my-production-secret-2024")
masked = encrypt_value("alice@example.com", key, cipher="chacha20-poly1305")
restored = decrypt_value(masked, key, cipher="chacha20-poly1305")

Composite strategy — chain strategies with optional final encryption:

from Iki_PII_Masker.facade import mask_dataframe, make_reversible_context
from Iki_PII_Masker.strategies import CompositeStrategy, RedactStrategy, MaskFormatStrategy

strategy = CompositeStrategy([
    RedactStrategy(),
    MaskFormatStrategy(),
])
ctx = make_reversible_context(
    "my-production-secret-2024",
    reversible_cipher="chacha20-poly1305",
)

mask_dataframe(adapter, "email:credit_card", strategy, ctx)

Multi-strategy pipeline on one adapter:

mask_dataframe(adapter, "email:full_name",  Strategy.pseudonymize, make_context(seed=42))
mask_dataframe(adapter, "credit_card",      Strategy.mask_format)
mask_dataframe(adapter, "dob:age",          Strategy.generalize)
mask_dataframe(adapter, "user_id",          Strategy.tokenize)
mask_dataframe(adapter, "password:ssn",     Strategy.null)

Adapters

Standard adapters (Polars / Pandas / DuckDB):

from Iki_PII_Masker.facade import create_adapter, Engine

adapter = create_adapter(Engine.polars)   # fastest general-purpose
adapter = create_adapter(Engine.pandas)   # use for Excel I/O
adapter = create_adapter(Engine.duckdb)   # use for files larger than RAM

SQLAlchemy adapter — mask a live database table:

from Iki_PII_Masker.facade import create_sql_adapter, mask_dataframe, Strategy

# Requires: pip install sqlalchemy psycopg2-binary
adapter = create_sql_adapter(
    url="postgresql+psycopg2://user:pass@localhost/mydb",
    table="users",
    id_column="id",
    chunk_size=500,
)
adapter.load()   # fetches all rows into memory
mask_dataframe(adapter, "email:phone", Strategy.fake)
adapter.save()   # writes batched UPDATEs back to the database

Supported databases: PostgreSQL, MySQL, MariaDB, SQLite, MS SQL Server, Oracle (anything with a SQLAlchemy driver).

XML adapter — mask XML documents by XPath:

from Iki_PII_Masker.facade import create_xml_adapter, load_data, save_data, mask_dataframe

# Requires no extra install — uses stdlib xml.etree (or lxml if installed)
adapter = create_xml_adapter(
    xpath="//user",                      # repeating row element
    pii_fields=["email", "phone", "name"],
)
load_data(adapter, Path("users.xml"))
mask_dataframe(adapter, "email:phone:name", Strategy.fake)
save_data(adapter, Path("masked.xml"))

JSONPath adapter — mask nested JSON:

from Iki_PII_Masker.facade import create_jsonpath_adapter

# Requires: pip install jsonpath-ng
adapter = create_jsonpath_adapter({
    "email": "$.users[*].contact.email",
    "phone": "$.users[*].contact.phone",
})
load_data(adapter, Path("data.json"))
mask_dataframe(adapter, "email:phone", Strategy.redact)
save_data(adapter, Path("masked.json"))

Profile-driven masking

ColumnRuleMap — apply per-column strategies in a single call:

from Iki_PII_Masker.facade import ColumnRuleMap, Strategy, make_context

rules = ColumnRuleMap({
    "email":       Strategy.fake,
    "full_name":   Strategy.pseudonymize,
    "credit_card": Strategy.partial,
    "ssn":         Strategy.null,
    "user_id":     Strategy.hash,
})
rules.apply(adapter, make_context(seed=42))

ProfileConfig — load rules from a YAML file:

# masking_profile.yaml
engine: polars
strategy: redact # default for auto-detected columns
seed: 42
auto: true # also auto-detect any PII not listed below
columns:
  email: fake
  full_name: pseudonymize
  credit_card: partial
  ssn: null
  user_id: tokenize
  dob: generalize
  phone: mask_format

from Iki_PII_Masker.facade import ProfileConfig, create_adapter

profile = ProfileConfig.from_yaml("masking_profile.yaml")
adapter = create_adapter(profile.engine)
load_data(adapter, Path("data.csv"))
profile.apply(adapter)
save_data(adapter, Path("masked.csv"))

Or build a profile in Python without a file:

profile = ProfileConfig.from_dict({
    "engine":   "polars",
    "strategy": "redact",
    "seed":     42,
    "auto":     True,
    "columns": {
        "email":     "fake",
        "ssn":       "null",
        "user_id":   "tokenize",
        "full_name": "pseudonymize",
    },
})
profile.apply(adapter)

Save a profile back to YAML for reuse:

profile.to_yaml("masking_profile.yaml")

In-memory pipe (BytesIO)

import io
from Iki_PII_Masker.facade import create_adapter, load_data, save_data
from Iki_PII_Masker.facade import mask_dataframe, make_context, Strategy, Engine, FileFormat

buf_in  = io.BytesIO(open("data.csv", "rb").read())
adapter = create_adapter(Engine.polars)
load_data(adapter, buf_in, FileFormat.csv)
mask_dataframe(adapter, "email:full_name", Strategy.fake, make_context(seed=99))

buf_out = io.BytesIO()
save_data(adapter, buf_out, FileFormat.csv)

---

PII Auto-Detection

Column-name detection

The --auto flag, detect command, and detect_pii() match column names against regex heuristics for ten built-in PII types:

PII Type	Matched column names (examples)
email	email, email_address, mail
phone	phone, mobile, cell, telephone, contact_number
name	full_name, first_name, last_name, username, name
address	address, street, city, state, zip, postal_code
ssn	ssn, social_security, national_id
dob	dob, date_of_birth, birthdate, birthday
ip	ip_address, ip, ipv4, ipv6
credit_card	credit_card, card_number, cc_number, pan
user_id	user_id, userid, account_id, customer_id
password	password, passwd, pwd

Cell-value detection

detect_pii_by_value() scans actual cell values with regex patterns — it catches columns with generic names (col_7, field_2) that still contain Social Security numbers, credit card numbers, emails, and so on.

from Iki_PII_Masker.facade import detect_pii, detect_pii_by_value

# Step 1 — fast name-based scan
name_hits  = detect_pii(adapter.columns)

# Step 2 — deeper value scan for anything missed
value_hits = detect_pii_by_value(adapter, sample_rows=100, threshold=0.3)

# Combined results
all_found  = {**name_hits, **value_hits}

threshold is the fraction of sampled non-null values that must match a pattern before a column is flagged (default 0.3 = 30 %).

Register a custom PII type

from Iki_PII_Masker.facade import PIIRegistry, PIIType

PIIRegistry.register(PIIType(
    name="api_key",
    patterns=[r"\bapi_key\b", r"\btoken\b", r"\baccess_key\b"],
    redact_label="[TOKEN]",
    faker_method="uuid4",
))

---

Reversible Masking — How It Works

When --reversible --key <secret> is passed (or make_reversible_context(secret) in Python):

1. A 32-byte AES key is derived from your secret using SHA-256.
2. Each value is encrypted with AES-256-GCM using a random 96-bit nonce.
3. The nonce + ciphertext + GCM tag are base64-encoded as ENC:<token> and stored in place of the original value.
4. pii_masker unmask --key <same-secret> (or unmask_dataframe) reverses step 3 → 1.

Because each value gets a fresh random nonce, identical inputs produce different ciphertext — preventing frequency analysis on the masked dataset.

Security note — key handling: The --key flag is visible in shell history and ps output. In production, pass the key via an environment variable:

export MASK_KEY=$(vault kv get -field=key secret/pii-key)
pii_masker mask data.csv --columns email --reversible --key "$MASK_KEY" -o out.csv

---

Performance

Benchmarked on a 10M-row, 500 MB CSV with 5 PII columns:

Engine	Strategy	Time	Notes
Polars	redact	~4s	Best all-rounder
Polars	hash	~5s
Polars	fake	~18s
Polars	pseudonymize	~19s	Slightly slower than fake
Polars	tokenize	~6s	Fast — SHA-256 based
Polars	generalize	~5s
Polars	mask_format	~6s
DuckDB	redact	~4s	Handles files larger than RAM
DuckDB	fake	~19s
Pandas	redact	~9s	Use for Excel I/O
Pandas	fake	~35s

Polars is the default for speed. Use DuckDB when your file is too large to fit in memory. Use Pandas only when you need Excel I/O or tight ecosystem integration. Use SQLAlchemy for masking data directly in a live database without exporting to files first.

---

Architecture

pii_masker is built around five design patterns that keep it easy to extend without touching existing code:

Strategy — each masking algorithm is an independent class. Adding a new algorithm means adding one file; no existing code changes.

Registry — PIIRegistry is the single source of truth for all PII metadata. Adding a new PII type is one entry in one place.

Adapter — all engines expose an identical interface to the rest of the codebase. Swapping or adding an engine requires one new class.

Factory — StrategyFactory, AdapterFactory, and FormatRegistry centralise all object creation so CLI functions contain zero branching logic.

Façade — facade.py is the single public door into the Python API. Every capability is exposed as a named action function so callers never import from internal sub-packages directly.

Package layout

src/Iki_PII_Masker/
├── facade.py                  ← public Python API (import from here)
├── service.py                 ← MaskingService orchestrator
├── reporter.py                ← Rich terminal output
├── cli.py                     ← argparse CLI entry point
├── app.py                     ← CLI command implementations
├── config/
│   ├── enums.py               ← Strategy, Engine, FileFormat
│   ├── registry.py            ← PIIType, PIIRegistry
│   ├── crypto.py              ← AES-256-GCM helpers
│   ├── io.py                  ← load/save routing
│   ├── value_detector.py      ← ValuePatternDetector (cell-value PII scan)
│   ├── xml_io.py              ← XMLAdapter
│   ├── jsonpath_io.py         ← JSONPathAdapter
│   ├── profile.py             ← ProfileConfig, ColumnRuleMap
│   └── utils.py               ← exit_error helper
├── strategies/
│   ├── base.py                ← BaseMaskingStrategy, MaskingContext
│   ├── redact.py
│   ├── fake.py
│   ├── hash.py
│   ├── partial.py
│   ├── null.py
│   ├── keep.py
│   ├── tokenize.py            ← TokenizeStrategy
│   ├── pseudonymize.py        ← PseudonymizeStrategy
│   ├── generalize.py          ← GeneralizeStrategy
│   ├── mask_format.py         ← MaskFormatStrategy
│   └── factory.py             ← StrategyFactory, FormatRegistry
└── adapters/
    ├── base.py                ← BaseDataFrameAdapter
    ├── polars_adapter.py
    ├── pandas_adapter.py
    ├── duckdb_adapter.py
    ├── sqlalchemy_adapter.py  ← SQLAlchemyAdapter
    └── factory.py             ← AdapterFactory

---

Integration Examples

dbt post-hook

dbt run --select sensitive_model && \
  pii_masker mask target/run/sensitive_model.csv \
    --auto --strategy fake \
    -o exports/masked_sensitive_model.csv

Apache Airflow

from airflow.operators.bash import BashOperator

mask_pii = BashOperator(
    task_id="mask_pii",
    bash_command=(
        "pii_masker mask {{ params.input }} "
        "--auto --strategy redact "
        "--engine polars "
        "-o {{ params.output }}"
    ),
    params={"input": "/data/raw.parquet", "output": "/data/masked.parquet"},
)

GitHub Actions — sanitize test fixtures

- name: Mask PII in test fixtures
  run: |
    pii_masker mask tests/fixtures/users.csv \
      --columns email:phone:full_name \
      --strategy fake \
      --seed 42 \
      -o tests/fixtures/users_masked.csv

Profile-driven CI masking

# .github/workflows/mask.yml
- name: Apply masking profile
  run: |
    python - <<'EOF'
    from Iki_PII_Masker.facade import ProfileConfig, create_adapter, load_data, save_data
    from pathlib import Path

    profile = ProfileConfig.from_yaml("masking_profile.yaml")
    adapter = create_adapter(profile.engine)
    load_data(adapter, Path("data/raw.csv"))
    profile.apply(adapter)
    save_data(adapter, Path("data/masked.csv"))
    EOF

Pre-commit hook — block raw PII from being committed

# .pre-commit-config.yaml
- repo: local
  hooks:
    - id: mask-pii
      name: Mask PII in fixture files
      language: system
      entry: pii_masker mask --auto --strategy redact --dry-run --report
      files: tests/fixtures/.*\.(csv|parquet)$

Mask a PostgreSQL table directly

from Iki_PII_Masker.facade import (
    create_sql_adapter, mask_dataframe, Strategy, make_context
)

adapter = create_sql_adapter(
    url="postgresql+psycopg2://user:pass@localhost/prod",
    table="customers",
)
adapter.load()
mask_dataframe(adapter, "email:phone:full_name", Strategy.fake, make_context(seed=42))
adapter.save()

---

Testing

The test suite lives in tests/ and covers all layers.

# Install dev dependencies
pip install -e ".[dev]"
pip install sqlalchemy jsonpath-ng pyyaml    # optional adapters

# Run all 207 tests
python -m pytest

# Run with coverage report
python -m pytest --cov=pii_masker --cov-report=term-missing

# Run a single file
python -m pytest tests/test_strategies.py -v

Test file	Scope	Tests
test_strategies.py	Unit — all 10 masking strategies	77
test_registry.py	Unit — PIIRegistry, FormatRegistry, ValuePatternDetector	23
test_adapters.py	Integration — Polars, Pandas, DuckDB, SQLAlchemy, XML, JSONPath	56
test_service.py	Unit — MaskingService + façade wrapper	19
test_profile.py	Unit — ProfileConfig + ColumnRuleMap	17
test_cli.py	End-to-end — real CLI via subprocess	15
Total		207

---

Examples

Generate sample data first

python examples/generate_sample_data.py          # creates examples/data/sample.*
python examples/generate_sample_data.py --rows 50000

The generated sample data now includes a notes free-text column used by the NER redaction example.

Python API examples (37 examples)

python examples/run_examples.py

#	Example	Façade feature used
01	Detect PII by column name	detect_pii, report_detection
02	Detect PII by cell values	detect_pii_by_value
03	Redact explicit columns	mask_dataframe, Strategy.redact
04	Fake data with seed	mask_dataframe, make_context(seed=42)
05	Pseudonymize — consistent fakes	Strategy.pseudonymize
06	Tokenize — stable opaque tokens	Strategy.tokenize
07	Generalize — ranges and year buckets	Strategy.generalize
08	MaskFormat — preserve structural separators	Strategy.mask_format
09	Hash with salt	Strategy.hash, make_context(salt=...)
10	Partial masking — keep last 4 digits	Strategy.partial, make_context(partial_keep=4)
11	Null out sensitive columns	Strategy.null
12	Reversible AES-256-GCM mask + unmask	make_reversible_context, unmask_dataframe
13	All three standard engines	create_adapter, Engine.polars/pandas/duckdb
14	SQLAlchemy — mask a live SQLite table	create_sql_adapter
15	XML adapter — XPath-based masking	create_xml_adapter
16	JSONPath adapter — nested JSON masking	create_jsonpath_adapter
17	ColumnRuleMap — per-column strategy map	ColumnRuleMap
18	ProfileConfig from dict	ProfileConfig.from_dict
19	ProfileConfig from YAML file	ProfileConfig.from_yaml, profile.to_yaml
20	Pipe simulation — BytesIO in-memory	load_data(buf, FileFormat.csv)
21	Dry run + masking report	mask_dataframe(dry_run=True), report_masking
22	Multi-strategy pipeline on one adapter	Multiple mask_dataframe passes
23	Keep strategy — preserve selected columns	Strategy.keep
24	HMAC hashing with a secret key	Strategy.hash, make_context(key=...)
25	PBKDF2 hashing — key-stretched one-way hash	Strategy.pbkdf2, make_context(key=...)
26	Direct cryptography helpers	encrypt_value, decrypt_value, derive_encryption_key
27	Truncate — preserve prefix, discard remainder	Strategy.truncate
28	Salted hash — stable one-way hash	Strategy.salted_hash, make_context(key=...)
29	HMAC hash — keyed deterministic hashing	Strategy.hmac, make_context(key=...)
30	Shuffle — randomize values within a column	Strategy.shuffle, make_context(seed=...)
31	Anonymize — generic anonymous placeholders	Strategy.anonymize
32	Perturb — slight noise for analytics-safe values	Strategy.perturb, make_context(...)
33	Bucketize — coarse value ranges	Strategy.bucketize, make_context(bucket_step=...)
34	Reversible cipher choice — ChaCha20-Poly1305	make_reversible_context(reversible_cipher=...), unmask_dataframe
35	Reversible cipher variants — AES-CCM / AES-SIV / AES-CBC-HMAC / RSA-OAEP / FF1 / FF3-1	make_reversible_context(reversible_cipher=...), unmask_dataframe
36	NER redaction — free-text entity masking	Strategy.ner_redact
37	KMS envelope — advanced optional KMS integration	reversible_cipher=kms-envelope, CLI KMS provider flags

---

Contributing

1. Fork the repo and create a feature branch.
2. Add or update tests in tests/ — run python -m pytest before pushing.
3. To register a new PII type, add a PIIType(...) entry to PIIRegistry._types — no other file needs to change.
4. To add a new masking strategy, subclass BaseMaskingStrategy, implement _apply(), register it in StrategyFactory, and add the enum value to Strategy.
5. To add a new engine, subclass BaseDataFrameAdapter, implement all required methods, and register it in AdapterFactory and the Engine enum.
6. All public Python API additions go through facade.py — internal classes are not part of the public surface.
7. New optional adapters (SQLAlchemy, XML, JSONPath) live in config/ or adapters/ and are imported lazily inside their factory functions so the core package has no extra hard dependencies.

---

License

MIT — see LICENSE for full text.