privyscope-en

English PII detection & masking engine — part of the privyscope series. Detects and masks person names, phone numbers, national IDs, emails, addresses, financial info, private dates, and credentials in English text.

⚠️ privyscope is a redaction aid, not an anonymization or compliance guarantee. See Limitations.

Install

pip install privyscope-en

60-second quickstart

Python

from privyscope_en import Privyscope

engine = Privyscope.from_pretrained()                 # downloads ONNX weights on first run
result = engine.redact("John Smith's number is 555-123-4567")

result.masked_text        # "<PER>'s number is <PHONE>"
result.detected_spans     # [DetectedSpan(label="PER", start=0, end=10, ...), ...]
result.summary            # {"span_count": 2, "by_label": {"PER": 1, "PHONE": 1}, ...}

CLI

privyscope redact "John Smith's number is 555-123-4567"
cat notes.txt | privyscope redact --operating-point high_recall

Documentation

Full guides live in docs/ — organised by what you want to do:

Entities

Base (regex + NER): PER · PHONE · ID_NUM · EMAIL · LOC · BANK · DATE · SECRET.

Regex-only extended entities: EIN · PASSPORT · DRIVER_LICENSE · MEDICAL · CRYPTO · IP · DEVICE · URL — see privyscope_en/entity_config.yaml.

Stage-1 patterns come from the pii-pattern-engine ruleset. Most carry a verification function — a checksum or dictionary validator (luhn, us_ssn_valid, iban_mod97) that a match must pass before it is redacted, so a number that merely looks like a card or SSN is left alone.

privyscope_en/regex_rules.yaml is generated by scripts/gen_regex_rules.py and is overwritten on every build — edit the mapping in that script, not the YAML. See CONTRIBUTING.

How it works

A two-stage hybrid pipeline (SRS §3.4), results merged via Union:

1. Regex filter — structurally obvious PII (phone, email, IDs, cards, secrets).
2. ONNX NER — a BIOES token classifier with a constrained Viterbi decoder for contextual PII (names, addresses, private dates).

Inference is ONNX Runtime only — no PyTorch at runtime. Recall-first, with runtime operating-point tuning (no retraining). PyTorch is needed only to fine-tune.

Model & performance

• Architecture — roberta-base encoder → BIOES token-classification head → constrained Viterbi decoder.

• Runtime artifact — INT8-quantized ONNX, ~120 MB (under the ≤ 150 MB budget), max sequence length 256. Weights download from Hugging Face Hub on first use, with a SHA-256 checksum.txt for integrity verification.

• Accuracy — entity-level strict micro-F1 = 0.997 on a held-out validation set (2,000 sentences, disjoint from training; typed/strict scoring over the full regex + NER pipeline). Per-entity strict F1:

  PER	LOC	DATE	ID_NUM	BANK	PHONE	SECRET
  1.00	1.00	1.00	0.99	0.99	0.98	1.00
  (I think it is overfit...)

  (EMAIL is matched deterministically by the regex stage; the sample contained no EMAIL instances.) The set is disjoint from training, so the score reflects generalization rather than memorization — out-of-distribution text (unusual names or contexts) will score lower. Reproduce with privyscope eval --lang en your_val.jsonl; see Evaluation & Output Modes.

Limitations

• Not an anonymization/compliance guarantee; use as one layer of privacy-by-design.
• Known failure modes: under-detection of uncommon/regional names; over-redaction of public entities in ambiguous contexts; fragmented spans in heavily mixed-format text; missed SECRET for novel credential formats.
• Extra human review recommended for medical/legal/financial/government workflows.

License

Apache-2.0. Weights are distributed on Hugging Face Hub under Apache-2.0 with a checksum.txt (SHA-256) for integrity verification. Contributions welcome — see CONTRIBUTING.md.