privyscope-en

English PII detection & masking engine — part of the privyscope series. Detects and masks person names, phone numbers, national IDs, emails, addresses, financial info, private dates, and credentials in English text.

⚠️ privyscope is a redaction aid, not an anonymization or compliance guarantee. See Limitations.

Install

pip install privyscope-en

60-second quickstart

Python

from privyscope_en import Privyscope

engine = Privyscope.from_pretrained()                 # downloads ONNX weights on first run
result = engine.redact("John Smith's number is 555-123-4567")

result.masked_text        # "<PER>'s number is <PHONE>"
result.detected_spans     # [DetectedSpan(label="PER", start=0, end=10, ...), ...]
result.summary            # {"span_count": 2, "by_label": {"PER": 1, "PHONE": 1}, ...}

CLI

privyscope redact "John Smith's number is 555-123-4567"
cat notes.txt | privyscope redact --operating-point high_recall

Documentation

Full guides live in docs/ — organised by what you want to do:

Entities

Base (regex + NER): PER · PHONE · ID_NUM · EMAIL · LOC · BANK · DATE · SECRET.

Regex-only extended entities: EIN · PASSPORT · DRIVER_LICENSE · MEDICAL · CRYPTO · IP · DEVICE · URL — see privyscope_en/entity_config.yaml.

Stage-1 patterns come from the pii-pattern-engine ruleset. Most carry a verification function — a checksum or dictionary validator (luhn, us_ssn_valid, iban_mod97) that a match must pass before it is redacted, so a number that merely looks like a card or SSN is left alone.

privyscope_en/regex_rules.yaml is generated by scripts/gen_regex_rules.py and is overwritten on every build — edit the mapping in that script, not the YAML. See CONTRIBUTING.

How it works

A two-stage hybrid pipeline (SRS §3.4), results merged via Union:

1. Regex filter — structurally obvious PII (phone, email, IDs, cards, secrets).
2. ONNX NER — a BIOES token classifier with a constrained Viterbi decoder for contextual PII (names, addresses, private dates).

Inference is ONNX Runtime only — no PyTorch at runtime. Recall-first, with runtime operating-point tuning (no retraining). PyTorch is needed only to fine-tune.

Model & performance

• Architecture — roberta-base encoder → BIOES token-classification head → constrained Viterbi decoder.

• Runtime artifact — INT8-quantized ONNX, ~120 MB (under the ≤ 150 MB budget), max sequence length 256. Weights download from Hugging Face Hub on first use, with a SHA-256 checksum.txt for integrity verification.

• Accuracy — entity-level strict micro-F1 = 0.839 on a held-out validation set (1,000 sentences, disjoint from training; typed/strict scoring over the full regex + NER pipeline). Per-entity strict F1:

  PER	LOC	DATE	EMAIL	BANK	PHONE	SECRET
  0.83	0.83	0.98	0.81	0.92	0.95	0.79

  This validation set is a deliberately hard stress test — multi-locale names plus spoken / typo / space-collapsed registers — scored on strict offset match, so the numbers are a conservative lower bound; clean text scores higher. In particular the EMAIL figure is depressed by noisy gold boundaries on concatenated tokens (gold ffxkt@… vs. the model's correct uhffxkt@…), not by regex misses. Reproduce with privyscope eval --lang en your_val.jsonl; see Evaluation & Output Modes.

Example: an online-shopping inquiry

A real customer message sent to an online store (~490 chars, with a name, email, phone, address, and card number) passed straight through engine.redact().

Input

Hello, my name is Michael Anderson. I ordered a pair of running shoes, but the package has still not arrived. My registered email is michael.anderson92@gmail.com and my mobile number is 415-555-0198. The delivery address is 1420 Sunset Boulevard, Los Angeles, CA 90026. I already paid with my credit card 4539-1488-0343-6467. Could you please check the shipment status, and if the parcel is lost, refund the amount to my card? I would really appreciate a quick reply. Thank you very much.

Masked output (result.masked_text)

Hello, my name is <PER>. I ordered a pair of running shoes, but the package has still not arrived. My registered email is <EMAIL> and my mobile number is <PHONE>. The delivery address is <LOC>. I already paid with my credit card <BANK>. Could you please check the shipment status, and if the parcel is lost, refund the amount to my card? I would really appreciate a quick reply. Thank you very much.

The 5 detected spans

Limitations

• Not an anonymization/compliance guarantee; use as one layer of privacy-by-design.
• Known failure modes: under-detection of uncommon/regional names; over-redaction of public entities in ambiguous contexts; fragmented spans in heavily mixed-format text; missed SECRET for novel credential formats.
• Extra human review recommended for medical/legal/financial/government workflows.

License

Apache-2.0. Weights are distributed on Hugging Face Hub under Apache-2.0 with a checksum.txt (SHA-256) for integrity verification. Contributions welcome — see CONTRIBUTING.md.