rahasya

rahasya (रहस्य, “secret”) tells you whether a piece of text contains personal information, such as an email address, a card number, a passport number or a home address, and can mask it. It does this with regular expressions and the check digits that many identifiers carry, not with a machine-learning model. It needs no network connection and no model files, and it runs in plain Python with only fastcore as a dependency.

It finds 34 kinds of identifier, listed in PATTERNS, and people’s names when you ask for them. Everything runs in your own process. rahasya does not send text anywhere, does not write it to disk, and keeps nothing between calls apart from the numbering dictionary you can choose to pass to redact.

On the author’s test corpora, which are not part of this repository, the patterns scored a precision of 1.000 at a recall of 1.000 on a held-out set, and 0.996 on the set they were tuned against. Precision is the share of findings that were real; recall is the share of real identifiers that were found. These are measurements on those corpora, not a guarantee for your text.

Install

uv add rahasya

Checking a text

pii_report scans a text and returns what it found. has_pii is True when the text contains at least one identifying kind, identifying counts those kinds, and kinds counts every kind found. In the example below an email address, a card number and an API key are found:

r = pii_report('Invoice for ada@example.com. Card 4111 1111 1111 1111. '
                'KEY=sk-abcdefghijklmnopqrstuvwxyz123456')
r.has_pii, r.identifying

(True, {'email': 1, 'card': 1, 'secret': 1})

Every kind counts towards has_pii except IP addresses. An IP address is still reported in kinds, but a server log full of them is not usually somebody’s private information. IDENTIFYING lists the kinds that count.

redact replaces each finding with a label naming its kind:

redact('Card 4111 1111 1111 1111 charged to ada@example.com')

'Card [CARD] charged to [EMAIL]'

Numbers are read in context

Many identifiers are long runs of digits, and so are order numbers, page ids and product codes. rahasya uses two tests to tell them apart. First, an identifier with a check digit must pass it: a card number must pass the Luhn check that every payment card carries, so a sixteen-digit number that fails it is not reported. Second, a number introduced by a word that names it as a reference, such as “Order”, “ISO” or “page”, is not an identifier.

[pii_report(t).has_pii for t in
 ('Order 4111 1111 1111 1112 shipped',   # fails Luhn
  'Conforms to EN 60601-1',              # a standard, not a ZIP
  'Card 4111 1111 1111 1111')]           # this one is a card

[False, False, True]

DESIGNATOR holds those reference words: standards bodies such as ISO, EN and RFC, and words such as invoice, page, commit and order, 76 in all. rahasya checks the 48 characters before a match that starts with a digit, and drops the match when one of those words ends just before it.

Regional identifiers

Nineteen of the 34 kinds are national identifiers. Most carry a checksum that the match must pass. A few have no usable check, such as the UK National Insurance number, or only a weak one, and rahasya finds those only when a label such as “NINO” or “TFN” comes before the number.

Each check is also exported on its own, such as aadhaar_ok, pesel_ok and nino_ok. gen_aadhaar makes a valid Aadhaar number from a random generator, for test data:

import random
a = gen_aadhaar(random.Random(0))            # a valid one, for a fixture
a, aadhaar_ok(a), pesel_ok('44051401359'), nino_ok('AB123456C')

('860487647593', True, True, True)

Names are opt-in

No pattern can tell a person’s name from a place, a product or a heading, so names are not looked for unless you pass ner=True. Even then, rahasya finds only names that follow an honorific such as Mr, Mrs, Dr or Prof. Dr Charles Babbage is found and a bare Ada Lovelace is not, and names are looked for only in the first 20,000 characters (NER_CHARS). The report’s scanned_ner field says whether names were looked for, so that “no names found” can be told apart from “names not looked for”.

(pii_report('Dr Charles Babbage signed it.').has_pii,
 pii_report('Dr Charles Babbage signed it.', ner=True).identifying,
 redact('Dr Charles Babbage signed it.', ner=True))

(False, {'person': 1}, 'Dr [PERSON] signed it.')

What is here

luhn, nhs_ok, aadhaar_ok, pesel_ok and the other check functions are exported too, since a checksum is useful on its own. So are the tables behind them, such as DNI_LETTERS and NRIC_LETTERS, and luhn_digits, so a test can build a number that passes. COMPILED holds every pattern compiled, as pii_spans uses it.

Why there is no model

The author compared two machine-learning detectors with these patterns. Both had lower precision and recall and ran about 200 times slower. A DeBERTa-v3 classifier in ONNX format, which needs about a gigabyte of model weights, did better only on names without an honorific, finding 5 of 8 test names against 2 of 8. A 350-million-parameter encoder found no names at all. These comparisons were run outside this repository.

Development

The notebooks in nbs/ are the source, and the Python modules are generated from them.

pip install -e .
nbdev-prepare