rahasya

rahasya (रहस्य, “secret”) tells you whether a text contains personal information, such as an email address, a card number, a passport number or a home address. It can also mask that information. It uses regular expressions and the check digits that many identifiers carry. It does not use a machine-learning model. It needs no network connection. It needs no model files. It runs in plain Python and depends only on fastcore.

rahasya finds 34 kinds of identifier, listed in PATTERNS. It finds people’s names when you ask for them.

Everything runs in your own process. rahasya does not send text anywhere and does not write it to disk. It keeps no state between calls, except the numbering dictionary that you can pass to redact.

The author measured the patterns on test corpora that are not part of this repository. Precision was 1.000 and recall was 1.000 on a held-out set. Precision was 0.996 on the set that the author tuned the patterns on. Precision is the share of findings that were real. Recall is the share of real identifiers that were found. These numbers describe those corpora. They do not guarantee the same results on your text.

Install

uv add rahasya

Checking a text

pii_report scans a text and returns what it found. has_pii is True when the text contains at least one identifying kind. identifying maps each identifying kind to its finding count. kinds maps every detected kind to its finding count. The example below finds an email address, a card number and an API key:

r = pii_report('Invoice for ada@example.com. Card 4111 1111 1111 1111. '
                'KEY=sk-abcdefghijklmnopqrstuvwxyz123456')
r.has_pii, r.identifying

(True, {'email': 1, 'card': 1, 'secret': 1})

Every kind counts towards has_pii except IP addresses. IDENTIFYING lists the kinds that count. An IP address is still reported in kinds. A server log full of IP addresses is not usually private information about a person.

redact replaces each finding with a label that names its kind:

redact('Card 4111 1111 1111 1111 charged to ada@example.com')

'Card [CARD] charged to [EMAIL]'

Numbers are read in context

Many identifiers are long runs of digits. Order numbers, page ids and product codes are also long runs of digits. rahasya uses two tests to tell them apart.

First, an identifier with a check digit must pass the check. A card number must pass the Luhn check that every payment card number passes. rahasya does not report a sixteen-digit number that fails it.

Second, a number that follows a word that marks it as a reference, such as “Order”, “ISO” or “page”, is not an identifier.

[pii_report(t).has_pii for t in
 ('Order 4111 1111 1111 1112 shipped',   # fails Luhn
  'Conforms to EN 60601-1',              # a standard, not a ZIP
  'Card 4111 1111 1111 1111')]           # this one is a card

[False, False, True]

DESIGNATOR holds 76 reference words. They include standards bodies such as ISO, EN and RFC, and words such as invoice, page, commit and order. For a match that starts with a digit, rahasya reads the 48 characters before the match. It drops the match when one of these words ends just before it.

Regional identifiers

Nineteen of the 34 kinds are national identifiers. For most of them, a match must pass a check. A few kinds have no usable check, such as the UK National Insurance number. Others have only a weak check. rahasya finds these kinds only when a label such as “NINO” or “TFN” comes before the number.

Each check is also exported, for example aadhaar_ok, pesel_ok and nino_ok. gen_aadhaar takes a random.Random and returns a valid Aadhaar number for test data:

import random
a = gen_aadhaar(random.Random(0))
a, aadhaar_ok(a), pesel_ok('44051401359'), nino_ok('AB123456C')

('860487647593', True, True, True)

Names are opt-in

No pattern can tell a person’s name from a place, a product or a heading. rahasya looks for names only when you pass ner=True. It then finds only names that follow an honorific such as Mr, Mrs, Dr or Prof. Dr Charles Babbage is found and a bare Ada Lovelace is not.

rahasya looks for names only in the first 20,000 characters (NER_CHARS) of each scanned part of the text. A text of up to 200,000 characters is one part.

The scanned_ner field of the report says whether rahasya looked for names. It separates “no names found” from “names not looked for”.

(pii_report('Dr Charles Babbage signed it.').has_pii,
 pii_report('Dr Charles Babbage signed it.', ner=True).identifying,
 redact('Dr Charles Babbage signed it.', ner=True))

(False, {'person': 1}, 'Dr [PERSON] signed it.')

What is here

luhn, nhs_ok, aadhaar_ok, pesel_ok and the other check functions are also exported. The tables behind them are exported too, such as DNI_LETTERS and NRIC_LETTERS, with luhn_digits. A test can use them to build a number that passes. COMPILED holds every pattern in compiled form, which pii_spans uses.

Why there is no model

The author compared two machine-learning detectors with these patterns. Both had lower precision and lower recall. Both ran about 200 times slower. One detector was a DeBERTa-v3 classifier in ONNX format, which needs about a gigabyte of model weights. It did better only on names without an honorific. It found 5 of 8 test names, and rahasya found 2 of 8. The other detector was an encoder with 350 million parameters. It found no names. The author ran these comparisons outside this repository.

Development

The notebooks in nbs/ are the source, and the Python modules are generated from them.

pip install -e .
nbdev-prepare