depii

Detect and redact personally identifiable information (PII) in text, nested data, logs, files and DataFrames.

• No dependencies in the core. Pure Python, 3.10 and later.
• Validated, not just matched. Card numbers, IBANs, Aadhaar, NHS numbers and other identifiers are checked against their check digits, so a random run of digits is not flagged.
• Country packs for 32 countries and the EU: the Americas, Europe, Asia-Pacific, Africa and the Middle East (see What is detected), plus a plug-in mechanism for more.
• Context-aware. Values after a label (Email: jane at example dot com) or in a table column, person names, street addresses, the same person mentioned several ways, and details that identify someone only together (age, city, job). Built in: no model, no download.
• Secrets too. API keys, tokens, private keys and password=... assignments.
• Several ways in: strings, nested dicts and lists, a logging filter, a command line tool, and pandas.

>>> import depii
>>> depii.redact("Mail jane@example.com or call +1 415 555 0132")
'Mail [EMAIL] or call [PHONE]'

Read this first

No tool finds every piece of PII, and this one does not either. Name detection relies on titles, greetings, labels and built-in name lists, so an unusual name mentioned once with no context is missed. Free-form descriptions ("the tall man from flat 4") are not found. Treat the output as reduced-risk data, not as anonymous data.

Using this library does not by itself make a system compliant with GDPR, HIPAA, India's DPDP Act or any other law. Measure recall on a sample of your own data before you depend on it; Evaluation shows how.

Install

pip install depii

Optional extras: depii[pandas] for DataFrames, depii[ner] for names and places through spaCy, depii[phone] for stricter validation of international phone numbers, depii[all] for all three.

Usage

Text

>>> from depii import Redactor, Mask
>>> redactor = Redactor(policy={"CREDIT_CARD": Mask(keep_last=4)})
>>> result = redactor.redact("jane@example.com paid with 4111 1111 1111 1111")
>>> result.text
'[EMAIL] paid with **** **** **** 1111'
>>> result.stats
{'EMAIL': 1, 'CREDIT_CARD': 1}

Build a Redactor once and reuse it. It is immutable and safe to share between threads.

scan reports what was found without changing the text. The repr of a span leaves the matched value out, so spans can be logged:

>>> depii.scan("SSN 078-05-1120")
[Span(start=4, end=15, entity='US_SSN', score=1.0, detector='us_ssn')]

Context

Beyond the shape of a value, depii looks at what surrounds it. All of this is on by default and needs nothing beyond pip install depii.

Labels and tables. The label decides, whatever the value looks like. This covers prose (Name: ..., my email is ..., I live at ...), JSON and key-value pairs ("fullName": ..., user_email=...), and table columns in Markdown or tab-separated text:

>>> depii.redact("Email: jane at example dot com")
'Email: [EMAIL]'
>>> print(depii.redact("| Name | Phone |\n|---|---|\n| Ravi Kumar | 98765 43210 |"))
| Name | Phone |
|---|---|
| [PERSON] | [PHONE] |

Names and addresses. Person names are found from titles (Dr. R. Iyer, Smt. Meena), greetings and sign-offs (Dear Aanu,, Regards, then a name), and lists of common given names and family names from many cultures. Street addresses are found from a house number next to a street word in several languages (221B Baker Street, 42 MG Road, Hauptstraße 5, 12 rue de Rivoli), and PO boxes.

The same person, everywhere. A value found once is redacted wherever else it appears, and the parts of a detected name are redacted on their own and linked to the full name. With Tokenize or Hash, every mention gets the same output:

>>> from depii import Tokenize, Vault
>>> Redactor(default_operator=Tokenize(Vault())).redact(
...     "Jane Doe called. Ms Doe will return; Jane agreed."
... ).text
'[PERSON_1] called. Ms [PERSON_1] will return; [PERSON_1] agreed.'

vault.restore() puts back the full name for every linked mention, so "Ms Doe" restores as "Ms Jane Doe".

Identifying combinations. An age, a gender, a city, an occupation or an employer is not personal data on its own. Several of them about one person can be. When at least three kinds occur within a few sentences (postcodes, birth dates and addresses count too), all of them are redacted:

>>> depii.redact("The patient is a 42-year-old woman from Pune who works as a nurse.")
'The patient is a [AGE] [GENDER] from [CITY] who works as a [OCCUPATION].'
>>> depii.redact("A 42-year-old woman attended.")
'A 42-year-old woman attended.'

Change the number with quasi_threshold, or turn this off with quasi_threshold=None. contextual=False turns off all of the above and keeps only the pattern detectors, which is several times faster on short strings such as log lines.

Sending text to an AI

Pseudonymizer replaces personal data with readable labels before text goes to an AI, and puts the real values back into the answer. One per conversation keeps every person on the same label across messages:

>>> p = depii.Pseudonymizer()
>>> p.mask("Aryan asked Ravan to email aryan@example.com")
'Person A asked Person B to email Email A'
>>> p.unmask("Person A should wait for Person B's reply.")
"Aryan should wait for Ravan's reply."
>>> p.table
{'Person A': 'Aryan', 'Person B': 'Ravan', 'Email A': 'aryan@example.com'}

The table stays with you. Never send it to the AI: it holds the real values, and sending it would undo the masking. Only the masked text goes out.

With a chat API, mask the whole message list, add p.instructions to the system prompt so the model keeps the labels intact, and unmask the reply as it streams:

import anthropic

client = anthropic.Anthropic()
with client.messages.stream(
    model="claude-opus-5-5",
    max_tokens=64000,
    system=p.instructions,
    messages=p.mask_messages(history),  # text, content blocks, tool calls, tool results
) as stream:
    for piece in p.unmask_stream(stream.text_stream):
        print(piece, end="")

More:

• Structured output. p.mask_obj(data) and p.unmask_obj(reply) handle JSON and tool-call arguments.
• Unknown labels. p.unknown_labels(reply) lists labels the model invented ("Person D").
• Saving the table. p.save("table.json") and Pseudonymizer.load("table.json") keep it across a restart. The file holds the real values: store it where the original data may live.
• A complete example. examples/ai_round_trip.py runs offline, or against Claude with --claude.

What the AI does to a label is handled: PERSON A, Person-A, [Person A], **Person A** (the bold stays) and Person A's all map back, while "a person a day" is left alone. A value already in the table is masked wherever it appears later, in any case, and a lone "Doe" after "Jane Doe" gets Jane Doe's label. Labels that already occur in your text are never reused, and masking text twice changes nothing, so earlier replies can go back into the history as they are.

From the shell, with one table file per conversation:

depii mask ticket.txt --table t.json -o masked.txt
depii unmask reply.txt --table t.json

Limits:

• Masking is only as good as detection: a name nothing recognises is sent as it is.
• If the model writes "Person A and B", the lone "B" is not restored; the instructions ask it to write labels out in full.
• A partial mention is restored as the full name, so "Ms Person A" comes back as "Ms Jane Doe".
• Labels hide values, not what the text says about them: the model can still infer gender or role from context.

Options

Operators

Hash needs a secret key of at least 16 bytes and uses HMAC-SHA256. An unkeyed hash of a phone number or national ID can be reversed by trying every possible value, so it is not offered. Mask never reveals more than half of a value. A Vault holds the original values: protect it as you would the unredacted data.

Nested data

>>> depii.redact_obj({
...     "note": "call jane@example.com",
...     "user": {"firstName": "Jane", "password": "hunter2", "age": 41},
... })
{'note': 'call [EMAIL]', 'user': {'firstName': '[PERSON]', 'password': '[SECRET]', 'age': 41}}

Field names are used as hints. A field called password, ssn, userEmail or phone_number has its whole value redacted, even when the value does not look like PII. Pass key_hints=False to turn this off. The input object is not modified.

Logging

import logging
from depii import PIIFilter

handler = logging.StreamHandler()
handler.addFilter(PIIFilter())
logging.getLogger().addHandler(handler)

Attach the filter to handlers, not loggers: a filter on a logger does not see records that propagate up from child loggers. The message, its arguments, exception text, stack info and extra fields are all redacted. If redaction itself fails, the message is replaced with a fixed notice instead of being written unredacted.

Command line

depii redact notes.txt -o clean.txt
cat app.log | depii redact - > clean.log
depii redact events.jsonl --format jsonl -o clean.jsonl
depii redact people.csv --format csv --operator mask --keep-last 4 -o clean.csv
depii scan notes.txt
depii entities --locales us,in

scan prints file:line:column: ENTITY (score) for each finding and exits with status 1 when anything is found, so it can gate a commit or a CI job. It does not print the matched values unless you pass --show-values.

For --operator hash, the key is read from the environment variable DEPII_HASH_KEY, so that it does not appear in the process list or shell history.

Text input is streamed in chunks, so large files do not need to fit in memory. A file that is one single very long line is the exception.

pandas

from depii.frames import redact_dataframe

clean = redact_dataframe(df)  # every text column
clean = redact_dataframe(df, columns=["notes"])  # only these

Column names are used as hints in the same way as field names.

A language model for names

The built-in name detection needs a title, a greeting, a label or a name it knows. A language model also finds names it has never seen, from how they are used. The optional spaCy detector adds one, at the cost of a model download and slower runs:

pip install "depii[ner]"
python -m spacy download en_core_web_sm

from depii import Redactor
from depii.detectors.ner import SpacyDetector

redactor = Redactor(detectors=[SpacyDetector()])

It emits PERSON, LOCATION and ORGANIZATION alongside the built-in detectors. Its accuracy is that of the spaCy model you load.

Your own detectors

>>> from depii import PatternDetector, Redactor
>>> employee_id = PatternDetector(
...     "EMPLOYEE_ID",
...     r"\bE\d{5}\b",
...     score=0.4,
...     context=("employee", "staff"),
...     require_context=True,
... )
>>> Redactor(detectors=[employee_id]).redact("staff E12345 badge").text
'staff [EMPLOYEE_ID] badge'

A country pack is a module that exports DETECTORS. A separate package can register one under the depii.locales entry-point group, and it is then loaded like a built-in pack:

[project.entry-points."depii.locales"]
br = "depii_br:DETECTORS"

What is detected

Field-name hints can also produce PERSON, LOCATION, PASSPORT, BANK_ACCOUNT, NATIONAL_ID, TAX_ID and DRIVING_LICENCE.

Behaviour worth knowing

• Context words. Shapes that are too common to trust alone need a nearby word: a bare nine-digit number is an SSN only next to "SSN", a date is a date of birth only next to "born" or "DOB", and so on. Most national numbers written without their usual separators fall in this group: 52998224725 is a Brazilian CPF only next to "CPF", while 529.982.247-25 is one on its own. Context words include the local language ("PESEL", "fødselsnummer", "マイナンバー", "주민등록번호"). strict=True removes this requirement.
• Context reaches across sentences. The window is about 48 characters before a value and 24 after. A second date shortly after "born 3 March 1985" is redacted as well.
• URLs. A URL is redacted when it carries credentials or a query string. A plain link is left alone unless strict=True.
• Aadhaar. A bare twelve-digit number starting with 2 to 9 that passes the Verhoeff check is treated as an Aadhaar number. About one in twelve arbitrary twelve-digit numbers does. If your data has no Indian identifiers, leave the in pack out.
• Labels can be ambiguous. A number that is valid as two things gets one label: a Swedish personnummer can also be a valid Finnish or Danish one, a twelve-digit number can be an Aadhaar or a Japanese My Number. It is redacted either way.
• Only the date is checked for some numbers. Danish CPR numbers since 2007 and Korean RRNs since 2020 have no check digit, and Singapore M-series FINs use an unpublished one, so for these the shape and the date are all there is to go on.
• Unicode. Full-width digits, non-Latin digits, unusual dashes and zero-width characters inside a value are handled. Internationalised email addresses are not.
• Not handled: images, PDFs, audio, and PII written out in words with no label ("write to jane at example dot com"). With a label ("Email: jane at example dot com") it is found.
• Names in scripts without capitals. Chinese, Japanese, Arabic and similar names are found only with a label, or with a Devanagari title such as श्री.

Evaluation

python benchmarks/evaluate.py
python benchmarks/throughput.py

evaluate.py scores the detectors on a generated corpus: valid identifiers planted in sentences, plus numeric text that holds no PII. On that corpus every listed type currently reaches a recall and precision of 1.0. This shows that the detectors agree with the formats they claim to support. It does not predict recall on real text, which is messier than anything a generator produces. To measure on your own data, label a sample and pass it to synth.evaluate in tests/synth.py.

Throughput on one development laptop (Python 3.12), for orientation only:

Loading only the packs you need is the simplest way to go faster, and it also reduces false positives. For high-volume logs, contextual=False trades the context features for speed.

Development

python -m venv .venv
.venv/bin/pip install -e ".[dev]"     # .venv\Scripts\pip on Windows
ruff check . && ruff format --check .
mypy
pytest

The test suite includes property tests (Hypothesis) and a timing test that guards every pattern against catastrophic regex backtracking. Test data is generated: do not add real personal data to this repository.

Licence

Apache-2.0. See LICENSE.