piihush

Fast PII and secret masking for RAG and LLM pipelines. piihush is a Rust extension for Python that removes sensitive data from your text before it reaches an external model provider.

import piihush

piihush.mask("Call 555-123-4567 or email jane@example.com")
# 'Call [PHONE_REDACTED] or email [EMAIL_REDACTED]'

• Fast: about 26 µs for a 1 KiB chunk, roughly 10 times faster than re.
• Safe on untrusted input: matching runs in linear time, so a crafted document or pattern can't trigger catastrophic backtracking.
• Context-aware: values next to a label such as Password: or Date of Birth: are masked while the label stays readable, and cards, IBANs and IPv6 addresses are validated rather than just pattern-matched.
• Built for pipelines: it releases the GIL, and works with threads, fork() and pickling.
• Local: no network calls, no models and no runtime dependencies in the default install. Better name detection is an optional add-on.

piihush reduces what reaches a model provider, but it does not catch everything. Read Limitations and Accuracy before you rely on it.

Contents

• Install
• Quick start
• Choosing a mode
• What gets detected
• Configuring a Masker
• Using piihush in a RAG pipeline
• Concurrency
• Better name detection
• API reference
• Troubleshooting
• Limitations
• Accuracy
• Performance
• How it works
• Contributing
• Sponsor
• License

Install

pip install piihush

Wheels cover Python 3.11+ on Linux (glibc and musl), macOS and Windows, on x86-64 and ARM64. You don't need a Rust toolchain.

To build the latest code from GitHub instead, install Rust first:

pip install git+https://github.com/aziaya/piihush

Quick start

For a one-off call, use the module-level functions. They use every built-in detector with the default settings.

import piihush

piihush.mask("Password: hunter2 and SSN 123-45-6789")
# 'Password: [PASSWORD_REDACTED] and SSN [SSN_REDACTED]'

piihush.mask_batch(["Mail a@b.io", "No PII here"])
# ['Mail [EMAIL_REDACTED]', 'No PII here']

piihush.detect("Mail jane@example.com")
# [Match(entity="EMAIL", start=5, end=21, text=<16 chars>)]

In an application, create one Masker and reuse it. Creating it compiles the patterns, which takes milliseconds. Using it is cheap and thread-safe.

from piihush import Masker

masker = Masker()
masker.mask("Reach me at +44 20 7946 0958")
# 'Reach me at [PHONE_REDACTED]'

Choosing a mode

Start with Masker unless your text contains names you must protect. You can also use both: run Masker on everything, and NerMasker only on documents that need it.

What gets detected

Run piihush.ENTITIES to list these names in code.

When a label is required, only the value is masked, so the text still makes sense:

piihush.mask("Date of Birth: 17/10/1944")
# 'Date of Birth: [DATE_OF_BIRTH_REDACTED]'

Configuring a Masker

Choose entities

Pass the entities you want. The default is all of them.

masker = Masker(["EMAIL", "API_KEY"])
masker.mask("a@b.io called 555-123-4567")
# '[EMAIL_REDACTED] called 555-123-4567'

To turn off one or two, filter the full list:

masker = Masker([e for e in piihush.ENTITIES if e != "IP_ADDRESS"])
masker.mask("from 10.0.0.1, a@b.io")
# 'from 10.0.0.1, [EMAIL_REDACTED]'

Add your own patterns

custom_patterns maps an entity name to a regular expression. Names use uppercase letters, digits and underscores.

masker = Masker(custom_patterns={"EMPLOYEE_ID": r"EMP-\d{6}"})
masker.mask("Badge EMP-123456")
# 'Badge [EMPLOYEE_ID_REDACTED]'

Patterns use Rust regex syntax. It has no lookaround or backreferences, which is what keeps matching linear-time.

Mask specific values

deny_list maps an entity name to exact values, such as customer names or specific email addresses. Matching ignores ASCII case and respects word boundaries, so Ann doesn't match inside Annual.

masker = Masker(deny_list={"CUSTOMER": ["Acme Corp", "Ann"]})
masker.mask("ACME CORP and Ann sent the Annual report")
# '[CUSTOMER_REDACTED] and [CUSTOMER_REDACTED] sent the Annual report'

Keep specific values

allow_list keeps values that a detector would otherwise mask, such as your own support address. A value must match the whole detected text. Case is ignored.

masker = Masker(allow_list=["support@ourcompany.example"])
masker.mask("Write to support@ourcompany.example, not a@b.io")
# 'Write to support@ourcompany.example, not [EMAIL_REDACTED]'

Change the placeholder

template sets the replacement text. {entity} becomes the entity name.

Masker(template="<{entity}>").mask("a@b.io")
# '<EMAIL>'

Masker(template="***").mask("a@b.io")
# '***'

Using piihush in a RAG pipeline

Mask at each point where text leaves your control.

Mask before you chunk

A value split across two chunks can't be detected, so mask the whole document first.

document = "Contact jane@example.com about the refund."

safe_document = masker.mask(document)
chunks = [safe_document[i : i + 500] for i in range(0, len(safe_document), 400)]

If you already have chunks, mask_batch is the fastest option. It keeps their order.

chunks = ["First chunk, a@b.io", "Second chunk"]
safe_chunks = masker.mask_batch(chunks)

Mask the user's question too

Text retrieved from your own store is only part of the prompt. Mask the question as well, and do it before any call to an external service, including embedding APIs.

question = "Why was the card 4111 1111 1111 1111 declined?"
safe_question = masker.mask(question)

Use it with LangChain

piihush works with plain strings, so it needs no special integration:

from langchain_core.documents import Document

documents = [Document(page_content="Mail a@b.io", metadata={"source": "crm"})]
safe_documents = [
    Document(page_content=text, metadata=doc.metadata)
    for doc, text in zip(documents, masker.mask_batch(d.page_content for d in documents))
]

Metadata isn't masked in this example. If yours contains PII, mask it too.

Concurrency

Threads

A Masker is immutable, so one instance can be shared across threads. Scanning releases the GIL, so threads run in parallel.

asyncio

There is no async API, because masking is CPU work with nothing to wait for.

• Short text: call mask directly. It takes microseconds, and handing off to a thread would cost more than it saves.
• Large documents or batches: use asyncio.to_thread, so the event loop stays responsive.

import asyncio

async def handle(chunks):
    return await asyncio.to_thread(masker.mask_batch, chunks)

asyncio.run(handle(["a@b.io"]))
# ['[EMAIL_REDACTED]']

Batches

mask_batch uses every CPU core once a batch reaches 256 KiB, and a single thread below that. Its threads exist only during the call, so there is no pool to tune.

Multiple processes

A Masker can be pickled for multiprocessing, and mask_batch is safe after fork(), so it works with Celery and gunicorn workers.

Better name detection

The default Masker only finds a name or city when something points to it, such as a title or label. To find them in free text, install the optional model:

pip install "piihush[ner]"

from piihush.ner import NerMasker

ner_masker = NerMasker()  # takes the same options as Masker
ner_masker.mask("Neslihan Kaya wrote from Berlin")
# '[PERSON_REDACTED] wrote from [ADDRESS_REDACTED]'

NerMasker runs all the Masker rules, then adds what the model finds. Once it finds a name, it masks that name everywhere in the same text.

• Model: the first run downloads dslim/bert-base-NER (MIT, 411 MB), pinned to an exact revision and cached by Hugging Face.
• Privacy: after that download, the model runs on your machine and your text never leaves it.
• Offline use: once the model is cached, set HF_HUB_OFFLINE=1.
• Cost: about 110 ms per short record and about 850 MB of memory.
• Other models: pass any callable that returns (entity, start, end) tuples, where the offsets are Python string indices:

def my_recognizer(text):
    start = text.find("Project Falcon")
    return [("PROJECT", start, start + 14)] if start >= 0 else []

custom = NerMasker(recognizer=my_recognizer)
custom.mask("Status of Project Falcon")
# 'Status of [PROJECT_REDACTED]'

API reference

Module functions

These use a shared Masker with default settings, created on first use.

Masker

Masker(entities=None, *, custom_patterns=None, deny_list=None, allow_list=None, template=None)

A bad configuration raises ValueError: an unknown entity, an invalid pattern, an invalid entity name, or an empty deny-list value.

Passing something other than str raises TypeError, and so does passing a single str to mask_batch.

Match

repr() hides the value so it doesn't end up in your logs, but match.text still holds it. Take care not to log that attribute.

piihush.ner.NerMasker

Takes the same arguments as Masker, plus recognizer, and has the same mask, mask_batch and detect methods. Unlike Masker, it can't be pickled, and mask_batch handles one text at a time. Without the extra packages, it raises ImportError with the install command.

Troubleshooting

A value wasn't masked. Check whether it matches a format in What gets detected. Add known values to deny_list, add a format to custom_patterns, or use NerMasker for names and places.

Too much was masked. Add the value to allow_list, or leave out the entity that matched it. Use detect to see which one that was:

[m.entity for m in piihush.detect("Hi Team, mail a@b.io")]
# ['PERSON', 'EMAIL']

A card number wasn't masked. Card numbers must pass the Luhn checksum, so made-up test numbers are often skipped. Real card numbers pass. Use custom_patterns if you want every 16-digit number masked.

ValueError: invalid pattern. Rust regex has no lookaround or backreferences.

ImportError from NerMasker. Run pip install "piihush[ner]".

Limitations

piihush is one layer of a privacy strategy. It does not make text anonymous, and it doesn't by itself satisfy GDPR, HIPAA or any other regulation.

• Nothing here catches everything. In testing, about 3 in 10 labelled values got past the best mode. Review the output before relying on it for sensitive data.
• The default mode finds patterns, not meaning. A name or city with no title, greeting or label around it isn't detected, and neither are medical details. Use deny_list for known values, or the optional model for free text.
• Unlabelled numbers are mostly left alone. A bare 9-digit SSN or account number looks like any other number. Use custom_patterns if you need them masked.
• Harmless text is sometimes masked. Context rules can over-match. In testing, 1.8% of ordinary words were lost with Masker and 3.4% with NerMasker.
• Split values can't be detected. Mask documents before you chunk them. A private key whose BEGIN line lands in an earlier chunk is the same case.
• Obfuscated text isn't detected, such as jane at example dot com or full-width digits. Apply unicodedata.normalize("NFKC", text) first if your input may contain those.
• Masking is one-way. Original values can't be restored from the output.
• English is the tested language. Other languages haven't been measured, and the model and most labels are English.

Accuracy

Tested on 3,000 English records of the independent ai4privacy/pii-masking-400k dataset, kept apart from the data used to build the rules. Each tool ran with its defaults on the same machine. "Masked" means the exact value no longer appears in the output.

Read these results with care:

• Presidio leads on many number types. It masks many more unlabelled numbers, and loses more ordinary text doing so.
• Random card numbers count against piihush. The dataset's card numbers are random, so most fail the checksum piihush uses to avoid false positives.
• The data is not everything. It has no API keys, so it doesn't reflect piihush's secret detection, and pii-veil is built mainly for Polish data.
• The test data was used more than once. NerMasker was scored on it twice, because two leaks were fixed after the first run. Both were found by hand, on other text, and its score rose by 0.1 points.

To reproduce these results, see the instructions in benches/accuracy.py.

Performance

Measured with benches/bench.py on a 2016 Intel Core i7-6920HQ (8 threads) with all detectors on, using generated text with a PII value about every 40 words. The re column is a smaller, roughly equivalent set of 6 patterns. Your results will vary, so run the script on your own hardware and data.

• Batch throughput: 145 MB/s for mask_batch, compared with 29 MB/s for a single-threaded loop.
• Worst case: the slowest hostile 1 MiB input tested took 841 ms.

Where a pattern has a literal to search for, the Rust regex, aho-corasick and memchr crates use SIMD (AVX2, SSSE3 or NEON), selected at runtime. Detectors also skip the regex entirely when a required byte such as @ is absent.

How it works

1. Each detector scans the text. Candidates that fail validation are rejected, and the scan resumes inside the rejected candidate so it can't hide a real match.
2. Overlapping matches merge into one span, so no fragment of either survives.
3. The spans are replaced with placeholders. Text with nothing to mask is returned as the same object, without copying.

Contributing

Bug reports and pull requests are welcome. See CONTRIBUTING.md, and report security issues as described in SECURITY.md. Use only made-up data in tests, issues and pull requests.

Sponsor

If piihush is useful to you, you can support its development through GitHub Sponsors.

License

MIT