piihush

Fast PII and secret masking for AI applications. piihush is a Rust extension for Python that removes sensitive data from your text before it reaches an external model provider, in RAG pipelines, AI agents, chatbots, logs and data pipelines.

import piihush

piihush.mask("Call 555-123-4567 or email jane@example.com")
# 'Call [PHONE_REDACTED] or email [EMAIL_REDACTED]'

• Fast: about 26 µs for a 1 KiB chunk, roughly 10 times faster than re.
• Safe on untrusted input: matching runs in linear time, so a crafted document or pattern can't trigger catastrophic backtracking.
• Context-aware: values next to a label such as Password: or Date of Birth: are masked while the label stays readable, and cards, IBANs and IPv6 addresses are validated rather than just pattern-matched.
• Built for pipelines: it releases the GIL, and works with threads, fork() and pickling.
• Local: no network calls, no models and no runtime dependencies in the default install. Better name detection is an optional add-on.

piihush reduces what reaches a model provider, but it does not catch everything. Read Limitations and Accuracy before you rely on it.

Contents

• Where to use it
• Install
• Quick start
• Choosing a mode
• What gets detected
• Configuring a Masker
• Using piihush in a RAG pipeline
• Using piihush with AI agents
• Chat with any LLM privately
• Concurrency
• Better name detection
• API reference
• Troubleshooting
• Limitations
• Accuracy
• Performance
• How it works
• Contributing
• Sponsor
• License

Where to use it

Install

pip install piihush

Optional add-ons:

piihush[llm] also adds LLM detection, for catching more PII at a much higher cost per record.

Wheels cover Python 3.11+ on Linux (glibc and musl), macOS and Windows, on x86-64 and ARM64. You don't need a Rust toolchain.

To build the latest code from GitHub instead, install Rust first:

pip install git+https://github.com/aziaya/piihush

Quick start

For a one-off call, use the module-level functions. They use every built-in detector with the default settings.

import piihush

piihush.mask("Password: hunter2 and SSN 123-45-6789")
# 'Password: [PASSWORD_REDACTED] and SSN [SSN_REDACTED]'

piihush.mask_batch(["Mail a@b.io", "No PII here"])
# ['Mail [EMAIL_REDACTED]', 'No PII here']

piihush.detect("Mail jane@example.com")
# [Match(entity="EMAIL", start=5, end=21, text=<16 chars>)]

In an application, create one Masker and reuse it. Creating it compiles the patterns, which takes milliseconds. Using it is cheap and thread-safe.

from piihush import Masker

masker = Masker()
masker.mask("Reach me at +44 20 7946 0958")
# 'Reach me at [PHONE_REDACTED]'

Choosing a mode

Start with Masker unless your text contains names you must protect. You can also use both: run Masker on everything, and NerMasker only on documents that need it.

What gets detected

Run piihush.ENTITIES to list these names in code.

When a label is required, only the value is masked, so the text still makes sense:

piihush.mask("Date of Birth: 17/10/1944")
# 'Date of Birth: [DATE_OF_BIRTH_REDACTED]'

Configuring a Masker

Choose entities

Pass the entities you want. The default is all of them.

masker = Masker(["EMAIL", "API_KEY"])
masker.mask("a@b.io called 555-123-4567")
# '[EMAIL_REDACTED] called 555-123-4567'

To turn off one or two, filter the full list:

masker = Masker([e for e in piihush.ENTITIES if e != "IP_ADDRESS"])
masker.mask("from 10.0.0.1, a@b.io")
# 'from 10.0.0.1, [EMAIL_REDACTED]'

Add your own patterns

custom_patterns maps an entity name to a regular expression. Names use uppercase letters, digits and underscores.

masker = Masker(custom_patterns={"EMPLOYEE_ID": r"EMP-\d{6}"})
masker.mask("Badge EMP-123456")
# 'Badge [EMPLOYEE_ID_REDACTED]'

Patterns use Rust regex syntax. It has no lookaround or backreferences, which is what keeps matching linear-time.

Mask specific values

deny_list maps an entity name to exact values, such as customer names or specific email addresses. Matching ignores ASCII case and respects word boundaries, so Ann doesn't match inside Annual.

masker = Masker(deny_list={"CUSTOMER": ["Acme Corp", "Ann"]})
masker.mask("ACME CORP and Ann sent the Annual report")
# '[CUSTOMER_REDACTED] and [CUSTOMER_REDACTED] sent the Annual report'

Keep specific values

allow_list keeps values that a detector would otherwise mask, such as your own support address. A value must match the whole detected text. Case is ignored.

masker = Masker(allow_list=["support@ourcompany.example"])
masker.mask("Write to support@ourcompany.example, not a@b.io")
# 'Write to support@ourcompany.example, not [EMAIL_REDACTED]'

Change the placeholder

template sets the replacement text. {entity} becomes the entity name.

Masker(template="<{entity}>").mask("a@b.io")
# '<EMAIL>'

Masker(template="***").mask("a@b.io")
# '***'

Using piihush in a RAG pipeline

Mask at each point where text leaves your control.

Mask before you chunk

A value split across two chunks can't be detected, so mask the whole document first.

document = "Contact jane@example.com about the refund."

safe_document = masker.mask(document)
chunks = [safe_document[i : i + 500] for i in range(0, len(safe_document), 400)]

If you already have chunks, mask_batch is the fastest option. It keeps their order.

chunks = ["First chunk, a@b.io", "Second chunk"]
safe_chunks = masker.mask_batch(chunks)

Mask the user's question too

Text retrieved from your own store is only part of the prompt. Mask the question as well, and do it before any call to an external service, including embedding APIs.

question = "Why was the card 4111 1111 1111 1111 declined?"
safe_question = masker.mask(question)

Use it with LangChain

piihush works with plain strings, so it needs no special integration:

from langchain_core.documents import Document

documents = [Document(page_content="Mail a@b.io", metadata={"source": "crm"})]
safe_documents = [
    Document(page_content=text, metadata=doc.metadata)
    for doc, text in zip(documents, masker.mask_batch(d.page_content for d in documents))
]

Metadata isn't masked in this example. If yours contains PII, mask it too.

Using piihush with AI agents

An agent passes far more data to its model than the user typed: tool results, database rows, search hits, files and memory. Mask that data before it enters the agent's context.

def lookup_customer(customer_id):
    return "Customer: Jane Doe, jane@example.com, Card 4111 1111 1111 1111"

def safe_tool(tool, *args):
    return masker.mask(tool(*args))

safe_tool(lookup_customer, 42)
# 'Customer: [PERSON_REDACTED], [EMAIL_REDACTED], Card [CREDIT_CARD_REDACTED]'

When the agent must act on a real value, such as sending an email, use a PrivacyGuard. The model works with [EMAIL_1], and your code restores the real value at the moment a tool runs:

from piihush.llm import PrivacyGuard

guard = PrivacyGuard()
context = guard.mask("Reply to jane@example.com")  # goes to the model
tool_argument = "[EMAIL_1]"                        # the model's tool call
guard.restore(tool_argument)
# 'jane@example.com'

Treat tool arguments from a model as untrusted. A malicious document can try to steer an agent into using a restored value somewhere it shouldn't, so limit what each tool can do.

Chat with any LLM privately

PrivateChat masks your message, sends it to a provider, and restores the real values in the reply. The provider only ever sees tokens such as [EMAIL_1].

pip install "piihush[llm]"

from piihush.llm import PrivateChat

chat = PrivateChat("anthropic", model="your-model-name")  # reads ANTHROPIC_API_KEY
reply = chat.send("Write a reminder to email jane@example.com")
# The provider receives: Write a reminder to email [EMAIL_1]
# You receive: a reply containing jane@example.com

for text in chat.stream("Now make it shorter"):
    print(text, end="")

Any other OpenAI-compatible service works through base_url:

chat = PrivateChat(base_url="http://localhost:1234/v1", model="my-model")

• Options: extra arguments go to the provider, as in chat.send(prompt, temperature=0, max_tokens=200).
• Better detection: pass a guard, as in PrivateChat(..., guard=PrivacyGuard(NerMasker())).
• Your own client: use guard.mask and guard.restore around any SDK, and guard.restore_stream for streaming.
• Keep guards separate: use one per conversation or user. A guard remembers what it masked, so sharing one could reveal one user's values to another.

Tested with real models: llama.cpp and Ollama Cloud. The others use each provider's documented address and have only run against a simulated server.

Concurrency

Threads

A Masker is immutable, so one instance can be shared across threads. Scanning releases the GIL, so threads run in parallel.

asyncio

There is no async API, because masking is CPU work with nothing to wait for.

• Short text: call mask directly. It takes microseconds, and handing off to a thread would cost more than it saves.
• Large documents or batches: use asyncio.to_thread, so the event loop stays responsive.

import asyncio

async def handle(chunks):
    return await asyncio.to_thread(masker.mask_batch, chunks)

asyncio.run(handle(["a@b.io"]))
# ['[EMAIL_REDACTED]']

Batches

mask_batch uses every CPU core once a batch reaches 256 KiB, and a single thread below that. Its threads exist only during the call, so there is no pool to tune.

Multiple processes

A Masker can be pickled for multiprocessing, and mask_batch is safe after fork(), so it works with Celery and gunicorn workers.

Better name detection

The default Masker only finds a name or city when something points to it, such as a title or label. To find them in free text, install the optional model:

pip install "piihush[ner]"

from piihush.ner import NerMasker

ner_masker = NerMasker()  # takes the same options as Masker
ner_masker.mask("Neslihan Kaya wrote from Berlin")
# '[PERSON_REDACTED] wrote from [ADDRESS_REDACTED]'

NerMasker runs all the Masker rules, then adds what the model finds. Once it finds a name, it masks that name everywhere in the same text.

• Model: the first run downloads dslim/bert-base-NER (MIT, 433 MB), pinned to an exact revision and cached by Hugging Face.
• Privacy: after that download, the model runs on your machine and your text never leaves it.
• Offline use: once the model is cached, set HF_HUB_OFFLINE=1.
• Speed: the model runs on NumPy, about 160 ms per short record, so it installs everywhere. With pip install "piihush[ner-fast]" it uses onnxruntime instead, at about 55 ms with identical results. onnxruntime isn't available on every platform.
• Memory: about 950 MB.
• Other models: pass any callable that returns (entity, start, end) tuples, where the offsets are Python string indices:

def my_recognizer(text):
    start = text.find("Project Falcon")
    return [("PROJECT", start, start + 14)] if start >= 0 else []

custom = NerMasker(recognizer=my_recognizer)
custom.mask("Status of Project Falcon")
# 'Status of [PROJECT_REDACTED]'

LLM detection

LlmRecognizer asks an LLM to find PII that the rules miss. Use it with NerMasker, which keeps all the rules running alongside it. The model can be local or hosted.

from piihush.llm import LlmRecognizer
from piihush.ner import BertRecognizer, NerMasker, combine

llm = LlmRecognizer("ollama", model="your-model-name")

rules_and_llm = NerMasker(recognizer=llm)
strongest = NerMasker(recognizer=combine(BertRecognizer(), llm))

Found values are labelled PII, so they become [PII_REDACTED].

Before you use it:

• The LLM sees your raw text. Use a model or service you trust with that data.
• A document can talk it out of reporting. In testing, one "ignore your instructions" sentence stopped the LLM reporting anything. The rules and NER model aren't affected, so you fall back to their results. Don't use an LLM on its own.
• It costs far more. Expect seconds per record, plus any provider fees.
• It only adds masking. A value counts only if it really appears in the text. If a request fails, it raises LLMError instead of letting text through unchecked.

API reference

Module functions

These use a shared Masker with default settings, created on first use.

Masker

Masker(entities=None, *, custom_patterns=None, deny_list=None, allow_list=None, template=None)

A bad configuration raises ValueError: an unknown entity, an invalid pattern, an invalid entity name, or an empty deny-list value.

Passing something other than str raises TypeError, and so does passing a single str to mask_batch.

Match

repr() hides the value so it doesn't end up in your logs, but match.text still holds it. Take care not to log that attribute.

piihush.ner.NerMasker

Takes the same arguments as Masker, plus recognizer, and has the same mask, mask_batch and detect methods. Unlike Masker, it can't be pickled, and mask_batch handles one text at a time. Without the extra packages, it raises ImportError with the install command.

Troubleshooting

A value wasn't masked. Check whether it matches a format in What gets detected. Add known values to deny_list, add a format to custom_patterns, or use NerMasker for names and places.

Too much was masked. Add the value to allow_list, or leave out the entity that matched it. Use detect to see which one that was:

[m.entity for m in piihush.detect("Hi Team, mail a@b.io")]
# ['PERSON', 'EMAIL']

A card number wasn't masked. Card numbers must pass the Luhn checksum, so made-up test numbers are often skipped. Real card numbers pass. Use custom_patterns if you want every 16-digit number masked.

ValueError: invalid pattern. Rust regex has no lookaround or backreferences.

ImportError from NerMasker. Run pip install "piihush[ner]".

Limitations

piihush is one layer of a privacy strategy. It does not make text anonymous, and it doesn't by itself satisfy GDPR, HIPAA or any other regulation.

• Nothing here catches everything. In testing, about 3 in 10 labelled values got past the best mode. Review the output before relying on it for sensitive data.
• The default mode finds patterns, not meaning. A name or city with no title, greeting or label around it isn't detected, and neither are medical details. Use deny_list for known values, or the optional model for free text.
• Unlabelled numbers are mostly left alone. A bare 9-digit SSN or account number looks like any other number. Use custom_patterns if you need them masked.
• Harmless text is sometimes masked. Context rules can over-match. In testing, 1.8% of ordinary words were lost with Masker and 3.4% with NerMasker.
• Split values can't be detected. Mask documents before you chunk them. A private key whose BEGIN line lands in an earlier chunk is the same case.
• Obfuscated text isn't detected, such as jane at example dot com or full-width digits. Apply unicodedata.normalize("NFKC", text) first if your input may contain those.
• mask is one-way. Original values can't be restored from its output. Use PrivacyGuard when you need them back.
• English is the tested language. Other languages haven't been measured, and the model and most labels are English.

Accuracy

Tested on 3,000 English records of the independent ai4privacy/pii-masking-400k dataset, kept apart from the data used to build the rules. Each tool ran with its defaults on the same machine. "Masked" means the exact value no longer appears in the output.

Read these results with care:

• Presidio leads on many number types. It masks many more unlabelled numbers, and loses more ordinary text doing so.
• Random card numbers count against piihush. The dataset's card numbers are random, so most fail the checksum piihush uses to avoid false positives.
• The data is not everything. It has no API keys, so it doesn't reflect piihush's secret detection, and pii-veil is built mainly for Polish data.
• The test data was used more than once. NerMasker was scored on it twice, because two leaks were fixed after the first run. Both were found by hand, on other text, and its score rose by 0.1 points.

To reproduce these results, see the instructions in benches/accuracy.py.

LLM detection results

Measured with Qwen 2.5 1.5B running locally on 200 unused records from the same dataset (benches/llm_detection.py). Other models will score differently.

• Best coverage: NerMasker + LLM masked the most, at about 40 times the cost of NerMasker alone, and it replaced more ordinary text.
• Prompt injection: one injected sentence took the LLM from finding 40 values in 50 records to finding 0.
• Small sample: with 200 records, treat small differences as noise.

Performance

Measured with benches/bench.py on a 2016 Intel Core i7-6920HQ (8 threads) with all detectors on, using generated text with a PII value about every 40 words. The re column is a smaller, roughly equivalent set of 6 patterns. Your results will vary, so run the script on your own hardware and data.

• Batch throughput: 145 MB/s for mask_batch, compared with 29 MB/s for a single-threaded loop.
• Worst case: the slowest hostile 1 MiB input tested took 841 ms.

Where a pattern has a literal to search for, the Rust regex, aho-corasick and memchr crates use SIMD (AVX2, SSSE3 or NEON), selected at runtime. Detectors also skip the regex entirely when a required byte such as @ is absent.

How it works

1. Each detector scans the text. Candidates that fail validation are rejected, and the scan resumes inside the rejected candidate so it can't hide a real match.
2. Overlapping matches merge into one span, so no fragment of either survives.
3. The spans are replaced with placeholders. Text with nothing to mask is returned as the same object, without copying.

Contributing

Bug reports and pull requests are welcome. See CONTRIBUTING.md, and report security issues as described in SECURITY.md. Use only made-up data in tests, issues and pull requests.

Sponsor

If piihush is useful to you, you can support its development through GitHub Sponsors.

License

MIT