noirdoc

German-first PII redaction and pseudonymization for documents. Local by default. Reversible when you need it.

Noirdoc redacts names, addresses, phone numbers, IBANs, Steuer-IDs, SVNRs, and the rest — from PDFs, DOCX, XLSX, and plain text — without sending anything to a third party. Under the hood it's a rules-based Presidio pipeline by default, and an ensemble (Presidio + GLiNER + Flair) when the [full] extra is installed. It's built for real-world German documents and mixed DE/EN text — the kind of stuff Mittelstand actually runs through an LLM.

Status: alpha (0.1.x). API will change before 1.0. Pin the minor version.

Prerequisites

• Python 3.12 or 3.13
• ~1 GB free disk if you install the [full] extra (spaCy + Flair + GLiNER weights)
• Optional: a Redis instance if you want shared mapping storage across workers ([redis] extra)

Install

# Baseline — Presidio + all file extractors + reversible mapper.
pip install noirdoc

# Full ensemble (adds GLiNER + Flair, large ML weights). Recommended for real work.
pip install noirdoc[full]
noirdoc models pull

# Optional distributed mapper backend.
pip install noirdoc[redis]

For anything beyond toy examples, use noirdoc[full] — the ensemble catches what the baseline misses, especially on German lowercase text.

Quickstart

# One-shot redact (ephemeral mapping, discarded on exit).
# PDFs come back as masked plain text, so the output is a .txt file.
noirdoc redact vertrag.pdf -o vertrag-clean.txt

# Persistent namespace — placeholders stay consistent across files and sessions.
noirdoc redact --namespace mandant-mueller brief.docx -o brief-clean.docx
noirdoc reveal --namespace mandant-mueller brief-clean.docx -o brief-revealed.docx
noirdoc lookup --namespace mandant-mueller "<<PERSON_3>>"

from noirdoc import Redactor

r = Redactor(namespace="mandant-mueller")
r.redact_file("vertrag.pdf", output="vertrag-clean.txt")  # PDFs fall back to plain text
r.redact_file("brief.docx", output="brief-clean.docx")
r.reveal_text(llm_response)  # un-redact the model's reply

Input:

Anna Müller, geboren am 12.03.1981 in München, erreichbar unter 0171-2345678, Steuer-ID 12 345 678 901, IBAN DE89 3704 0044 0532 0130 00.

Output:

<<PERSON_1>>, geboren am <<DATE_TIME_1>> in <<LOCATION_1>>, erreichbar unter <<PHONE_NUMBER_1>>, Steuer-ID <<DE_STEUER_ID_1>>, IBAN <<IBAN_CODE_1>>.

Commands

Run noirdoc <cmd> --help for the full flag list on any subcommand.

Before you start

A few honest caveats before you ship this into a pipeline:

• Best results need [full]. On first use (or via noirdoc models pull) the full extra downloads roughly 560 MB of weights: spaCy de_core_news_lg, Flair ner-german-large, and a GLiNER multilingual model. Budget disk and bandwidth.

• Not every format survives redaction with formatting intact. PDFs and images always come back as masked plain text, and DOCX/XLSX fall back to plain text when reconstruction fails. You can tell it happened by three signals: the CLI writes *_redacted.txt (an explicit non-.txt -o target is redirected to <stem>.txt with a warning on stderr), and the SDK's RedactionResult has mime_type="text/plain" and reconstructed=False, with reason saying why. A .docx/.xlsx path never silently receives UTF-8 text.

• PDF reveal is not supported yet. Round-tripping placeholders back into a PDF is a hard problem (position drift, font metrics, image-based redactions). PDFs redact cleanly; reveal is pass-through. DOCX, XLSX, and plain text round-trip.

• XLSX output is re-serialized by openpyxl. Redaction covers cells, docProps (core + custom properties), cell comments, headers/footers, pivot caches, chart string caches and titles, hyperlink tooltips/display and mailto: targets (the placeholder is percent-encoded in the target — mailto:%3C%3CEMAIL_1%3E%3E — because </> are illegal URI characters; reveal decodes it back), AutoFilter criteria, conditional-formatting text criteria and string-literal formulas, data-validation inline lists and prompts, and pivot-table captions and label filters — and reveals them all. Threaded comments, xl/persons and app.xml Manager/Company are dropped on write (counted, not reversible); shapes, slicers, macros and other parts openpyxl cannot model do not survive the round-trip. Row 1 is treated as a header only where its own text matches the header keyword map or the column below it is unclassified; otherwise it is redacted like any data row, so a sheet with no header row does not leak its first record. Not covered yet: non-mailto: hyperlink URL targets, defined names and sheet titles (both would require rewriting every formula that references them).

• DOCX redaction covers the text surfaces and the package metadata. One shared walker drives extraction, redaction and reveal over the XML, covering: body text, content controls (w:sdt), tables nested at any depth, text boxes and shapes (both mc:AlternateContent branches on rewrite), tracked changes (inserted text is redacted like any other; deleted text is shown to the detector, then stripped from the output entirely — it is not reversible), headers/footers, review comments, and footnotes/endnotes. Inline pictures, line breaks and tab stops survive the rewrite, and a placeholder inside a hyperlink stays inside the hyperlink. A second walker pseudonymizes the package parts — docProps (core creator/lastModifiedBy/title/…, app.xml Manager/Company/HyperlinkBase/TitlesOfParts, custom properties), comment authors/initials, word/people.xml identities and the mailto: targets of hyperlink relationships (percent-encoded in the target — mailto:%3C%3CEMAIL_1%3E%3E — because </> are illegal URI characters) — all reversibly. The docProps/thumbnail.jpeg preview (a rendered image of the original page) and customXml/ data islands cannot be pseudonymized and are dropped on write, reported separately from the entity count because carrying them says nothing about whether a document holds PII (most Word templates ship both). Not covered yet: non-mailto: hyperlink targets, field instructions (w:instrText — mail-merge field codes), tracked-change author attributes, image alt text, and embedded objects.

• Alpha API. Classes and CLI flags may change between 0.1.x and 0.2.x. Pin accordingly.

• Detector quality depends on the upstream models. Presidio + Flair + GLiNER do the heavy lifting. Noirdoc adds German-specific recognizers on top, but it does not train models.

German-first

Noirdoc defaults to German (language="de") with fallback to ["de", "en"] for mixed documents. What that actually means:

• Custom recognizers in src/noirdoc/detection/presidio_detector.py:
  • GermanPhoneRecognizer — German phone formats (0171-..., +49...)
  • GermanSVNRRecognizer — Sozialversicherungsnummer with checksum
  • GermanSteuerIDRecognizer — 11-digit Steuer-ID with checksum
  • InvertedNameRecognizer — registered for both de and en to catch "Nachname, Vorname" patterns
• Flair ner-german-large (XLM-R, F1 92.3 % on CoNLL-03 DE) handles lowercase German text — the case where spaCy tends to drop names.
• GLiNER multilingual catches entity types the others miss.
• German-style lowercase financial terms, German IBANs, German date formats, and German address patterns are covered in the test suite (tests/test_presidio_detector.py).

If you're working with German legal, medical, HR, or financial documents, this is what the defaults are tuned for.

Supported formats

PDF reveal is an open contribution target — see CONTRIBUTING.md.

Advanced: shared mapping storage

The [redis] extra ships a RedisMappingBackend that plugs into the lower-level MappingStore — the same primitive Noirdoc Cloud uses for request-scoped, encrypted, TTL-bounded mapping persistence across workers. It is not wired into Redactor(namespace=...), which persists to the local filesystem under ~/.noirdoc/namespaces/. Use MappingStore when you have multiple workers that need to share pseudonym mappings for the same request, or when you want encrypted-at-rest mappings with automatic expiry.

import asyncio
from cryptography.fernet import Fernet
from redis.asyncio import Redis

from noirdoc.mappings.backends.redis_backend import RedisMappingBackend
from noirdoc.mappings.store import MappingStore

async def main() -> None:
    redis = Redis.from_url("redis://localhost:6379")
    store = MappingStore(
        backend=RedisMappingBackend(redis),
        encryption_key=Fernet.generate_key(),  # keep stable across workers
    )
    # store.save(request_id=..., tenant_id=..., mapper=...)
    # mappings = await store.load(request_id)

asyncio.run(main())

The encryption_key must be identical across workers that need to read the same mappings. MappingStore.save() accepts a ttl_days kwarg (default 30).

Noirdoc Cloud

Don't want to run this yourself? Noirdoc Cloud is the hosted API wrapper: a privacy-preserving reverse proxy for LLM calls that uses this exact pipeline, plus multi-tenancy, audit, and provider key management. Compliance story: what's on GitHub is what the cloud runs.

Development

This repo uses the shared noirdoc tooling standard (uv + ruff/mypy). Common tasks go through make:

make install   # set up the dev environment
make check     # lint + format-check + typecheck + test — run before pushing
make test      # run fast tests (excludes slow ML-model tests)

Run make help for the full list of targets (also: make lint, make fmt, make typecheck, make test-slow, make models).

Contributing

Bug reports, detectors, and format support are all welcome. See CONTRIBUTING.md for dev setup, tests, and the recognizer pattern.

Security

Report vulnerabilities via GitHub's private vulnerability reporting — see SECURITY.md. Please don't open public issues for security bugs.

Changelog

See CHANGELOG.md. Follows Keep a Changelog and SemVer.

License

MIT © 2026 Antonio Maiolo / Nextaim GmbH. See LICENSE.

Built by Nextaim GmbH · noirdoc.de