wardcat 1.0.0

PII detection and anonymization for LLM inputs — regex, NER and on-prem LLM hybrid engine

██╗    ██╗ █████╗ ██████╗ ██████╗  ██████╗ █████╗ ████████╗
██║    ██║██╔══██╗██╔══██╗██╔══██╗██╔════╝██╔══██╗╚══██╔══╝
██║ █╗ ██║███████║██████╔╝██║  ██║██║     ███████║   ██║   
██║███╗██║██╔══██║██╔══██╗██║  ██║██║     ██╔══██║   ██║   
╚███╔███╔╝██║  ██║██║  ██║██████╔╝╚██████╗██║  ██║   ██║   
 ╚══╝╚══╝ ╚═╝  ╚═╝╚═╝  ╚═╝╚═════╝  ╚═════╝╚═╝  ╚═╝   ╚═╝   

PII detection and anonymization for LLM inputs — hybrid regex + NER + on-prem LLM engine.

wardcat scans text for personally identifiable information (PII) before it reaches an LLM, and either warns about or replaces the sensitive data with salted SHA-256 hashes. It supports Turkish, English, German, and French out of the box.

📖 Documentation lives in docs/ (MkDocs + Material, with an auto-generated API reference). Preview it locally with uv run --group docs mkdocs serve.

Detection is opt-in: a bare Wardcat() detects nothing — you configure the layers and entities you want. Here is the full picture — all three layers, a per-entity action policy, the semantic guardrail, and anonymization:

import os
from wardcat import Wardcat, Backend, Entity, Action

# One guard, all three detection layers — regex + SpaCy NER + on-prem LLM.
guard = (
    Wardcat(salt=os.environ["WARDCAT_SALT"])               # your app supplies the salt
    .with_ner(language="tr")                               # names / orgs  (needs wardcat[ner])
    .with_llm(backend=Backend.OLLAMA, model="gemma3:12b")  # contextual + semantic (needs Ollama)
    # An action per entity: hash IDs, mask contact details, redact names, flag IPs.
    .add_entities([Entity.CREDIT_CARD, Entity.IBAN, Entity.TC_ID], action=Action.HASH)
    .add_entities([Entity.EMAIL, Entity.PHONE], action=Action.MASK)
    .add_entity(Entity.PERSON, Action.REDACT)
    .add_entity(Entity.IP_ADDRESS, Action.WARN)
)

text = "Ben Ahmet Yılmaz, kartım 4111 1111 1111 1111, e-posta [e-mail]."

# 1) Semantic guardrail: does the text contain sensitive info at all? (holistic LLM yes/no)
if guard.is_sensitive(text):

    # 2) Anonymize the PII before the text is stored, logged, or forwarded.
    result = guard.scan(text)
    print(result.sanitized_text)
    # Ben [PERSON], kartım [CREDIT_CARD:b22b36262d8d2769], e-posta a****@example.com.

    print(result.redacted())   # PII-free dict, safe for logs / APIs

The NER and LLM layers are optional. Regex-only detection needs no models — just Wardcat(salt=...).add_entity(Entity.CREDIT_CARD, Action.HASH) and .scan(...). Add with_ner(...) / with_llm(...) when you want names or contextual/semantic detection. See Installation.

---

Features

• Hybrid detection — Regex + SpaCy NER + on-prem LLM (Ollama, vLLM, OpenAI-compatible, HuggingFace Transformers)
• Semantic sensitivity gate — is_sensitive(text) → bool: a holistic yes/no on whether text is safe to send onward (LLM-only), catching confidential content the typed detectors miss (unreleased financials, deal terms, a confidential project); optional per-language prompt
• Ensemble adjudication (optional) — the LLM verifies/relabels/drops regex & NER candidates and adds what they missed, in one call; deterministic regex results are always protected
• Four actions — warn (keep text, report only), hash ([TYPE:16hex] via SHA-256 + salt; the default when action is omitted), redact ([TYPE] label, no hash), mask (entity-aware partial masking)
• Checksum validation — TC_ID (Nüfus İdaresi algorithm), IBAN (mod-97), and CREDIT_CARD (Luhn mod-10) validated before flagging — eliminates false positives
• Rainbow table protection — user-defined salt for all hashes
• Two APIs — method chaining (programmatic) and YAML (declarative)
• Async & batch — scan_async / scan_batch / is_sensitive_async; concurrent requests overlap (native async LLM I/O), one shared guard is safe to reuse across scans
• Multilingual support — Turkish, English, German, and French for names, addresses, birth dates, and phone numbers; plus Spanish, Italian, Dutch address patterns; TC_ID, IBAN, SSN, NIN, DNI/NIE, UK postcodes, US ZIP+4, EU VAT numbers and more
• Secret detection — API keys and tokens (OpenAI, Anthropic, Stripe, AWS, Google, GitHub, GitLab, Slack, Twilio, SendGrid, npm) and PEM private keys
• Passport detection — contextual passport number detection (regex keyword-based + LLM) for any country
• Evasion resistance — confusable/homoglyph folding catches lookalike-character tricks (Cyrillic/fullwidth/Arabic-Indic) that fool ASCII-oriented regex
• Fault tolerance — a layer that can't run (e.g. LLM backend down) degrades gracefully and is surfaced on ScanResult.warnings instead of crashing the scan
• DoS protection — inputs exceeding 500 KB are rejected
• Safe logging API — result.redacted() returns a PII-free dict for logs and APIs

Disclaimer. wardcat is a best-effort PII detector — it does not catch everything (false negatives and positives are expected) and is not legal advice or a substitute for compliance review (e.g. GDPR/KVKK); using it does not by itself make a system compliant. Validate it against your own data and requirements. Provided "as is" (MIT — see LICENSE).

---

Contents

Installation · Quick Start · Supported Entity Types · Output Structure · Security · Configuration · Project Structure · Testing · Known Limitations · Development

---

Installation

# Base install — regex detection + Ollama/OpenAI-compatible LLM backend
pip install wardcat

# + SpaCy NER (PERSON, ORG, ADDRESS detection)
pip install "wardcat[ner]"

# Everything at once (SpaCy + Transformers)
pip install "wardcat[all]"

Or, for development, install from source with uv:

git clone https://github.com/oguzhantopcu0/wardcat.git
cd wardcat
uv sync                 # base: regex + Ollama/OpenAI-compatible LLM backend
uv sync --extra ner     # + SpaCy NER (PERSON, ORG, ADDRESS)
uv sync --extra all     # + HuggingFace Transformers backend

To use SpaCy NER (PERSON, ORG, ADDRESS detection) you need a language model. The simplest path is to let wardcat resolve and download it for you via the language= builder:

from wardcat import Wardcat, Language

# Turns NER on, picks the right model from the catalog, and downloads it if missing
guard = Wardcat().with_ner(language=Language.EN)                   # → en_core_web_sm
guard = Wardcat().with_ner(language=Language.TR, spacy_size="md")  # → tr_core_news_md

Or download a model yourself with SpaCy's own CLI:

uv run python -m spacy download en_core_web_sm     # English (recommended)
uv run python -m spacy download tr_core_news_md    # Turkish (recommended)

If a requested SpaCy model is not installed, wardcat automatically falls back to any installed model of the same language and logs a warning. SpaCy is not required if you only need regex-based detection.

---

Quick Start

Upgrading from a pre-1.0 version? Each breaking change is listed with migration steps in the CHANGELOG — e.g. NER is now configured only via with_ner(...) (0.7.0) and LLM backends are no longer user-extensible (0.9.0). As of 1.0, the public API is stable (semantic versioning).

Programmatic API

from wardcat import Wardcat, Entity, Action

guard = (
    Wardcat(salt="my-secret-salt")
    .add_entity(Entity.CREDIT_CARD, Action.HASH)
    .add_entity(Entity.EMAIL,       Action.WARN)
    .add_entity(Entity.TC_ID,       Action.HASH)
)

result = guard.scan("""
  Customer: Ali Veli, TC: 12345678950
  Card: 4111 1111 1111 1111
  Email: [e-mail]
""")

print(result.sanitized_text)
# Customer: Ali Veli, TC: [TC_ID:86349f34a1bc2d5e]
# Card: [CREDIT_CARD:b22b36262d8d2769]
# Email: [e-mail]   ← warn: text is kept

for v in result.violations:
    print(f"[{v.action}] {v.entity_type}: {v.original!r}")
# [hash] TC_ID: '12345678950'
# [hash] CREDIT_CARD: '4111 1111 1111 1111'
# [warn] EMAIL: '[e-mail]'

Typo-proof config with Entity and Action constants

Prefer constants over bare strings — your IDE autocompletes them and a typo is caught at edit time instead of becoming a silent runtime warning. They are fully interchangeable with the string forms (Entity.EMAIL == "EMAIL"):

from wardcat import Wardcat, Entity, Action

guard = (
    Wardcat(salt="my-secret-salt")
    .add_entity(Entity.CREDIT_CARD, action=Action.HASH)
    .add_entity(Entity.EMAIL,       action=Action.REDACT)
    .add_entity(Entity.PHONE,       action=Action.MASK)
)

# Batch form — also accepts Entity keys and Action values:
guard.add_entities({
    Entity.CREDIT_CARD: Action.HASH,
    Entity.EMAIL:       Action.REDACT,
})

Choosing which filters run, and on which layer

Each entity can be detected by one or more of three layers — regex (deterministic), ner (SpaCy), and llm (contextual/semantic). When you enable an entity it runs on every layer that supports it; pass layers=[...] to target a specific layer — for example, keep a semantic-only entity off the regex/NER path:

# Detect EMAIL with regex only; leave SPECIAL_CATEGORY (GDPR Art. 9) to the LLM
guard.add_entity("EMAIL", action="redact", layers=["regex"])
guard.add_entity("SPECIAL_CATEGORY", action="redact", layers=["llm"])

To turn on many filters at once, use add_entities(). It accepts a list, a {name: action} mapping, or a {name: {...}} mapping for per-entity control, and applies them in a single rebuild:

from wardcat import Wardcat, turkish_entities

guard = Wardcat()

# a) a whole predefined group with one action
guard.add_entities(turkish_entities(), action="hash")

# b) an explicit list
guard.add_entities(["EMAIL", "CREDIT_CARD", "IBAN"], action="redact")

# c) per-entity actions and layers in one call
guard.add_entities({
    "CREDIT_CARD":      "hash",
    "EMAIL":            {"action": "mask"},
    "SPECIAL_CATEGORY": {"action": "redact", "layers": ["llm"]},
})

Predefined groups (core_entities, financial_entities, turkish_entities, european_entities, uk_entities, us_entities, network_entities, identity_entities, all_entities) are importable from wardcat and pair naturally with add_entities().

Enable everything, then prune

Entity.ALL turns on every known entity in one call; remove_entity() (and remove_entities()) then disables the ones you do not want. This "allow-list by exclusion" pattern is often the quickest way to a strict policy:

from wardcat import Wardcat, Entity

guard = (
    Wardcat(salt="my-secret-salt")
    .add_entity(Entity.ALL, action="hash")   # everything on, hashed
    .remove_entity(Entity.ORG)               # …except organisation names
    .remove_entities([Entity.UUID, Entity.MAC_ADDRESS])
)

# remove_entity(Entity.ALL) disables everything again.

To change the action of an entity that is already enabled — without touching which layers it runs on — use change_entity_action():

guard.change_entity_action(Entity.EMAIL, Action.HASH)      # warn → hash
guard.change_entity_action(Entity.ALL, Action.REDACT)      # every enabled entity → redact

change_entity_action() never silently re-enables an entity: changing the action of a removed or never-added entity raises ConfigError — add it first with add_entity().

To inspect the current policy at any point:

guard.enabled_entities()            # {"CREDIT_CARD", "EMAIL", ...} — what's on
guard.get_entity_action("EMAIL")    # "hash"  (None if the entity is not enabled)
guard.entity_policy()               # {"CREDIT_CARD": "hash", "EMAIL": "warn", ...}

# Discover what wardcat *can* detect (and on which layer):
Wardcat.supported_entities()        # every known entity type
Wardcat.supported_entities("ner")   # {"PERSON", "ORG", "ADDRESS"}
Wardcat.supported_entities("llm")   # contextual/semantic types

Removing an entity that was never enabled is a no-op (an unknown name logs a warning, to catch typos). Passing a bare string to add_entities() / remove_entities() (instead of a list) raises ConfigError — use the singular add_entity() / remove_entity() for one entity. Entity.All is a deprecated alias of Entity.ALL.

Declarative API (YAML)

from wardcat import Wardcat

guard = Wardcat(config_path="config/my_policy.yaml")
result = guard.scan(text)

# config/my_policy.yaml
salt: ""          # read from env in production
use_ner: false   # NER is off by default; set true AND provide a model below
# spacy_model: "en_core_web_sm"      # one model
# spacy_models: ["en_core_web_sm", "de_core_news_sm"]   # or several (multilingual)

entities:
  CREDIT_CARD: { enabled: true,  action: hash }
  EMAIL:       { enabled: true,  action: warn }
  TC_ID:       { enabled: true,  action: hash }
  IBAN:        { enabled: true,  action: hash }
  PERSON:      { enabled: true,  action: hash }
  ORG:         { enabled: false, action: warn }

Batch Scanning

guard = Wardcat(salt="s").add_entities(["EMAIL", "CREDIT_CARD"])
results = guard.scan_batch([
    "[e-mail]",
    "Card: 4111 1111 1111 1111",
    "Clean text.",
])

for r in results:
    print(r.is_clean, len(r.violations))
# False 1
# False 1
# True  0

Async & concurrency

Every scanning entry point has an async twin — scan_async, scan_batch_async, is_sensitive_async. CPU layers (regex, NER) run in a thread pool; the LLM layer uses native async I/O (httpx.AsyncClient), so concurrent requests overlap instead of blocking each other:

import asyncio

guard = Wardcat(salt="s").with_llm(model="llama3.1:8b").add_entity("EMAIL")

texts = ["mail me at [e-mail]", "card 4111 1111 1111 1111", "clean text"]

async def main():
    # one async scan
    result = await guard.scan_async(texts[0])

    # many at once — their LLM calls overlap (asyncio.gather)
    results = await asyncio.gather(*(guard.scan_async(t) for t in texts))

asyncio.run(main())

Concurrency notes

• Build one guard and share it. A Wardcat instance is safe to reuse across concurrent scans (the SpaCy model cache, LLM cache, and detectors are shared and lock-protected). Don't reconfigure it (add_entity / with_ner / with_llm) while it is serving scans — configure once at startup.
• scan_async is non-blocking, but the LLM server is the real ceiling. Your requests overlap in Python, yet a single Ollama on one GPU may still process them near-sequentially. For genuine parallel LLM throughput use vLLM (continuous batching) or raise OLLAMA_NUM_PARALLEL with enough VRAM.
• Regex/NER-only scans are already sub-millisecond to ~100 ms, so concurrency there is rarely the bottleneck.

FastAPI (async handler):

from fastapi import FastAPI
from wardcat import Wardcat

guard = Wardcat(salt="s").add_entities(["EMAIL", "CREDIT_CARD", "TC_ID"])
app = FastAPI()

@app.post("/scan")
async def scan(text: str):
    result = await guard.scan_async(text)      # await — does not block the loop
    return {"sanitized": result.sanitized_text, "clean": result.is_clean}

Catching every occurrence (value propagation)

Model-based layers (SpaCy NER, the LLM) sometimes report a value that repeats in a document only once, which would leave the other occurrences unredacted. Enable with_propagation() so that once any layer detects a value, every whole-token occurrence of it is anonymized — with that value's entity type and action:

guard = (
    Wardcat(salt="s")
    .with_ner(language="en")
    .add_entity(Entity.PERSON, Action.REDACT)
    .with_propagation()          # a name caught once → redacted everywhere
)

It is off by default because it can over-redact (e.g. a short, common name). Only exact, token-bounded matches at least min_length chars long (default 3) are propagated, and deterministic regex spans still win overlaps — a propagated match never displaces a checksum-validated one. Structural PII (email, phone, IBAN…) is already caught exhaustively by the regex layer, so propagation mainly helps names and other model-only entities.

On-prem LLM

Fluent setup (recommended). Instead of passing a dozen llm_* / spacy_* constructor arguments, use the chainable builders — they keep each layer's config in one place and read top-to-bottom:

from wardcat import Wardcat, Backend, Language

guard = (
    Wardcat(salt="s")
    .with_ner(language=Language.TR)                       # NER layer
    .with_llm(backend=Backend.OLLAMA, model="llama3.2",   # LLM layer
              adjudicate=True)
)

with_ner() and with_llm() both return self, so they chain back-to-back (regex + NER + LLM all active).

The LLM detector is configured only via with_llm() (or a YAML config_path) — it is not a set of constructor arguments. backend is the backend type (use the Backend constants — Backend.OLLAMA, Backend.VLLM, Backend.OPENAI_COMPATIBLE, Backend.TRANSFORMERS; plain strings work too); the address goes to base_url.

Option 1 — Run locally with Ollama:

# Install Ollama: https://ollama.com
ollama pull llama3.2

from wardcat import Wardcat, Backend

guard = Wardcat(salt="s").with_llm(
    backend=Backend.OLLAMA,
    model="llama3.2",
    base_url="http://localhost:11434",
)

Option 2 — vLLM server:

vllm serve meta-llama/Llama-3.1-8B-Instruct   # default port 8000

from wardcat import Wardcat, Backend

guard = Wardcat(salt="s").with_llm(
    backend=Backend.VLLM,
    model="meta-llama/Llama-3.1-8B-Instruct",   # the served model name
    base_url="http://localhost:8000/v1",         # vLLM's default endpoint
    # api_key="..."   # only if vLLM was started with --api-key
)

Backend.VLLM sends the full chat message array natively and defaults to vLLM's http://localhost:8000/v1. For other OpenAI-compatible servers (LM Studio, LocalAI, LiteLLM), use Backend.OPENAI_COMPATIBLE with your endpoint's base_url.

Option 3 — HuggingFace Transformers (GPU/CPU):

from wardcat import Wardcat, Backend

guard = Wardcat(salt="s").with_llm(
    backend=Backend.TRANSFORMERS,
    model="meta-llama/Llama-3.1-8B-Instruct",
    load_in_8bit=True,  # optional: reduce VRAM usage
)

Note: Loopback HTTP (localhost / 127.0.0.1 / ::1) is allowed with no warning — it never leaves the machine, so the common local-Ollama setup needs no allow_http. HTTP to a remote host is blocked (pass allow_http=True to override); use HTTPS in production via a reverse proxy (nginx, Caddy).

Custom actions (extensible)

Actions (hash/redact/mask/warn) come from a registry, so you can add your own — tokenize, encrypt, format-preserving masking — without changing wardcat. An action maps a span to its replacement (or None to keep the text):

from wardcat import Wardcat, register_action

# ctx carries the salt; the span has .entity_type, .text, .start, .end
register_action("tokenize", lambda span, ctx: f"<{span.entity_type}:{vault.put(span.text)}>")

guard = Wardcat(salt="s").add_entity("EMAIL", "tokenize")   # use it like any built-in

Detection and anonymization are separate stages: DetectionEngine finds the spans, and a standalone Anonymizer applies the actions — so you can reuse the action registry or the Anonymizer independently.

LLM backends are not user-extensible. wardcat ships four — ollama, openai_compatible, vllm, transformers — selected via the Backend enum. A third-party backend would sit outside wardcat's safety checks (the plaintext-HTTP-to-remote guard, PII handling), so point a built-in at your endpoint instead: openai_compatible covers most OpenAI-style gateways.

Ensemble adjudication

By default the three layers run independently and their findings are merged (union). With with_llm(adjudicate=True), the engine instead sends the regex/NER candidates to the LLM, which — in a single call — verifies each one (keep / relabel / drop) and adds any PII the other layers missed. Deterministic, checksum-validated regex spans are always kept regardless of the LLM verdict. This sharply reduces NER noise (e.g. job titles mislabeled as names, or a model run on the wrong language):

guard = Wardcat().with_ner(spacy_model="de_core_news_sm").with_llm(
    model="gemma3:12b",
    adjudicate=True,   # LLM acts as detector + arbiter in one call
)

Adjudication has no effect in LLM-only deployments (no regex/NER candidates to judge) — the LLM simply runs as a pure detector.

Is this text sensitive? (semantic gate)

Sometimes you don't need what the PII is — just a yes/no on whether a text is safe to send onward. is_sensitive() asks the configured LLM for a single holistic judgement (a general semantic decision, not the per-entity extraction of scan()), so it also catches things the enumerated detectors don't — unreleased financials, deal terms, a confidential project:

from wardcat import Wardcat, Backend

guard = Wardcat().with_llm(backend=Backend.OLLAMA, model="gemma3:12b")

guard.is_sensitive("Ödeme için IBAN TR58 0000 0011 1111 1111 1111 11")   # True
guard.is_sensitive("Aksa Teknoloji'nin açıklanmamış Q2 net kârı 47,3M TL")  # True (confidential)
guard.is_sensitive("What's the weather like tomorrow?")                    # False

# guardrail before calling an external service
if guard.is_sensitive(user_text):
    raise ValueError("won't forward sensitive text")

• LLM-only — configured through with_llm(...); no entities to enable, no regex/NER runs. Raises ConfigError if the LLM layer isn't set up.
• Fail-closed — if the backend is unreachable the error propagates (a guardrail never silently treats sensitive text as safe). Empty text is False.
• Localized prompt — with_llm(language="tr") (or de/fr) uses a system prompt written in that language, which can help smaller models; any other value uses the English, multilingual-aware prompt. (This affects only is_sensitive(), not the scan() detection prompt.)
• Long inputs are chunked at paragraph boundaries — any sensitive chunk makes the whole text sensitive — and oversized input is rejected (max_text_bytes).
• Async: await guard.is_sensitive_async(text).

---

Examples

Runnable scripts in examples/:

File	Shows
demo.py	Programmatic + YAML APIs
batch_and_async.py	scan_batch and the async API (regex-only, no services)
llm_hybrid.py	regex + NER + LLM with ensemble adjudication (needs Ollama)
asgi_middleware.py	Copy-paste ASGI middleware (FastAPI/Starlette) that scans request bodies — wardcat ships no web-framework code; this is a self-contained example

---

Supported Entity Types

Regex (built-in, no dependencies)

Financial & Identity

Entity	Default Action	Description
CREDIT_CARD	hash	Visa, MC, Amex, Discover — with or without separators; Luhn (mod-10) validated
IBAN	hash	International IBAN — mod-97 checksum validated
SSN	hash	US Social Security Number (123-45-6789)
NIN	hash	UK National Insurance Number (AB123456C)
TC_ID	hash	Turkish national ID — 11 digits, Nüfus İdaresi checksum validated
EU_NATIONAL_ID	hash	Spanish DNI (12345678Z) / NIE (X1234567L), French INSEE (15 digits)
CODICE_FISCALE	hash	Italian tax code (RSSMRA85T10A562S)
VAT_NUMBER	warn	EU VAT (DE/FR/GB/IT/ES/AT/NL prefixed) + Turkish Vergi No (keyword-based)
PASSPORT	hash	Passport numbers — keyword-based (passport no:, pasaport, Reisepass, passeport)
DATE_OF_BIRTH	hash	Birth dates — month names (TR/EN/DE/FR) or keyword + numeric/ISO date
VEHICLE_PLATE	warn	Turkish vehicle plates (34 ABC 123)
FINANCIAL_AMOUNT	redact	Monetary amounts (₺/$/€/£, 45.000 TL, 2.1 milyon TL) — off by default

Contact & Location

Entity	Default Action	Description
EMAIL	warn	RFC-compliant email addresses
PHONE	warn	Turkish (0, +90), French national (01 23 45 67 89), German mobile (0151 …), and international E.164 (+1, +44, …)
ADDRESS	warn	Turkish (Cad., Sok., Mah.), English (Street, Avenue, Road…), French (Rue, Allée…), Spanish (Calle, Avenida…), Italian (Piazza, Corso…), Dutch (straat, gracht…), German (Straße, Weg, Platz…)
POSTAL_CODE	warn	Turkish postal codes (01000–81999)
UK_POSTAL_CODE	warn	British postcodes (SW1A 1AA, GU21 6TH, M1 1AE)
US_ZIP_CODE	warn	US ZIP+4 codes (12345-6789)

Network & Technical

Entity	Default Action	Description
IP_ADDRESS	warn	IPv4 addresses
IPv6	warn	IPv6 addresses (full and compressed forms)
MAC_ADDRESS	warn	Network hardware address (00:1A:2B:3C:4D:5E)
UUID	warn	RFC 4122 UUID / GUID
JWT	hash	JSON Web Token (starts with eyJ)
CUSTOM_SECRET	hash	API keys & tokens: OpenAI/Anthropic (sk-, sk-ant-), Stripe (sk_live_), AWS (AKIA), Google (AIza, ya29.), GitHub (ghp_), GitLab (glpat-), Slack (xoxb-, webhook URLs), Twilio (SK/AC), SendGrid (SG.), npm (npm_), and PEM private-key blocks

SpaCy NER (requires spacy + language model)

Entity	Default Action	Description
PERSON	hash	Person names (first + last) — cross-language
ORG	warn	Organization / company names
ADDRESS	warn	Location entities (complements regex)

NER requires an explicit model — there is no default. NER is off by default; calling with_ner() without a model (or language) raises ConfigError. Choose a model in a documented way via language= (recommended) or spacy_model=. Running, say, the Turkish model on German text produces noisy results, so pick the model per language (or rely on the LLM layer for cross-language names). A multilingual gazetteer filters out job titles, HR terms, and abbreviations (EN/DE/FR/TR) that NER models commonly mislabel.

Pick a language, not a model name. Use the Language constants (or their ISO codes) and an optional size tier — wardcat resolves the right model from its catalog and downloads it automatically if it is missing:

from wardcat import Wardcat, Language

# Selecting a language turns NER on and implies auto-download (off with auto_download=False)
guard = Wardcat().with_ner(language=Language.DE)                  # → de_core_news_sm
guard = Wardcat().with_ner(language=Language.FR, spacy_size="md") # → fr_core_news_md (sm/md/lg/trf)
guard = Wardcat().with_ner(language=Language.TR, spacy_size="lg") # → tr_core_news_lg

# Or name the SpaCy package(s) explicitly — one or several:
guard = Wardcat().with_ner(spacy_model="en_core_web_sm")
guard = Wardcat().with_ner(spacy_model=["en_core_web_sm", "de_core_news_sm"])

Supported languages: Language.EN, DE, FR, ES, IT, NL, PT, TR (plain ISO codes like "de" are also accepted). If the requested size is unavailable for a language, the recommended model is used.

Multi-language text

For text that mixes several languages you have two options:

1. LLM layer (recommended, no extra models). The LLM prompt is multilingual, so a single LLM-enabled guard detects names across languages without loading any SpaCy model:

   guard = Wardcat().with_llm(model="llama3.1:8b")  # handles EN+DE+FR+TR in one call
   

2. Multiple SpaCy models. Pass a list of languages — wardcat loads one NER model per language and the engine merges their results. Each model adds RAM and roughly multiplies NER scan time, so this is opt-in:

   from wardcat import Wardcat, Language
   guard = Wardcat().with_ner(language=[Language.EN, Language.DE, Language.FR])  # 3 NER models, auto-downloaded
   

   This is explicit — you list the languages; wardcat does not guess the language of the input. The regex layer is multilingual regardless of these choices.

LLM (requires on-prem LLM backend)

Entity	Default Action	Description
PASSPORT	hash	Passport numbers of any country — contextual detection (e.g. Passport: A12345678)
EU_NATIONAL_ID	hash	French INSEE, German Personalausweis — contextual variants not caught by regex
UK_POSTAL_CODE	warn	Postcodes in ambiguous contexts
US_ZIP_CODE	warn	ZIP codes in ambiguous contexts
CUSTOM_SECRET	hash	Contextual secrets — password=VALUE, api_key=VALUE, access codes
SPECIAL_CATEGORY	redact	GDPR Art.9 special-category data — health, religion, ethnicity, political opinion, sexual orientation, trade-union, genetic/biometric. Off by default (semantic, LLM-only)
(any above)	—	LLM supplements and verifies all regex/NER entity types

GDPR special categories: SPECIAL_CATEGORY flags sensitive statements that have no pattern — a stated health condition, religious or political affiliation, or trade-union membership — which only the LLM can detect. It is off by default; enable it under llm_detector.entities. Because it is semantic and subjective, expect lower precision than structural entities.

---

Output Structure

scan() and scan_batch() return a ScanResult per text:

@dataclass
class ScanResult:
    original_text:  str               # unmodified input — contains raw PII
    sanitized_text: str               # anonymized output — safe to forward to LLM
    violations:     List[Violation]   # all detected violations
    is_clean:       bool              # True if no violations found
    warnings:       List[str]         # non-fatal issues — a layer that could not run

    def redacted(self) -> dict:       # PII-free dict — safe for logging and APIs
        ...

@dataclass
class Violation:
    entity_type: str          # e.g. "CREDIT_CARD", "EMAIL"
    original:    str          # raw value from input — contains PII
    start:       int          # start index in original text
    end:         int          # end index in original text
    action:      Action       # WARN | HASH | REDACT | MASK
    replacement: str | None   # "[TYPE:16hex]" for hash, "[TYPE]" for redact, masked value for mask, None for warn
    confidence:  float        # checksum 1.0 · structural regex 0.97 · fuzzy regex 0.90 · NER/LLM 0.85

Degraded scans — check warnings. If a detector layer cannot run (most commonly the LLM backend being unreachable), the scan still returns the other layers' results but records the failure in result.warnings. A non-empty warnings means detection was degraded — some PII may have been missed — so you are not silently misled into thinking every layer ran:

result = guard.scan(text)
if result.warnings:
    logger.warning("PII scan degraded: %s", result.warnings)

---

Security

Hashing

Sensitive values are replaced in the sanitized text using the format [TYPE:16hex] (64-bit entropy):

4111 1111 1111 1111  →  [CREDIT_CARD:b22b36262d8d2769]
12345678950          →  [TC_ID:86349f34a1bc2d5e]

Limitation — deterministic hashing is not anonymization. The same value always hashes to the same token (this is intentional: it lets you correlate records). But because the hash is deterministic and SHA-256 is fast, low-entropy PII (phone numbers, SSNs, TC IDs) can be recovered by brute force by anyone who has the salt and knows the value format. Treat hashed output as pseudonymized, not anonymized: keep the salt secret, and use redact/mask when you need values that cannot be reversed.

Salt

Salt prevents rainbow table attacks. Set it via environment variable in production:

import os
guard = Wardcat(salt=os.environ["WARDCAT_SALT"]).add_entity("CREDIT_CARD", "hash")

Never hardcode the salt in source code or config files.

Safe Logging

ScanResult.original_text and violations[].original contain raw PII. Use result.redacted() for logs and API responses:

result = guard.scan(text)

# ✗ Leaks PII to logs
logger.info("result: %s", result.original_text)

# ✓ Safe — no raw PII
logger.info("result: %s", result.redacted())

Input Size Limit

Inputs exceeding 500 KB raise a ValueError. Split large documents into smaller chunks before scanning.

---

Configuration

The library never reads environment variables. Pass everything explicitly — Wardcat(salt=...).with_llm(base_url=..., allow_http=...) or a YAML config_path. Read secrets from the environment in your application and hand them to the constructor; wardcat itself does no implicit environment lookup.

To allow plaintext HTTP to a remote LLM (blocked by default), pass with_llm(allow_http=True).

---

Project Structure

wardcat/
├── src/wardcat/
│   ├── guard.py              # Wardcat — main interface
│   ├── core/
│   │   ├── engine.py         # DetectionEngine — overlap resolution, action application
│   │   └── models.py         # Action, Violation, ScanResult
│   ├── detectors/
│   │   ├── base.py           # BaseDetector ABC
│   │   ├── regex_detector.py # 25+ regex patterns with checksum/Luhn validation
│   │   ├── ner_detector.py   # SpaCy NER (multilingual) + gazetteer FP filter
│   │   └── llm_detector.py   # LLM-based detection with hallucination filter
│   ├── llm/
│   │   ├── backends/         # ollama, openai_compat, transformers
│   │   ├── model_catalog.py  # Supported model list
│   │   └── prompt.py         # PII detection prompt builder
│   ├── config/
│   │   └── loader.py         # YAML loader, env var overrides
│   └── utils/
│       └── hashing.py        # SHA-256 + salt
├── tests/
│   ├── unit/                 # Component-level tests
│   └── integration/          # Scenario and adversarial tests
├── config/
│   └── default.yaml          # Example policy file
└── pyproject.toml

---

Testing

# All tests
uv run pytest tests/unit/ tests/integration/

# Unit tests only
uv run pytest tests/unit/

# NER tests (requires SpaCy model)
uv run pytest -m ner

# Live LLM tests against a real Ollama model (auto-skipped if unavailable)
uv run pytest -m slow tests/integration/test_llm_live.py
# Pick the model: WARDCAT_TEST_LLM_MODEL=llama3.2:1b uv run pytest -m slow ...

# Skip slow tests (fast run)
uv run pytest -m "not slow"

# Coverage report
uv run pytest --cov=src/wardcat --cov-report=term-missing

---

Known Limitations

Case	Description	Mitigation
Homoglyph domain	Confusable folding (normalize_confusables, on by default) catches Cyrillic/Greek lookalikes and fullwidth/Arabic digits (ali@tеst.com, ４111…), but it is a curated skeleton, not the full Unicode table — an exotic lookalike may slip through	Add an allowlist/extra validation for high-stakes fields; the LLM layer can flag suspicious text
US_ZIP_CODE	A bare 5-digit ZIP is matched only with a ZIP: keyword; otherwise ZIP+4 (12345-6789) is required, to avoid false positives	Enable POSTAL_CODE (catches any bare 5-digit), or the LLM layer for contextual ZIPs
EU_NATIONAL_ID	Spanish DNI/NIE and French INSEE are matched by regex; German IDs are not	Enable the LLM layer — it catches German (and other) national IDs contextually
PASSPORT	Regex needs a passport keyword (passport no:, pasaport, Reisepass, passeport) — bare passport numbers are too generic to match safely	Enable the LLM layer — it catches unlabeled passport numbers
FINANCIAL_AMOUNT / VAT_NUMBER	FINANCIAL_AMOUNT is off by default; a bare Turkish Vergi No needs a keyword	Enable FINANCIAL_AMOUNT for confidential docs; use a Vergi No/VKN keyword or the LLM layer for VAT
Multilingual NER	One SpaCy model loads per language; wardcat bundles no language detection by design (keeps the core dependency-light)	Detect the language yourself (check supported_languages()) and pass several models via language=[...], or use the language-agnostic LLM layer
European addresses	Regex needs a street-type keyword (Straße, Rue, Calle…); unnumbered informal addresses may be missed	Use the NER ADDRESS / LLM layer, or add a custom_patterns rule for your address format
Turkish NER (tr_core_news_trf)	The transformer model is incompatible with SpaCy 3.5+	Use tr_core_news_md or tr_core_news_lg
Turkish NER quality	tr_core_news_md/lg are news-trained and may miss names in non-standard contexts	Combine with .with_llm(...) — the LLM catches names NER misses

---

Development

uv sync --dev
uv run pytest tests/unit/ tests/integration/ tests/benchmark/

Requires Python 3.11+.

Detection quality. tests/benchmark/ scores the detectors two ways: a false-positive suite (detectors must stay quiet on clean text) and a precision/recall harness over a labelled, checksum-valid corpus. Both run in CI. Print the P/R report:

uv run python -m tests.benchmark.eval_harness

Widen coverage by adding rows to CORPUS in tests/benchmark/eval_harness.py.