datafog 4.3.0b10

Lightning-fast PII detection and anonymization library with 190x performance advantage

DataFog Python

DataFog is a Python library for detecting and redacting personally identifiable information (PII).

It provides:

• Fast structured PII detection via regex
• Optional NER support via spaCy and GLiNER
• A simple agent-oriented API for LLM applications
• Backward-compatible DataFog and TextService classes

Installation

# Core install (regex engine)
pip install datafog

# Add spaCy support
pip install datafog[nlp]

# Add GLiNER + spaCy support
pip install datafog[nlp-advanced]

# Everything
pip install datafog[all]

Quick Start

import datafog

text = "Contact [e-mail] or call [phone]"
clean = datafog.sanitize(text, engine="regex")
print(clean)
# Contact [EMAIL_1] or call [PHONE_1]

For LLM Applications

import datafog

# 1) Scan prompt text before sending to an LLM
prompt = "My SSN is 123-45-6789"
scan_result = datafog.scan_prompt(prompt, engine="regex")
if scan_result.entities:
    print(f"Detected {len(scan_result.entities)} PII entities")

# 2) Redact model output before returning it
output = "Email me at [e-mail]"
safe_result = datafog.filter_output(output, engine="regex")
print(safe_result.redacted_text)
# Email me at [EMAIL_1]

# 3) One-liner redaction
print(datafog.sanitize("Card: 4111-1111-1111-1111", engine="regex"))
# Card: [CREDIT_CARD_1]

Guardrails

import datafog

# Reusable guardrail object
guard = datafog.create_guardrail(engine="regex", on_detect="redact")

@guard
def call_llm() -> str:
    return "Send to [e-mail]"

print(call_llm())
# Send to [EMAIL_1]

Engines

Use the engine that matches your accuracy and dependency constraints:

• regex:
  • Fastest and always available.
  • Best for structured entities: EMAIL, PHONE, SSN, CREDIT_CARD, IP_ADDRESS, DATE, ZIP_CODE.
• spacy:
  • Requires pip install datafog[nlp].
  • Useful for unstructured entities like person and organization names.
• gliner:
  • Requires pip install datafog[nlp-advanced].
  • Stronger NER coverage than regex for unstructured text.
• smart:
  • Cascades regex with optional NER engines.
  • If optional deps are missing, it degrades gracefully and warns.

Backward-Compatible APIs

The existing public API remains available.

DataFog class

from datafog import DataFog

result = DataFog().scan_text("Email [e-mail]")
print(result["EMAIL"])

TextService class

from datafog.services import TextService

service = TextService(engine="regex")
result = service.annotate_text_sync("Call [phone]")
print(result["PHONE"])

CLI

# Scan text
datafog scan-text "[e-mail]"

# Redact text
datafog redact-text "[e-mail]"

# Replace text with pseudonyms
datafog replace-text "[e-mail]"

# Hash detected entities
datafog hash-text "[e-mail]"

Telemetry

DataFog includes anonymous telemetry by default.

To opt out:

export DATAFOG_NO_TELEMETRY=1
# or
export DO_NOT_TRACK=1

Telemetry does not include input text or detected PII values.

Development

git clone https://github.com/datafog/datafog-python
cd datafog-python
python -m venv .venv
source .venv/bin/activate  # Windows: .venv\Scripts\activate
pip install -e ".[all,dev]"
pytest tests/