Skip to main content

DataFog Python

DataFog is a Python library for detecting and redacting personally identifiable information (PII).

It provides:

  • Fast structured PII detection via regex
  • An offline PII firewall for AI agents: a Claude Code hook and a LiteLLM gateway guardrail (introduced in 4.6)
  • Optional NER support via spaCy and GLiNER
  • A simple agent-oriented API for LLM applications
  • Backward-compatible DataFog and TextService classes

Agent & Gateway Firewall

DataFog includes two ready-made enforcement points that catch PII at the moment it would leave your machine — offline, in microseconds, with matched values never echoed into logs or transcripts:

  • Claude Code hook (datafog-hook): gates agent tool calls (shell commands, web requests, file writes, MCP tools) and warns the model when prompts or tool results carry PII. ~105–110ms per invocation including process startup and the native engine load (DATAFOG_BACKEND=python trims ~20ms at the cost of detection coverage). Easiest install is the Claude Code plugin:

    /plugin marketplace add DataFog/datafog-claude-plugin
    /plugin install datafog@datafog
    

    Manual hook setup and limitations: examples/claude_code_hook/.

  • LiteLLM guardrail (DataFogGuardrail): redacts or blocks PII in requests and responses at the gateway, for any LiteLLM-proxied provider. In-process (tens of microseconds per message scanned on the native engine), no sidecar service. Setup: examples/litellm_guardrail/.

Both default to the high-precision entity set (EMAIL, PHONE, CREDIT_CARD, SSN); noisier types are opt-in. Known-safe values can be exempted with an allowlist: scan(text, allowlist=[...]) for exact values, allowlist_patterns=[...] for full-match regexes (e.g. ^\d{10}$ to stop unix timestamps matching as phone numbers) — available in both adapters and the API. Presidio-style entity names (EMAIL_ADDRESS, PHONE_NUMBER, US_SSN) are accepted as aliases for easy migration.

Every performance number above is reproducible with one command — methodology, pinned payloads, and comparisons against Presidio and spaCy NER live in benchmarks/.

Installation

# Core install: native engine, no extra packages needed
pip install datafog

# Add spaCy support
pip install datafog[nlp]

# Add GLiNER + spaCy support
pip install datafog[nlp-advanced]

# Add local OCR support
pip install datafog[ocr]

# Add Spark/distributed support
pip install datafog[distributed]

# Everything
pip install datafog[all]

pip install datafog pulls datafog-core, which is the native engine. Wheels exist for Linux (glibc and musl, x86_64/aarch64/s390x/ppc64le), macOS and Windows (amd64/arm64); anywhere else the source build needs a Rust toolchain. If Core cannot be imported, detection falls back to the pure-Python engine and warns once — DATAFOG_BACKEND=python selects that path deliberately.

The nlp-advanced extra includes SentencePiece and protobuf for GLiNER's multilingual tokenizer; installing the OCR extra is not required for GLiNER.

Backends

backend defaults to auto: the native engine when datafog-core is importable, the pure-Python regex engine otherwise. DATAFOG_BACKEND sets the default and backend="python" / backend="rust" override it per call. auto never selects the native engine for engine="spacy", "gliner" or "smart", which stay on the Python ML path.

The native engine adds detectors the Python engine never had — API_KEY, BEARER_TOKEN, JWT, PRIVATE_KEY, CREDENTIAL_URI, ACCOUNT_NUMBER, US_ROUTING_NUMBER, NPI, UUID and a German set — and serves prose PERSON and ADDRESS from an embedded model, so name and address detection needs no spaCy and no model download. Findings keep engine="regex" provenance and confidence=1.0 as compatibility values, not native confidence estimates. Reviewed differences from the Python engine are listed in tests/contracts/rust-1.3.0.json. datafog.native exposes the native schema directly.

Python 3.13 support is certified for the core SDK, CLI, nlp, nlp-advanced, and ocr install profiles. Donut OCR still requires a model that is available locally before runtime use. distributed and all remain optional, heavier profiles and are not part of the lightweight core path.

Python 3.14 support is certified for the core SDK and CLI. Optional profiles remain dependent on upstream Python 3.14 package availability until they are covered by the corresponding CI install-profile checks.

Quick Start

import datafog

text = "Contact john@example.com or call (555) 123-4567"
clean = datafog.sanitize(text, engine="regex")
print(clean)
# Contact [EMAIL_1] or call [PHONE_1]

For LLM Applications

import datafog

# 1) Scan prompt text before sending to an LLM
prompt = "My SSN is 123-45-6789"
scan_result = datafog.scan_prompt(prompt, engine="regex")
if scan_result.entities:
    print(f"Detected {len(scan_result.entities)} PII entities")

# 2) Redact model output before returning it
output = "Email me at jane.doe@example.com"
safe_result = datafog.filter_output(output, engine="regex")
print(safe_result.redacted_text)
# Email me at [EMAIL_1]

# 3) One-liner redaction
print(datafog.sanitize("Card: 4111-1111-1111-1111", engine="regex"))
# Card: [CREDIT_CARD_1]

German Structured PII

German structured PII is country-specific and opt-in. Use explicit locale selection or entity-type filtering when you want German VAT IDs, German IBANs, tax IDs, postal codes, passports, or residence permits.

import datafog

text = "Steuer-ID 12345678901 liegt vor."

print(datafog.scan(text, engine="regex").entities)
# []

print(datafog.scan(text, engine="regex", locales=["de"]).entities)
# [Entity(type='DE_TAX_ID', text='12345678901', ...)]

Guardrails

import datafog

# Reusable guardrail object
guard = datafog.create_guardrail(engine="regex", on_detect="redact")

@guard
def call_llm() -> str:
    return "Send to admin@example.com"

print(call_llm())
# Send to [EMAIL_1]

Engines

Use the engine that matches your accuracy and dependency constraints:

  • regex:
    • Fastest and always available.
    • Best for default structured entities: EMAIL, PHONE, SSN, CREDIT_CARD, IP_ADDRESS, DATE, ZIP_CODE (DOB and ZIP are accepted as input aliases).
    • Use locales=["de"] for German structured IDs such as DE_VAT_ID, DE_IBAN, DE_TAX_ID, DE_POSTAL_CODE, and passport or residence permit numbers.
  • spacy:
    • Requires pip install datafog[nlp].
    • Useful for unstructured entities like person and organization names.
  • gliner:
    • Requires pip install datafog[nlp-advanced].
    • Stronger NER coverage than regex for unstructured text.
  • smart:
    • Cascades regex with optional NER engines.
    • If optional deps are missing, it degrades gracefully and warns.

Optional OCR And Spark Surfaces

The 4.x line keeps the main package story centered on lightweight text PII screening. OCR and Spark remain supported optional surfaces for users who already rely on them, but they are not required for the core import, default scan/redact helpers, or guardrail helpers.

  • OCR:
    • Install datafog[ocr] for local image OCR helpers.
    • URL-based image downloading also needs datafog[web,ocr].
    • Tesseract usage requires the system tesseract binary.
    • Python 3.13 is validated for the OCR install profile, Pillow, pytesseract, and system Tesseract smoke checks.
    • Donut OCR requires datafog[nlp-advanced,ocr] and a model already available locally.
  • Spark:
    • Install datafog[distributed] for SparkService.
    • Spark PII UDF helpers also require datafog[nlp] and an installed spaCy model.
    • A Java runtime is required by PySpark.

DataFog deprecates OCR and Spark with visible use-time warnings; their APIs and extras will be removed in 5.0. The last supported version is 4.10.0 — pin it with pip install "datafog==4.10.0" — and every retired surface is tracked in the retirement schedule. See the 4.9 migration guide for the transition plan, and VERSIONING.md for what a version number promises.

Backward-Compatible APIs

The existing public API remains available.

DataFog class

from datafog import DataFog

result = DataFog().scan_text("Email john@example.com")
print(result["EMAIL"])

TextService class

from datafog.services import TextService

service = TextService(engine="regex")
result = service.annotate_text_sync("Call (555) 123-4567")
print(result["PHONE"])

CLI

# Scan text
datafog scan-text "john@example.com"

# Redact text
datafog redact-text "john@example.com"

# Replace text with pseudonyms
datafog replace-text "john@example.com"

# Hash detected entities
datafog hash-text "john@example.com"

# Enable German regex identifiers
datafog redact-text "Steuer-ID 12345678901" --locale de

Telemetry

DataFog telemetry is disabled by default.

To opt in:

export DATAFOG_TELEMETRY=1

To force telemetry off:

export DATAFOG_NO_TELEMETRY=1
# or
export DO_NOT_TRACK=1

Telemetry does not include input text or detected PII values.

Development

The 4.9 migration guide explains opt-in Rust detection, the native datafog.v5 preview, and the revised 5.0 retirement schedule for detect/process, OCR, and Spark. The Python detector remains the default.

4.10.0 requires Core >=1.3.0,<2 and capability contract 1, and installs it by default. See the 4.10.0 release notes and the 4.10 migration guide. Entity labels, locales, and activation settings come from the installed Core, allowing compatible releases to add detectors without a Python update. German detection is opt-in through locale or entity selection; UUID is opt-in through entity_types=["UUID"]. Core's structured-only PERSON is unavailable for explicit Rust text selection. The legacy overlap policy can suppress NPI in favor of PHONE; the migration guide explains native alternatives. Lock the Core version if detection output must remain reproducible.

The 4.8.1 compatibility contract records published Python behavior for the Rust migration, with frozen fixtures and instructions for independently reproducing them from the release wheel.

git clone https://github.com/datafog/datafog-python
cd datafog-python
python -m venv .venv
source .venv/bin/activate  # Windows: .venv\Scripts\activate
pip install -e ".[all,dev]"
pip install -r requirements-dev.txt
pytest tests/

Metadata

Release files for datafog 4.10.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for datafog 4.10.0
File Size Uploaded Provenance Actions
datafog-4.10.0.tar.gz 127.0 kB Without provenance Details for datafog-4.10.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for datafog 4.10.0
File Interpreter ABI Platform Actions
datafog-4.10.0-py3-none-any.whl Python 3 none any Details for datafog-4.10.0-py3-none-any.whl

Total release size: 213.4 kB

Release files / datafog-4.10.0.tar.gz

Download URL datafog-4.10.0.tar.gz
Size 127.0 kB
Tags Source
SHA-256 checksum
How to use checksums
dc6f38e37d83c85eec7b0f7566bccd6cebb4047e0d17df580c8fc3070b6ab6d4
BLAKE2b-256 checksum
How to use checksums
781b209b9bc1eeac1fb05398b3f721750ac6c9be30552d54e69b4a7d5429fc0b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.11.18

Release files / datafog-4.10.0-py3-none-any.whl

Download URL datafog-4.10.0-py3-none-any.whl
Size 86.4 kB
Tags Python 3
SHA-256 checksum
How to use checksums
16acd5a41cf10b6c2fcf86a364adf175595ded6c663fb221f67e1214729a9aa6
BLAKE2b-256 checksum
How to use checksums
203684ff7dccd3e0e5b69e1c11688d36ab5f096aad8f4b8c733bc58f936b85d2
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.11.18

Security for datafog 4.10.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

4.10.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page