veildata 0.2.0

Framework for redacting and revealing PII.

🕶️ VeilData

A lightweight framework for redacting and revealing Personally Identifiable Information (PII).

---

🧠 Why VeilData

Modern AI systems touch sensitive data every day.
VeilData makes it easy to redact, anonymize, and later restore information.

---

🚀 Quick Start

Installation

From PyPI

pip install veildata

Run from Docker

docker build -t veildata .
alias veildata="docker run --rm -v \$(pwd):/app veildata"
veildata redact data/input.csv --out data/redacted.csv

Running in Docker

docker build -t veildata .
docker run -it ghcr.io/veildata/veildata:latest

For Development

git clone https://github.com/VeilData/veildata.git
cd veildata
curl -LsSf https://astral.sh/uv/install.sh | sh
uv venv
uv sync

Quickstart Guide

Mark sensitive data

veildata redact input.txt

Example config.yaml

patterns:
  EMAIL: "\\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Z|a-z]{2,}\\b"

Reveal previously redact data

veildata reveal redacted.txt

** Using Docker**

docker run --rm -v $(pwd):/app veildata redact input.txt --out redacted.txt

File Input/Output (CLI)

Redact a file, save the output, and keep a token store for reversibility:

veildata redact input.txt --output redacted.txt --store store.json

Reveal the file using the stored tokens:

veildata reveal redacted.txt --store store.json

Python SDK Examples

Regex-based Redaction

from veildata import Compose, RegexRedactor, TokenStore

# Create a shared TokenStore for reversible Redaction
store = TokenStore()

# Define your Redaction pipeline with the shared store
redactor = Compose([
    RegexRedactor(r"[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}", store=store),  # email
    RegexRedactor(r"\b\d{3}-\d{3}-\d{4}\b", store=store),                           # phone
])

text = "Contact John at [e-mail] or call 123-456-7890."

# --- redact the data ---
redacted_text = redactor(text)
print(redacted_text)
# -> Contact John at [REDACTED_1] or call [REDACTED_2].

# --- reveal it later ---
revealed_text = store.reveal(redacted_text)
print(revealed_text)
# -> Contact John at [e-mail] or call 123-456-7890.

spaCy Named Entity Recognition

# Requires `pip install veildata[spacy]`
from veildata.redactors.ner_spacy import SpacyNERRedactor
from veildata import TokenStore

# Shared token store for reversible revealing
store = TokenStore()

redactor = SpacyNERRedactor(
    entities=["PERSON", "ORG"],
    store=store
)

text = "Apple was founded by Steve Jobs in Cupertino."
redacted = redactor(text)
print(redacted)  # -> [REDACTED_1] was founded by [REDACTED_2] in Cupertino.

BERT NER Redaction

# Requires `pip install veildata[bert]`
from veildata.redactors.ner_bert import BERTNERRedactor
from veildata import TokenStore

store = TokenStore()
redactor = BERTNERRedactor(
    model_name="dslim/bert-base-NER",
    store=store
)

text = "John Smith works at Google in New York."
redacted = redactor(text)
print(redacted)  # -> [REDACTED_1] works at [REDACTED_2] in [REDACTED_3].

File Input/Output

from pathlib import Path
from veildata import Compose, RegexRedactor, TokenStore

# Setup redactor
store = TokenStore()
redactor = Compose([
    RegexRedactor(r"\b\d{3}-\d{3}-\d{4}\b", store=store)
])

# Read from file
input_path = Path("input.txt")
if input_path.exists():
    text = input_path.read_text()
    
    # redact
    redacted_text = redactor(text)
    
    # Write to file
    Path("redacted.txt").write_text(redacted_text)
    
    # Save store for later revealing
    store.save("store.json")

---

⚙️ CLI Configuration

spaCy PII Detection

ml:
  spacy:
    enabled: true
    model: "en_core_web_lg"
    pii_labels:
      - PERSON
      - ORG
      - GPE
      - LOC
      - NORP

BERT-Style PII Detection

ml:
  bert:
    enabled: true
    model_path: "models/pii-bert-base"
    threshold: 0.5
    label_mapping:
      EMAIL: ["B-EMAIL", "I-EMAIL"]
      PHONE: ["B-PHONE", "I-PHONE"]
      SSN: ["B-SSN", "I-SSN"]

Hybrid Detection

When using --detect-mode hybrid:

1. Run regex rules on text → produce spans with types
2. Run ML/NLP engines (spaCy + BERT) → produce spans with types + scores
3. Merge spans:
   • If spans overlap with same type → keep the union
   • If spans overlap with different types → configurable resolution

options:
  detect_mode: hybrid  # default: rules
  hybrid:
    prefer: ml  # ml | rules | longest_span

---

🛠️ Continuous Integration

• CI: .github/workflows/ci.yml runs linting, formatting, build, and tests on every push or PR.
• Publish: .github/workflows/publish.yml auto-publishes to PyPI when a new v* tag or release is created.