datagate-llm

The inference boundary layer between your data and outbound AI requests.

Scan text for sensitive data — PII, secrets, credentials, and sector-specific identifiers — before it leaves your system and reaches an LLM API.

The Problem

In 2023, Samsung engineers accidentally leaked proprietary source code and internal meeting notes by pasting them into ChatGPT. The data was retained and potentially used for training. This is not a hypothetical risk — it is the default behavior when you send unrestricted text to an external AI model.

datagate-llm is the layer you put in front of that API call. It checks what you are about to send, tells you what it found, and lets you decide: flag it, redact it, or block it.

Install

pip install datagate-llm

Zero dependencies. Python 3.9+. Works offline.

Quickstart

from datagate_llm import scan

# Basic scan
result = scan("Contact Alice at alice@company.com or call 415-555-0192")
print(result["safe"])        # False
print(result["risk_score"])  # 0.8 (or similar)
print(result["findings"])    # list of matched spans

# Redact mode — replace PII before sending to an LLM
result = scan(
    "My SSN is 123-45-6789 and card number 4111111111111111",
    mode="redact"
)
print(result["redacted_text"])
# "My SSN is [REDACTED:universal/ssn] and card number [REDACTED:universal/credit_card]"

# Block mode — hard stop on high-risk content
result = scan("AKIAIOSFODNN7EXAMPLEKEY", sectors=["technology"], mode="block")
if result["action"] == "block":
    raise ValueError("Refusing to send credentials to LLM")

# Multi-sector scan
result = scan(
    "Patient MRN: AB12345, account 123456789012",
    sectors=["healthcare", "finance"]
)
for finding in result["findings"]:
    print(finding["rule_id"], finding["severity"], finding["confidence"])

What It Detects

Custom Pattern Templates

No regex knowledge needed. Use built-in templates to describe your organisation's sensitive identifiers.

from datagate_llm import add_pattern

add_pattern(
    name="patient_id",
    template="prefix_digits",
    prefix="PT",
    n=8,
    severity="high"
)

Available templates:

For patterns not covered by templates, use add_rule() directly with a regex string:

from datagate_llm import add_rule

# Variable length, optional parts, OR conditions
add_rule(
    id="custom/complex_id",
    pattern=r"PT-?\d{6,10}",
    severity="high",
    sector="custom"
)

How It Works

text input
    │
    ▼
tokenize()          ← NFKC normalization, zero-width char removal
    │
    ▼
match()             ← regex scan against compiled rule set
    │
    ▼
score()             ← context-aware confidence (boost / suppress words)
    │
    ▼
resolve()           ← remove overlapping spans, keep highest confidence
    │
    ▼
aggregate()         ← single risk_score in [0.0, 1.0]
    │
    ▼
build_result()      ← assemble final dict with action, findings, fingerprint

Every step is a pure function. No network calls. No disk writes. No global state except the in-process rule cache.

Scan Modes

Honest Limits

• Regex-only: datagate-llm uses deterministic pattern matching. It will not catch PII embedded in obfuscated prose, paraphrased content, or novel formats it has never seen.
• English-centric: Phone and ID patterns currently target US formats. International variants may be missed.
• No semantic understanding: "The patient's temperature was 98.6" will not be flagged as health data because there is no pattern for it. Semantic scanning requires the optional onnxruntime layer (not yet released).
• False positives are possible: Short patterns like SWIFT codes can match arbitrary uppercase strings. Use context.suppress words in your rule JSON to reduce noise.
• Not a compliance tool: Passing a scan does not mean a document is HIPAA, GDPR, or PCI-DSS compliant. Use this as one layer of defense, not the only one.

Contributing

See CONTRIBUTING.md. In short: add rules in JSON, add tests, open a PR.

License

MIT. See LICENSE.