credgate

Detects personal data before it leaves the machine, with Indian identifiers built in: Aadhaar (with Verhoeff checksum), Aadhaar Virtual ID, PAN, GSTIN, voter ID, passport, driving licence, UPI IDs, IFSC, bank accounts, payment cards, +91 mobile numbers, ABHA, and names and addresses in Latin and Indic scripts.

The detection rules live in a shared, language-neutral spec, so this package and the R package credgate find exactly the same matches. Part of the CredGate project; early development.

Use it with local LLMs (Ollama, vLLM)

Redact before the prompt reaches the model; the model only ever sees placeholders such as [PAN] and [PERSON_NAME].

import credgate, ollama

note = "Customer name: Priya Nair. PAN ABCPN1234K, mobile +91 98765 43210, wants the EMI date moved."
print(credgate.preview(note, min_confidence=0.5))   # what will be masked, and why
safe = credgate.redact(note, min_confidence=0.5)    # "Customer name: [PERSON_NAME]. PAN [PAN], mobile [MOBILE_IN], ..."
reply = ollama.chat(model="gemma3:1b", messages=[{"role": "user", "content": f"Summarise: {safe}"}])

With vLLM (or Ollama's /v1, LM Studio, any OpenAI-compatible server):

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
client.chat.completions.create(model="<served model>",
                               messages=[{"role": "user", "content": credgate.redact(note, min_confidence=0.5)}])

From the shell: credgate redact notes.txt | ollama run gemma3:1b "Summarise:". More runnable examples (ticket classification, RAG ingestion) are in the examples folder.

Check the preview. Names are found next to an honorific, relation or field label (Shri, S/o, Name:, नाम:); a name in running text without such a cue ("Customer Priya Nair called") is not detected yet. Redaction is irreversible: the model's answer cannot mention the real values (reversible tokenisation and an automatic proxy are on the roadmap).

How it compares

_{About the data: the 700 documents are synthetic - KYC forms, bank messages, call-centre notes, Hinglish chats, Hindi text and hospital summaries generated by benchmark/generate_corpus.py and included in the repository under benchmark/corpus/. No real personal data was used. Types both tools cover, exact span matching, thresholds tuned on a separate dev split for every system; time is for 1,400 documents. Synthetic data generated by us, so read with that in mind.}

credgate is built for Indian identifiers and Indian-language context, and that is what this benchmark measures. Presidio is broader and more mature: it detects many types credgate does not (international IDs, IBAN, IP addresses, URLs, dates), finds names without an honorific or label, offers reversible encryption, redacts images, and wins here on voter IDs without context. Method, per-type results, where Presidio is ahead, and how to reproduce: benchmark/README.md. The comparison is independent and not affiliated with Microsoft.

Installation

Python 3.11 or later; depends only on regex and PyYAML.

pip install credgate

The development version: pip install "git+https://github.com/dwarpal/credgate#subdirectory=pkg/python".

With Nix: nix build github:dwarpal/credgate#credgate.

Usage

import credgate

spans = credgate.detect_pii("Name: Priya Nair; PAN: ABCPN1234K; UPI: priya.n@oksbi")
for s in spans:
    print(s)
# Span(doc=0, start=6, end=16, type='person_name', value='XXXXX Nair', confidence=0.8)
# Span(doc=0, start=23, end=33, type='pan', value='XXXXXX234K', confidence=0.97)
# Span(doc=0, start=40, end=53, type='upi_id', value='XXXXX.X@Xksbi', confidence=0.99)

spans[1].value    # 'ABCPN1234K' - printing masks values, the attribute does not
spans[1].reasons  # "matched PAN pattern; context 'PAN'; holder type: individual"

detect_pii() takes a string or a list of strings and returns Span objects with the document index, 0-based end-exclusive positions, type, data class, value, canonical form, confidence and reasons. span.to_dict() gives a JSON-ready dict with the value masked (reveal=True to include it).

Redact identifiers, or preview what would be redacted and why:

credgate.redact("Name: Priya Nair; PAN: ABCPN1234K")
# 'Name: [PERSON_NAME]; PAN: [PAN]'

print(credgate.preview("Name: Priya Nair; PAN: ABCPN1234K; order no 878245671885"))
# 3 items would be redacted:
#   Name: [PERSON_NAME]; PAN: [PAN]; order no [AADHAAR]
#
#   [PERSON_NAME]  XXXXX Nair    0.80  matched Person name pattern
#   [PAN]          XXXXXX234K    0.97  matched PAN pattern; context 'PAN'; ...
#   [AADHAAR]      XXXXXXXX1885  0.42  ...; negative context 'order'

Scan a table (a pandas DataFrame, or a dict of column -> values) to see which columns hold PII; column names count as context:

report = credgate.scan_pii({
    "cust_pan": ["ABCPE1234F", "AAACR5055K", "XYZPQ9876R"],
    "notes": ["Called, Aadhaar 542229371182", "no issues", "mobile 9876543210"],
})
print(report)
# Scanned 2 column(s); PII found in 2: cust_pan, notes
#   cust_pan  pan              100% of 3 (3 whole value)  confidence 0.97  e.g. XXXXXX234F
#   notes     aadhaar           33% of 3 (0 whole value)  confidence 0.97  e.g. XXXXXXXX1182
#   notes     mobile_in         33% of 3 (0 whole value)  confidence 0.93  e.g. XXXXXX3210
pandas.DataFrame(report.to_records())  # as a table

Context makes generic patterns trustworthy; pass a column name or similar:

credgate.detect_pii("ABC1234567")                      # [] - could be anything
credgate.detect_pii("ABC1234567", context="voter_id")  # voter ID, confidence 0.97

Checksums, synthetic test data and your own detectors:

credgate.verhoeff_valid("542229371182")
credgate.synth_indian_pii(5, seed=1)   # synthetic records; never use as anonymised data

credgate.register_detector(credgate.Detector(
    id="employee_id", data_class="internal", label="Employee ID",
    patterns=credgate.bounded("EMP[0-9]{6}"),
))

Command line

credgate detect notes.txt                      # masked report
credgate detect --lines --json customers.csv   # one JSON object per match
git diff --cached | credgate detect --fail     # non-zero exit if PII is found
credgate redact notes.txt > notes.redacted.txt  # replace PII with [PAN] etc.
credgate preview notes.txt                     # what would be redacted, and why
credgate scan customers.csv                    # which columns hold PII
credgate detectors                             # list detectors

Values are masked unless you pass --reveal.

Limits

Names and addresses are found only next to a cue: an honorific (Shri, Smt, श्रीमती), a relation (S/o, D/o), a field label (Name:, पता:) or a house/village keyword followed by a PIN code. A bare 12-digit number may pass the Aadhaar checksum by chance and is reported with lower confidence.

Licence

Apache License 2.0.