credgate

Detects personal data before it leaves the machine, with Indian identifiers built in: Aadhaar (with Verhoeff checksum), Aadhaar Virtual ID, PAN, GSTIN, voter ID, passport, driving licence, UPI IDs, IFSC, bank accounts, payment cards, +91 mobile numbers, ABHA, and names and addresses in Latin and Indic scripts.

The detection rules live in a shared, language-neutral spec, so this package and the R package credgate find exactly the same matches. Part of the CredGate project; early development.

Use it with local LLMs (Ollama, vLLM)

Redact before the prompt reaches the model; the model only ever sees placeholders such as [PAN] and [PERSON_NAME].

import credgate, ollama

note = "Customer name: Priya Nair. PAN ABCPN1234K, mobile +91 98765 43210, wants the EMI date moved."
print(credgate.preview(note, min_confidence=0.5))   # what will be masked, and why
safe = credgate.redact(note, min_confidence=0.5)    # "Customer name: [PERSON_NAME]. PAN [PAN], mobile [MOBILE_IN], ..."
reply = ollama.chat(model="gemma3:1b", messages=[{"role": "user", "content": f"Summarise: {safe}"}])

With vLLM (or Ollama's /v1, LM Studio, any OpenAI-compatible server):

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
client.chat.completions.create(model="<served model>",
                               messages=[{"role": "user", "content": credgate.redact(note, min_confidence=0.5)}])

From the shell: credgate redact notes.txt | ollama run gemma3:1b "Summarise:". More runnable examples (ticket classification, RAG ingestion) are in the examples folder.

Check the preview. Names are found next to an honorific, relation or field label (Shri, S/o, Name:, नाम:); a name in running text without such a cue ("Customer Priya Nair called") is not detected yet. Redaction is irreversible: the model's answer cannot mention the real values (reversible tokenisation and an automatic proxy are on the roadmap).

How it compares

_{About the data: the 700 documents are synthetic - KYC forms, bank messages, call-centre notes, Hinglish chats, Hindi text and hospital summaries generated by benchmark/generate_corpus.py and included in the repository under benchmark/corpus/. No real personal data was used. Types both tools cover, exact span matching, thresholds tuned on a separate dev split for every system; time is for 1,400 documents. Synthetic data generated by us, so read with that in mind.}

Where Presidio does better. credgate is built for Indian identifiers and Indian-language context, and that is what this benchmark measures. Presidio is broader and more mature:

• Voter IDs without context: Presidio scores 1.000 here, credgate 0.677 (credgate needs a nearby keyword such as "voter" or "EPIC").
• Many more entity types: international IDs (US, UK, EU, ...), IBAN, IP and MAC addresses, URLs, crypto wallets, dates, nationality and religion - credgate detects none of these.
• Names in running text: Presidio's NER finds names without an honorific or field label; credgate does not yet.
• Anonymisation: Presidio can mask, hash or reversibly encrypt; credgate only redacts irreversibly so far.
• Beyond text: image (OCR) and structured-data redaction, a REST service and Docker images.
• Maturity and extensibility: years in production, many languages and NLP back-ends, custom recognisers; credgate is pre-alpha.
• The corpus favours credgate: we generated it; on general English text Presidio may well do better.

Full method, per-type results and how to reproduce: benchmark/README.md. The comparison is independent and not affiliated with Microsoft.

Installation

Python 3.11 or later; depends only on regex and PyYAML.

pip install credgate

The development version: pip install "git+https://github.com/dwarpal/credgate#subdirectory=pkg/python".

With Nix: nix build github:dwarpal/credgate#credgate.

Usage

import credgate

spans = credgate.detect_pii("Name: Priya Nair; PAN: ABCPN1234K; UPI: priya.n@oksbi")
for s in spans:
    print(s)
# Span(doc=0, start=6, end=16, type='person_name', value='XXXXX Nair', confidence=0.8)
# Span(doc=0, start=23, end=33, type='pan', value='XXXXXX234K', confidence=0.97)
# Span(doc=0, start=40, end=53, type='upi_id', value='XXXXX.X@Xksbi', confidence=0.99)

spans[1].value    # 'ABCPN1234K' - printing masks values, the attribute does not
spans[1].reasons  # "matched PAN pattern; context 'PAN'; holder type: individual"

detect_pii() takes a string or a list of strings and returns Span objects with the document index, 0-based end-exclusive positions, type, data class, value, canonical form, confidence and reasons. span.to_dict() gives a JSON-ready dict with the value masked (reveal=True to include it).

Redact identifiers, or preview what would be redacted and why:

credgate.redact("Name: Priya Nair; PAN: ABCPN1234K")
# 'Name: [PERSON_NAME]; PAN: [PAN]'

print(credgate.preview("Name: Priya Nair; PAN: ABCPN1234K; order no 878245671885"))
# 3 items would be redacted:
#   Name: [PERSON_NAME]; PAN: [PAN]; order no [AADHAAR]
#
#   [PERSON_NAME]  XXXXX Nair    0.80  matched Person name pattern
#   [PAN]          XXXXXX234K    0.97  matched PAN pattern; context 'PAN'; ...
#   [AADHAAR]      XXXXXXXX1885  0.42  ...; negative context 'order'

Scan a table (a pandas DataFrame, or a dict of column -> values) to see which columns hold PII; column names count as context:

report = credgate.scan_pii({
    "cust_pan": ["ABCPE1234F", "AAACR5055K", "XYZPQ9876R"],
    "notes": ["Called, Aadhaar 542229371182", "no issues", "mobile 9876543210"],
})
print(report)
# Scanned 2 column(s); PII found in 2: cust_pan, notes
#   cust_pan  pan              100% of 3 (3 whole value)  confidence 0.97  e.g. XXXXXX234F
#   notes     aadhaar           33% of 3 (0 whole value)  confidence 0.97  e.g. XXXXXXXX1182
#   notes     mobile_in         33% of 3 (0 whole value)  confidence 0.93  e.g. XXXXXX3210
pandas.DataFrame(report.to_records())  # as a table

Context makes generic patterns trustworthy; pass a column name or similar:

credgate.detect_pii("ABC1234567")                      # [] - could be anything
credgate.detect_pii("ABC1234567", context="voter_id")  # voter ID, confidence 0.97

Checksums, synthetic test data and your own detectors:

credgate.verhoeff_valid("542229371182")
credgate.synth_indian_pii(5, seed=1)   # synthetic records; never use as anonymised data

credgate.register_detector(credgate.Detector(
    id="employee_id", data_class="internal", label="Employee ID",
    patterns=credgate.bounded("EMP[0-9]{6}"),
))

Command line

credgate detect notes.txt                      # masked report
credgate detect --lines --json customers.csv   # one JSON object per match
git diff --cached | credgate detect --fail     # non-zero exit if PII is found
credgate redact notes.txt > notes.redacted.txt  # replace PII with [PAN] etc.
credgate preview notes.txt                     # what would be redacted, and why
credgate scan customers.csv                    # which columns hold PII
credgate detectors                             # list detectors

Values are masked unless you pass --reveal.

Limits

Names and addresses are found only next to a cue: an honorific (Shri, Smt, श्रीमती), a relation (S/o, D/o), a field label (Name:, पता:) or a house/village keyword followed by a PIN code. A bare 12-digit number may pass the Aadhaar checksum by chance and is reported with lower confidence.

Licence

Apache License 2.0.