anonymizer-ner 1.0.0

A zero-shot NER and document anonymization package.

Anonymizer

A lightweight Python library for Zero-Shot Named Entity Recognition (NER) and Document Anonymization, powered by GLiNER 2.

Anonymizer allows you to extract custom entities from PDFs and Word documents, dynamically redact them, and most importantly, it includes a built-in memory override so you can easily correct the AI when it makes mistakes.

Installation

pip install anonymizer_ner

(Note: Not actually published to PyPI yet. Install from source).

Features

• Zero-Shot Extraction: Define what you want to extract using natural language descriptions (e.g., "address": "street addresses").
• Document Anonymization: Physically redact the extracted entities from PDFs (using secure PyMuPDF redaction) or DOCX files.
• Feedback Memory Loop: If the AI mislabels something, you can add a correction to the memory database. On all future extractions, the package will automatically override the AI and correctly label that specific text.

Usage Guide

1. Extracting Entities

import anonymizer_ner

# 1. Define AI entities using natural language
ai_entities = {
    "person": "Names of individuals",
    "address": "Street addresses and locations",
    "organization": "Companies or corporations"
}

# 2. Define strict pattern matching using Regex
regex_rules = {
    "aadhaar": r"\b\d{4}\s?\d{4}\s?\d{4}\b",
    "pan_card": r"\b[A-Z]{5}[0-9]{4}[A-Z]{1}\b"
}

with open("document.pdf", "rb") as f:
    matches = anonymizer_ner.extract(
        file_bytes=f.read(),
        filename="document.pdf",
        entity_definitions=ai_entities,
        regex_rules=regex_rules, # Pass the regex rules here!
        threshold=0.5
    )

for match in matches:
    print(f"[{match.label}] {match.text} (Page: {match.page}, Line: {match.line})")

2. Anonymizing Documents

You can securely redact all the entities you found. You can choose different strategies for how the text is hidden:

import anonymizer_ner

with open("document.pdf", "rb") as f:
    pdf_bytes = f.read()

# Default strategy: "redact" (blackout boxes)
redacted_bytes = anonymizer_ner.anonymize(
    file_bytes=pdf_bytes, 
    filename="document.pdf", 
    matches=matches,
    strategy="redact"
)

# Other strategies available:
# strategy="replace_with_tags" -> Replaces "John" with "PERSON"
# strategy="replace_with_fake" -> Replaces "John" with a fake name like "Michael"
# strategy="replace_with_indexed_tags" -> Replaces "HDFC Bank" with "ORGANIZATION_1" consistently

with open("redacted_document.pdf", "wb") as f:
    f.write(redacted_bytes)

3. Using the Feedback Loop (Memory)

If the AI makes a mistake (e.g., it tags "John Doe" as an address), you can add a correction. By default, this is saved to ~/.anonymizer_ner/memory.json, but you can isolate it per project:

import anonymizer_ner

# Set a custom memory path for this specific project (optional)
anonymizer_ner.set_memory_path("./my_project_memory.json")

# Save a correction to the local memory database
anonymizer_ner.add_correction("John Doe", "person")

This correction is saved locally to ~/.anonymizer_ner/memory.json.

The next time you call anonymizer_ner.extract(), two things happen automatically:

1. The AI's internal prompt is augmented with your examples to help it generalize.
2. If the text "John Doe" is found anywhere in the document, it is guaranteed to be tagged as a person with 100% confidence, overriding any AI predictions.

4. Using a Custom Fine-Tuned Model

By default, the package uses fastino/gliner2-base-v1. If you have fine-tuned your own GLiNER 2 model on specific domain data, you can easily load it either globally or per-extraction.

Set it globally:

import anonymizer_ner

# Can be a Hugging Face repo ID or a local path to the model weights
anonymizer_ner.set_model("path/to/my/finetuned/gliner_model")

# All subsequent extract() calls will now use your model
matches = anonymizer_ner.extract(...)

Set it per extraction:

matches = anonymizer_ner.extract(
    file_bytes=f.read(),
    filename="document.pdf",
    entity_definitions=entities,
    model_name="path/to/my/finetuned/gliner_model"
)

5. Processing Multiple Documents (Low RAM)

The anonymizer_ner package is designed to be highly memory efficient. It breaks large documents into small chunks so a 10,000-page document uses the exact same amount of RAM as a 5-page document.

However, if you are running this on a standard laptop and need to process hundreds of documents, do not use multiprocessing. Loading multiple AI models at the same time will crash your RAM. Instead, use a simple for loop to process them sequentially in complete safety:

import os
import anonymizer_ner

# Completely safe for low-RAM machines!
for filename in os.listdir("my_pdfs/"):
    if not filename.endswith(".pdf"):
        continue
        
    with open(f"my_pdfs/{filename}", "rb") as f:
        matches = anonymizer.extract(f.read(), filename, ai_entities)
        
    # Anonymize and save...

---

Technical Details

• Model: fastino/gliner2-base-v1 (runs locally, downloaded automatically on first use)
• PDF Parsing: PyMuPDF
• DOCX Parsing: python-docx