Skip to main content

presidio-redefiner

A Presidio recognizer that uses ReDefiNER to find personal data. ReDefiNER is a 0.8B model that extracts the spans a plain-English rulebook defines, so Presidio can find what your own written policy describes, or what the default PII rulebook of jungroo-ai/redefiner-pii describes.

This is an unofficial community plugin. It is not part of Presidio, and it is not maintained or endorsed by the Presidio maintainers or by Microsoft.

Install

pip install "presidio-redefiner[anonymizer]"
python -m spacy download en_core_web_sm    # for Presidio's slim NLP engine in the example below

The first run downloads ReDefiNER's weights (about 0.8 GB) from Hugging Face.

The model weights (jungroo-ai/redefiner-0.8b, about 770 MB) are downloaded from Hugging Face the first time the recognizer analyzes a text, unless they are already in your Hugging Face cache. The model runs on a CPU; an NVIDIA GPU is used when PyTorch can use CUDA.

Usage

from presidio_analyzer import AnalyzerEngine
from presidio_analyzer.nlp_engine import NlpEngineProvider
from presidio_anonymizer import AnonymizerEngine
from presidio_redefiner import ReDefiNERRecognizer

nlp = NlpEngineProvider(nlp_configuration={"nlp_engine_name": "slim", "models": [{"lang_code": "en", "model_name": "en_core_web_sm"}]}).create_engine()
analyzer = AnalyzerEngine(nlp_engine=nlp)
analyzer.registry.add_recognizer(ReDefiNERRecognizer())
text = "Please call Rosa Diaz on +44 20 7946 0958 or email rdiaz@example.net."
results = analyzer.analyze(text=text, language="en")
print(AnonymizerEngine().anonymize(text=text, analyzer_results=results).text)
# Please call <PERSON> on <PHONE_NUMBER> or email <EMAIL_ADDRESS>.

ReDefiNER runs beside Presidio's own recognizers (email, phone, credit card and other patterns), and Presidio combines their results. The slim NLP engine gives tokens and lemmas only, without spaCy's named entities, so names and addresses come from ReDefiNER. With Presidio's default NLP engine, spaCy's entities are added as well.

Use your own rulebook

Write the types you want found, with a one-line description each, and anything that should never be found under NOT_AN_ENTITY:

# continues the example above
from presidio_analyzer import RecognizerRegistry
from presidio_analyzer.nlp_engine import NoOpNlpEngine

policy = """[RULES]
G1 Extract every piece of personal data of the supplied types. Return exact spans from the text.

[TYPE] PERSON
[DESCRIPTION] A person's name, without titles such as "Dr." or "Ms".

[TYPE] EMPLOYEE_ID
[DESCRIPTION] An employee number, such as "E-48213".

[TYPE] SALARY
[DESCRIPTION] A person's pay: salary, bonus or hourly rate, with its currency.

[TYPE] NOT_AN_ENTITY
NE1 Names of teams, departments and products, and the company's own name.
"""
recognizer = ReDefiNERRecognizer(rulebook=policy, entity_mapping={"EMPLOYEE_ID": "ID"})

registry = RecognizerRegistry()
registry.add_recognizer(recognizer)
analyzer = AnalyzerEngine(registry=registry, nlp_engine=NoOpNlpEngine(models=[{"lang_code": "en", "model_name": "no_op"}]))
text = "Dr. Maria Okafor (E-40917) joins the Payments team at Acme on a salary of $96,500."
results = analyzer.analyze(text=text, language="en")
print(AnonymizerEngine().anonymize(text=text, analyzer_results=results).text)
# Dr. <PERSON> (<ID>) joins the Payments team at Acme on a salary of <SALARY>.
  • The registry above holds only this recognizer, so only the rulebook decides what is found. If you add the recognizer to Presidio's default registry instead, Presidio's own recognizers still mark what they find (every email address, date, URL and so on), whatever the rulebook says. The no-op NLP engine needs no spaCy model.
  • rulebook also takes a pathlib.Path to a rulebook file, or a rulebook dict (see redefiner.to_text).
  • Types come back under their own names unless entity_mapping renames them (see below).
  • The rulebook format, and which kinds of rules the model follows, are described in the ReDefiNER README. Rules that go against what the model already knows, such as calling a person's name something other than a person, are mostly ignored. Try a rulebook on a few dozen of your own texts before you rely on it.

Recognizers that use the same model and settings share it, so a default PII recognizer and several policy recognizers load the weights once.

Options

Argument Default Meaning
model "jungroo-ai/redefiner-pii" A local folder or Hugging Face repo. The default holds a PII rulebook and takes its weights from jungroo-ai/redefiner-0.8b.
rulebook None Your rulebook as text, a dict or a pathlib.Path. None uses the model's default rulebook.
entity_mapping None Rulebook type to Presidio entity name. Entries add to or override the default mapping; a type mapped to None is dropped.
min_confidence 0.0 Drop spans below this ReDefiNER confidence. 0 keeps every span the model returns, as redefiner redact does; raise it to drop low-confidence spans.
device None "cpu", "cuda", "cuda:1", ... None uses CUDA when PyTorch can use it, else the CPU.
max_span_tokens None The longest span in model tokens. None uses the model's setting.
revision None The Hugging Face revision of model.
name, supported_language, version "ReDefiNERRecognizer", "en", "0.1.0" As for any Presidio recognizer. The model reads English only.

Entity names

Rulebook types are renamed to Presidio's entity names with presidio_redefiner.DEFAULT_ENTITY_MAPPING. For the 20 types of the default PII rulebook:

Rulebook type Presidio entity
EMAIL EMAIL_ADDRESS
PHONE PHONE_NUMBER
ADDRESS LOCATION
GOVERNMENT_ID, ID_NUMBER ID
PERSON, LOCATION, DATE_TIME, IP_ADDRESS, URL, ORGANIZATION unchanged (already Presidio names)
TITLE, USERNAME, COORDINATE, OCCUPATION, AGE, GENDER, DEMOGRAPHIC, FINANCIAL, SECRET unchanged

FINANCIAL covers card numbers and bank account numbers alike, so it is not renamed to Presidio's CREDIT_CARD; remap it with entity_mapping if your pipeline needs another name. The mapping also renames a few names other rulebooks may use: NAME to PERSON; DATE, TIME and DATE_OF_BIRTH to DATE_TIME; CARD_NUMBER and CREDIT_CARD_NUMBER to CREDIT_CARD; IBAN to IBAN_CODE. Any other type keeps its own name.

The recognizer's supported entities are its rulebook's types after this mapping, read from the rulebook when the recognizer is created. Each result keeps the original rulebook type in result.recognition_metadata["redefiner_label"], so a GOVERNMENT_ID and an ID_NUMBER, both reported as ID, can still be told apart.

How it runs

  • Creating the recognizer reads only the rulebook (for the default rulebook, the model's config.json and rulebook file). The weights load on the first analyze() call; call recognizer.load_model() to load them earlier.
  • In one process, recognizers with the same model, device, max_span_tokens and revision share one loaded model. A loaded model handles one text at a time, so calls from several threads wait for each other.
  • The first text after a new rulebook takes longer, because the rulebook is read by the model once and then cached.
  • A result's score is ReDefiNER's confidence for the span. By default every span the model returns is kept, which favours finding more personal data over marking less text. To drop low-confidence spans, raise min_confidence (or pass score_threshold to analyze(), which filters every recognizer); choose the cutoff on a few dozen of your own texts.
  • Long texts are split into overlapping windows by ReDefiNER itself, so Presidio's text chunkers are not needed.

Do not rely on it alone

Automatic redaction should not be your only safeguard. The model can miss personal data, mark text that is not personal data, or give a span the wrong type, and its confidences are model scores, not guarantees. Check the output, keep people in the loop for sensitive documents, and combine it with other controls such as access limits and Presidio's pattern recognizers.

The model can give a number-like value the wrong type (for example, an IP address reported as PHONE_NUMBER). The value is still masked, but filtering by entity type can then miss it, so for redaction keep all types on.

Tests

pip install -e ".[dev]"
pytest                 # fast tests, with a fake model
pytest -m slow -s      # loads the real model on CPU from the Hugging Face cache

The slow test loads jungroo-ai/redefiner-pii (about 0.8 GB of weights on the first run, then cached); set REDEFINER_PII_POINTER to a local pointer folder or another repo to test that instead.

Licence

Apache License 2.0. See LICENSE.

Metadata

Release files for presidio-redefiner 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for presidio-redefiner 0.1.0
File Size Uploaded Provenance Actions
presidio_redefiner-0.1.0.tar.gz 20.0 kB Without provenance Details for presidio_redefiner-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for presidio-redefiner 0.1.0
File Interpreter ABI Platform Actions
presidio_redefiner-0.1.0-py3-none-any.whl Python 3 none any Details for presidio_redefiner-0.1.0-py3-none-any.whl

Total release size: 33.8 kB

Release files / presidio_redefiner-0.1.0.tar.gz

Download URL presidio_redefiner-0.1.0.tar.gz
Size 20.0 kB
Tags Source
SHA-256 checksum
How to use checksums
0613c6639bcc9e725f206ba34c625400fd0aca566fbaed6095c63a7ee535d3a8
BLAKE2b-256 checksum
How to use checksums
327ea35f1fafa12f3eab98985f78fac8d3b07382ccfba40ac677cd65e0747e08
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.14.6

Release files / presidio_redefiner-0.1.0-py3-none-any.whl

Download URL presidio_redefiner-0.1.0-py3-none-any.whl
Size 13.8 kB
Tags Python 3
SHA-256 checksum
How to use checksums
2a7846c49d8e92eea9a13d49651852db7b005cb68cb2a14d7a7e47ee43e74d0a
BLAKE2b-256 checksum
How to use checksums
da460bcc59805925bd3c1baf21a0dafdd1ff0c6b7989d2ab672906c1163d319b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.14.6

Security for presidio-redefiner 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page