presidio-redefiner
A Presidio recognizer that uses
ReDefiNER to find personal data. ReDefiNER is a 0.8B model that extracts
the spans a plain-English rulebook defines, so Presidio can find what your own written policy describes, or what the
default PII rulebook of jungroo-ai/redefiner-pii describes.
This is an unofficial community plugin. It is not part of Presidio, and it is not maintained or endorsed by the Presidio maintainers or by Microsoft.
Install
pip install "presidio-redefiner[anonymizer]"
python -m spacy download en_core_web_sm # for Presidio's slim NLP engine in the example below
The first run downloads ReDefiNER's weights (about 0.8 GB) from Hugging Face.
The model weights (jungroo-ai/redefiner-0.8b, about 770 MB) are downloaded from Hugging Face the first time the
recognizer analyzes a text, unless they are already in your Hugging Face cache. The model runs on a CPU; an NVIDIA
GPU is used when PyTorch can use CUDA.
Usage
from presidio_analyzer import AnalyzerEngine
from presidio_analyzer.nlp_engine import NlpEngineProvider
from presidio_anonymizer import AnonymizerEngine
from presidio_redefiner import ReDefiNERRecognizer
nlp = NlpEngineProvider(nlp_configuration={"nlp_engine_name": "slim", "models": [{"lang_code": "en", "model_name": "en_core_web_sm"}]}).create_engine()
analyzer = AnalyzerEngine(nlp_engine=nlp)
analyzer.registry.add_recognizer(ReDefiNERRecognizer())
text = "Please call Rosa Diaz on +44 20 7946 0958 or email rdiaz@example.net."
results = analyzer.analyze(text=text, language="en")
print(AnonymizerEngine().anonymize(text=text, analyzer_results=results).text)
# Please call <PERSON> on <PHONE_NUMBER> or email <EMAIL_ADDRESS>.
ReDefiNER runs beside Presidio's own recognizers (email, phone, credit card and other patterns), and Presidio combines their results. The slim NLP engine gives tokens and lemmas only, without spaCy's named entities, so names and addresses come from ReDefiNER. With Presidio's default NLP engine, spaCy's entities are added as well.
Use your own rulebook
Write the types you want found, with a one-line description each, and anything that should never be found under
NOT_AN_ENTITY:
# continues the example above
from presidio_analyzer import RecognizerRegistry
from presidio_analyzer.nlp_engine import NoOpNlpEngine
policy = """[RULES]
G1 Extract every piece of personal data of the supplied types. Return exact spans from the text.
[TYPE] PERSON
[DESCRIPTION] A person's name, without titles such as "Dr." or "Ms".
[TYPE] EMPLOYEE_ID
[DESCRIPTION] An employee number, such as "E-48213".
[TYPE] SALARY
[DESCRIPTION] A person's pay: salary, bonus or hourly rate, with its currency.
[TYPE] NOT_AN_ENTITY
NE1 Names of teams, departments and products, and the company's own name.
"""
recognizer = ReDefiNERRecognizer(rulebook=policy, entity_mapping={"EMPLOYEE_ID": "ID"})
registry = RecognizerRegistry()
registry.add_recognizer(recognizer)
analyzer = AnalyzerEngine(registry=registry, nlp_engine=NoOpNlpEngine(models=[{"lang_code": "en", "model_name": "no_op"}]))
text = "Dr. Maria Okafor (E-40917) joins the Payments team at Acme on a salary of $96,500."
results = analyzer.analyze(text=text, language="en")
print(AnonymizerEngine().anonymize(text=text, analyzer_results=results).text)
# Dr. <PERSON> (<ID>) joins the Payments team at Acme on a salary of <SALARY>.
- The registry above holds only this recognizer, so only the rulebook decides what is found. If you add the recognizer to Presidio's default registry instead, Presidio's own recognizers still mark what they find (every email address, date, URL and so on), whatever the rulebook says. The no-op NLP engine needs no spaCy model.
rulebookalso takes apathlib.Pathto a rulebook file, or a rulebook dict (seeredefiner.to_text).- Types come back under their own names unless
entity_mappingrenames them (see below). - The rulebook format, and which kinds of rules the model follows, are described in the ReDefiNER README. Rules that go against what the model already knows, such as calling a person's name something other than a person, are mostly ignored. Try a rulebook on a few dozen of your own texts before you rely on it.
Recognizers that use the same model and settings share it, so a default PII recognizer and several policy recognizers load the weights once.
Options
| Argument | Default | Meaning |
|---|---|---|
model |
"jungroo-ai/redefiner-pii" |
A local folder or Hugging Face repo. The default holds a PII rulebook and takes its weights from jungroo-ai/redefiner-0.8b. |
rulebook |
None |
Your rulebook as text, a dict or a pathlib.Path. None uses the model's default rulebook. |
entity_mapping |
None |
Rulebook type to Presidio entity name. Entries add to or override the default mapping; a type mapped to None is dropped. |
min_confidence |
0.0 |
Drop spans below this ReDefiNER confidence. 0 keeps every span the model returns, as redefiner redact does; raise it to drop low-confidence spans. |
device |
None |
"cpu", "cuda", "cuda:1", ... None uses CUDA when PyTorch can use it, else the CPU. |
max_span_tokens |
None |
The longest span in model tokens. None uses the model's setting. |
revision |
None |
The Hugging Face revision of model. |
name, supported_language, version |
"ReDefiNERRecognizer", "en", "0.1.0" |
As for any Presidio recognizer. The model reads English only. |
Entity names
Rulebook types are renamed to Presidio's entity names with presidio_redefiner.DEFAULT_ENTITY_MAPPING. For the 20
types of the default PII rulebook:
| Rulebook type | Presidio entity |
|---|---|
EMAIL |
EMAIL_ADDRESS |
PHONE |
PHONE_NUMBER |
ADDRESS |
LOCATION |
GOVERNMENT_ID, ID_NUMBER |
ID |
PERSON, LOCATION, DATE_TIME, IP_ADDRESS, URL, ORGANIZATION |
unchanged (already Presidio names) |
TITLE, USERNAME, COORDINATE, OCCUPATION, AGE, GENDER, DEMOGRAPHIC, FINANCIAL, SECRET |
unchanged |
FINANCIAL covers card numbers and bank account numbers alike, so it is not renamed to Presidio's CREDIT_CARD;
remap it with entity_mapping if your pipeline needs another name. The mapping also renames a few names other
rulebooks may use: NAME to PERSON; DATE, TIME and DATE_OF_BIRTH to DATE_TIME; CARD_NUMBER and
CREDIT_CARD_NUMBER to CREDIT_CARD; IBAN to IBAN_CODE. Any other type keeps its own name.
The recognizer's supported entities are its rulebook's types after this mapping, read from the rulebook when the
recognizer is created. Each result keeps the original rulebook type in
result.recognition_metadata["redefiner_label"], so a GOVERNMENT_ID and an ID_NUMBER, both reported as ID, can
still be told apart.
How it runs
- Creating the recognizer reads only the rulebook (for the default rulebook, the model's
config.jsonand rulebook file). The weights load on the firstanalyze()call; callrecognizer.load_model()to load them earlier. - In one process, recognizers with the same
model,device,max_span_tokensandrevisionshare one loaded model. A loaded model handles one text at a time, so calls from several threads wait for each other. - The first text after a new rulebook takes longer, because the rulebook is read by the model once and then cached.
- A result's score is ReDefiNER's confidence for the span. By default every span the model returns is kept, which
favours finding more personal data over marking less text. To drop low-confidence spans, raise
min_confidence(or passscore_thresholdtoanalyze(), which filters every recognizer); choose the cutoff on a few dozen of your own texts. - Long texts are split into overlapping windows by ReDefiNER itself, so Presidio's text chunkers are not needed.
Do not rely on it alone
Automatic redaction should not be your only safeguard. The model can miss personal data, mark text that is not personal data, or give a span the wrong type, and its confidences are model scores, not guarantees. Check the output, keep people in the loop for sensitive documents, and combine it with other controls such as access limits and Presidio's pattern recognizers.
The model can give a number-like value the wrong type (for example, an IP address reported as PHONE_NUMBER). The
value is still masked, but filtering by entity type can then miss it, so for redaction keep all types on.
Tests
pip install -e ".[dev]"
pytest # fast tests, with a fake model
pytest -m slow -s # loads the real model on CPU from the Hugging Face cache
The slow test loads jungroo-ai/redefiner-pii (about 0.8 GB of weights on the first run, then cached); set
REDEFINER_PII_POINTER to a local pointer folder or another repo to test that instead.
Licence
Apache License 2.0. See LICENSE.
Metadata
Release files for presidio-redefiner 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| presidio_redefiner-0.1.0.tar.gz | 20.0 kB | Without provenance | Details for presidio_redefiner-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| presidio_redefiner-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for presidio_redefiner-0.1.0-py3-none-any.whl |
Total release size: 33.8 kB
Release files / presidio_redefiner-0.1.0.tar.gz
| Download URL | presidio_redefiner-0.1.0.tar.gz |
|---|---|
| Size | 20.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
0613c6639bcc9e725f206ba34c625400fd0aca566fbaed6095c63a7ee535d3a8
|
|
BLAKE2b-256 checksum How to use checksums |
327ea35f1fafa12f3eab98985f78fac8d3b07382ccfba40ac677cd65e0747e08
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.14.6
|
Release files / presidio_redefiner-0.1.0-py3-none-any.whl
| Download URL | presidio_redefiner-0.1.0-py3-none-any.whl |
|---|---|
| Size | 13.8 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
2a7846c49d8e92eea9a13d49651852db7b005cb68cb2a14d7a7e47ee43e74d0a
|
|
BLAKE2b-256 checksum How to use checksums |
da460bcc59805925bd3c1baf21a0dafdd1ff0c6b7989d2ab672906c1163d319b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.14.6
|
Security for presidio-redefiner 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.