llama-index-redefiner
Personal-data masking and entity extraction for LlamaIndex, using ReDefiNER: an open model (0.8B parameters, Apache 2.0) that finds what a rulebook written in plain English defines. It runs locally on CPU or GPU. English only.
| Component | Where it goes | What it does |
|---|---|---|
ReDefiNERPIINodePostprocessor |
query engine node_postprocessors |
Masks personal data in retrieved nodes before the LLM sees them, like LlamaIndex's NERPIINodePostprocessor, and keeps a mapping so answers can be mapped back |
ReDefiNERPIIMasker |
IngestionPipeline transformations |
Masks personal data in node text before embedding and indexing, so it never reaches the embedding model or the vector store |
ReDefiNEREntityExtractor |
IngestionPipeline transformations |
Adds the entities a rulebook defines to node metadata, like LlamaIndex's EntityExtractor |
The PII components use the rulebook of
jungroo-ai/redefiner-pii by default (20 types: names, contact
details, addresses, dates, ID and card numbers and more). Pass your own rulebook to mask by your own policy.
Install
pip install llama-index-redefiner
The first call downloads the model weights (about 0.8 GB) from Hugging Face.
Mask retrieved nodes
from llama_index_redefiner import ReDefiNERPIINodePostprocessor
query_engine = index.as_query_engine(node_postprocessors=[ReDefiNERPIINodePostprocessor()])
A node that reads "Hi, this is Meera Nair from Brightwater Dental. Call me on 020 7946 0155." reaches the LLM as
"Hi, this is [PERSON_12] from [ORGANIZATION_28]. Call me on [PHONE_59]." The mapping from tags to the original text is
kept in node.metadata["__pii_node_info__"], which is excluded from what the LLM and the embedding model see, as with
LlamaIndex's own PII postprocessors.
Mask before indexing
from llama_index.core import Document
from llama_index.core.ingestion import IngestionPipeline
from llama_index_redefiner import ReDefiNERPIIMasker
pipeline = IngestionPipeline(transformations=[ReDefiNERPIIMasker()])
nodes = pipeline.run(documents=[Document(text="Hi, this is Meera Nair from Brightwater Dental. Call me on 020 7946 0155.")])
nodes[0].get_content()
# 'Hi, this is [PERSON] from [ORGANIZATION]. Call me on [PHONE].'
Nothing of the masked text is kept. Put a text splitter and an embedding model after it in the same pipeline as usual.
Extract entities into metadata
from llama_index_redefiner import ReDefiNEREntityExtractor
rulebook = """[TYPE] PERSON
[DESCRIPTION] A person's name.
[TYPE] ORGANIZATION
[DESCRIPTION] A company.
"""
pipeline = IngestionPipeline(transformations=[ReDefiNEREntityExtractor(rulebook=rulebook)])
pipeline.run(documents=[Document(text="Hi, this is Meera Nair from Brightwater Dental.")])[0].metadata
# {'entities': ['Meera Nair', 'Brightwater Dental']}
With label_entities=True each type gets its own key ({"PERSON": [...], "ORGANIZATION": [...]}), renamed through
entity_map if given.
Rulebooks and settings
A rulebook lists the types you want, with a one-line description each, and can add rules and a list of what must
never be found. See the rulebook guide.
rulebook takes the text or the path of a rulebook file.
Every component also takes model, min_confidence, none_bias, threshold, max_span_tokens, device,
revision and batch_size; see the ReDefiNER README for what the model settings do.
Limits
- Not the only safeguard. The model misses some personal data and marks some text that is not personal. Check it on your own documents.
- Speed. On CPU the first call reads the rulebook (several seconds for the PII rulebook) and is then cached; later short texts take under a second. All nodes of a call go to the model in one batch.
- English only.
Development
pip install -e ".[dev]"
pytest # fast tests with a stand-in model
pytest -m slow # the real model on CPU
Licence
Apache License 2.0. See LICENSE. Made by Jungroo AI Labs.
Metadata
Release files for llama-index-redefiner 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| llama_index_redefiner-0.1.0.tar.gz | 12.3 kB | Without provenance | Details for llama_index_redefiner-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| llama_index_redefiner-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for llama_index_redefiner-0.1.0-py3-none-any.whl |
Total release size: 23.6 kB
Release files / llama_index_redefiner-0.1.0.tar.gz
| Download URL | llama_index_redefiner-0.1.0.tar.gz |
|---|---|
| Size | 12.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
eb76bb2f12cd4751b7ed3adaf12e364875cfd286814a7746bae0c77e1307fa8c
|
|
BLAKE2b-256 checksum How to use checksums |
af5489f6a1828bbd4d94fe0bc7ef745f2d6cac5dc50ceae88b8fff35161ae6dd
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.14.6
|
Release files / llama_index_redefiner-0.1.0-py3-none-any.whl
| Download URL | llama_index_redefiner-0.1.0-py3-none-any.whl |
|---|---|
| Size | 11.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
2e014ced03c1f937a3b328a8aad4c8c08fdbe0f88d12cda55d3e714e7cc48694
|
|
BLAKE2b-256 checksum How to use checksums |
d45193874d284aee1b08bd97da1efb0979956b1aa5e51737570ea075369f9636
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.14.6
|
Security for llama-index-redefiner 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.